Why We Spent 20 Years Protecting Databases from Analytics (and Why AI Just Broke That Truce)

8:53am on a Monday

I want to take you back roughly twenty years.

It is 8:53am on a Monday morning in central London and I am sitting at my desk staring at the screen. My coffee is untouched. My palms are sweating.

The ETL job is still running.

If that phrase means nothing to you, it was enough in the mid-2000s to strike fear into the heart of any production DBA. At the time I was the database lead for a SaaS platform serving customers across the UK, most of whom would start logging in from 9am.

The weekend ETL process (Extract, Transform, Load) pulled data out of the operational database, reshaped it and pushed it into the enterprise data warehouse. It ran every weekend and it was supposed to finish long before Monday morning.

It had not.

CPU pinned. Redo logs churning. User sessions beginning to queue.

In a few minutes customers would start calling support. Support would start calling me. And somewhere upstairs, the CEO would notice that his dashboard was still showing last week’s numbers.

I had two options. Let it run and hope it finished before the system buckled under real user traffic. Or kill it and spend the next hour watching rollback potentially take even longer, while guaranteeing that the warehouse would be stale until the following weekend.

How did we end up building systems where that was a normal Monday morning dilemma?

Why We Built the Wall

For the better part of two decades, the industry answered that question in one consistent way: keep analytics away from operational systems.

Transactional databases were designed to process orders, update accounts and record events predictably. Analytical workloads were different. They were heavy, exploratory and often poorly constrained. They scanned large portions of data, built aggregates, joined everything to everything and consumed CPU and I/O in bursts that were difficult to forecast.

Putting the two together in the same system was a recipe for contention.

So we separated them.

We built ETL pipelines. We built data warehouses. Later, we built data lakes and lakehouses. We introduced replication, change data capture and streaming. Each innovation was, in its own way, an attempt to preserve the integrity of the system of record while still making data available for analysis.

This was not fashion. It was defensive architecture.

The separation protected revenue-generating systems from analytical curiosity. It provided workload isolation. It gave operations teams a fighting chance of keeping Monday morning uneventful.

It Was Never Just About Performance

Enterprise environments rarely have a single system of record. A CRM system holds customer interactions. An ordering platform tracks transactions. Billing lives somewhere else. Supply chain somewhere else again.

The warehouse became not just a safety valve for analytics, but a unifying layer. It was the place where disparate operational systems could be reconciled into something coherent.

For years, this model worked.

Dashboards were allowed to be slightly stale. Reports could reflect yesterday’s state. Humans tolerate delay. In fact, they often prefer it. Analysis takes time, and decisions are rarely made in milliseconds.

The truce held because the consumer was human.

AI Changes the Consumer

AI agents change that.

An AI agent does not log in at 9am. It does not wait for a dashboard refresh. It does not tolerate yesterday’s numbers if it is expected to act on what is true right now.

Inference is not reporting. It is decision execution at machine speed.

If an agent is recommending a next action, approving a transaction, adjusting a price or triggering a workflow, the freshness of the underlying data becomes materially important. Close enough is no longer good enough. Staleness is no longer cosmetic. It alters outcomes.

The architectural assumption that analytics can safely run on a delayed copy of operational data begins to fracture.

This does not mean warehouses were a mistake. It does not mean lakehouses are obsolete. It does not mean streaming pipelines were misguided.

It means they were optimised for a different consumer.

For years, we optimised for human analysis. Now we are increasingly optimising for machine-driven action.

That is a different problem.

The Balance of Trade-Offs

For two decades, the answer was clear: keep them apart.

Protect the system of record. Move the data. Analyse it somewhere else. Accept a little delay in exchange for stability and control.

That architecture was forged in moments exactly like that Monday morning at 8:53am, CPU pinned, redo logs churning, business users about to log in.

AI does not invalidate that history. It simply changes the balance of trade-offs.

The truce between operational databases and analytics was built for a world where humans consumed insight.

We are now entering a world where machines consume state.

And that changes the conversation.

AI Doesn’t Read Dashboards… and That Changes Everything for Databases

A bank executive opens a fraud dashboard in Microsoft Power BI.

Losses by region, chargeback ratios, transaction velocity trends and a heatmap of anomalous activity. The numbers refresh within minutes. Data flows out of the system of record, is reshaped and aggregated, then presented for interpretation.

This is contemporary analytics: fast and operationally impressive. But it remains interpretive. It explains what is happening, while intervention occurs elsewhere – inside a fraud model embedded in the execution path, deciding in milliseconds whether money moves or an account is frozen.

Reporting systems describe what has already occurred. Even when refreshed every few minutes, they are retrospective. Inference systems are anticipatory. They evaluate the present in order to shape what happens next.

For two decades, enterprise data platforms were built around a deliberate separation between systems of record and analytical platforms. The system of record handled revenue-generating transactions; analytics operated on copies refreshed hourly or even every few minutes. Latency narrowed, but the boundary remained.

AI systems do not consume summaries, however fresh. They make decisions inside the transaction itself. A real-time fraud model does not want a recently refreshed extract; it requires the authoritative state of the business at the moment of decision. When automation replaces interpretation, data freshness becomes a decision integrity requirement. That shift changes the role of the database entirely.

Snapshots ≠ State

The difference is not batch versus real time. It is snapshot versus canonical state.

A snapshot is a materialised representation of state at a prior point in time – even if that point is only moments earlier. It may be refreshed frequently or continuously streamed, but it remains a copy. The system of record contains the canonical state of the enterprise – balances, limits, flags and relationships – reflecting the legal and financial truth when a transaction commits.

In fraud detection, that distinction is decisive. A dashboard can tolerate slight delay because its purpose is explanation. A model embedded in the execution path cannot. It must evaluate current balance, velocity and account status, not a recently materialised representation.

For years, we increased the distance between analytics and the system of record to protect transactional stability. That separation reduced risk in a world where insight followed action.

Automation reverses that order. Insight now precedes action. Once decisions are automated, the gap between a copy of the data and the authoritative source becomes consequential.

When Almost Right Is Still Wrong

If a fraud dashboard is slightly stale, an analyst may adjust a threshold next week. When a fraud model evaluates incomplete or delayed state, the error is executed immediately and repeated at scale.

False declines can lock customers out within minutes. False approvals can leak substantial losses before discrepancies surface in reporting. Automation compresses time and amplifies mistakes because there is no interpretive buffer.

Real-time intervention is inevitable. Competitive pressure and regulatory scrutiny demand it. But once decisions are automated, tolerance for architectural distance shrinks. A delay harmless in reporting can be material in a decision stream. A dataset “close enough” for analytics may be insufficient for automated intervention.

The risk is not that dashboards are wrong; it is that forward-looking systems may act on something almost right.

Databases in the Age of Intervention

When fraud detection becomes automated intervention rather than retrospective analysis, the requirements on the data platform change. Freshness is defined at the moment of decision, not by refresh intervals.

Replication patterns take on new significance. Asynchronous copies and downstream materialisations were designed to protect the system of record. They optimise scale and isolation, but every layer introduces potential lag or divergence. For reporting, that trade-off is acceptable. For automated decisions in revenue-generating workflows, it becomes risk.

Workload separation also looks different. When analytics is retrospective, distance protects performance. When inference is embedded in operational workflows, proximity to live transactional state matters. The challenge is enabling safe, predictable access without compromising correctness.

Fraud detection is simply the clearest example. Dynamic pricing, credit approvals, supply chain routing and clinical triage all follow the same pattern. The model is not generating a report about what happened; it is evaluating the present to influence what happens next.

For decades, enterprise architecture assumed intelligence followed events. As AI systems become anticipatory and automated, intelligence precedes action. The database is no longer simply the foundation of record-keeping.

It becomes part of how the future is decided – whether we are comfortable with that or not.

Databases Were Built for Humans – AI Agents Change the Equation

For more than a decade, the industry has been preparing for a data explosion.

Zettabytes. Exponential curves. Hockey sticks on slides. Whether it was IDC’s DataSphere forecasts or countless vendor keynotes, the message was consistent: the amount of data created and stored worldwide was about to grow very, very fast.

And to be fair, that part largely went to plan.

Enterprises adapted. Storage scaled out. Cloud elasticity became normal. Analytical workloads were pushed away from systems of record. The industry did the work required to survive – and even thrive – in a world of exploding data volumes.

What almost nobody questioned, however, was a much quieter assumption baked into all of that planning.

The Assumption Nobody Revisited

All of those forecasts, explicit or implicit, assumed that the users of enterprise systems would remain human.

Humans are slow. Humans are bursty. Humans sleep.

Even power users have natural limits, predictable working patterns and an instinct for self-preservation when systems start pushing back. Entire generations of database design, connection management and capacity planning quietly depend on those characteristics.

It wasn’t a bad assumption. It was a reasonable one. Until it wasn’t.

A Step-Change, Not a Trend

What has changed is not just how much data exists, but who – or what – is accessing it.

AI agents introduce a new class of user into enterprise computing: non-human, machine-speed actors operating directly against application logic and data sources. This isn’t a continuation of an existing trend. It’s a step-change.

You’re not adding more users along the same curve. You’re changing the curve itself.

The data explosion was predicted. The user explosion – at least in this form – was not.

Why AI Agents Break Old Rules

AI agents don’t just behave like very enthusiastic humans.

They are fundamentally different:

Speed: they operate at machine speed, turning milliseconds into meaningful units of work
Relentlessness: they don’t pause, sleep or slow down unless explicitly forced to
Unpredictability: agentic workflows fan out, retry, amplify and cascade in ways humans never could

These aren’t “power users”. They’re closer to autonomous load generators.

When Agents Hit Systems of Record

Critically, AI agents don’t want last night’s report.

They want now.

That pulls them towards operational systems of record – the RDBMS platforms that were carefully protected for the last twenty years from exactly this kind of access pattern. Read replicas help, until they don’t. Caches help, until coherence matters. Copy lag becomes a business problem, not a technical detail.

The long-standing truce between OLTP and everything else is under strain.

Capacity Planning Enters the Chaos Zone

Traditional infrastructure planning assumes that tomorrow looks broadly like yesterday, just a bit bigger.

AI agents break that assumption.

Sudden workload spikes. Non-linear fan-out. Cost curves that move faster than budgeting cycles. Organisations are forced into an uncomfortable choice: over-provision aggressively and accept unpredictable cloud bills, or under-provision and risk outages in systems that now sit directly on critical decision paths.

Capacity planning stops being optimisation. It becomes risk management.

This Is Already Happening

None of this is theoretical.

Organisations are already talking openly about AI agents as part of their workforce – not as tools, but as actors performing work at scale.

Enterprises are comfortable counting tens of thousands of AI agents as “workers”, but it shouldn’t be surprising when those workers behave very differently to humans – and place very different demands on the systems beneath them.

The Equation Has Changed

The data explosion followed the forecast.

The explosion in users did not.

Databases were built for humans – slow, bursty, predictable ones – and that assumption shaped everything from architecture to cost models. AI agents don’t fit that mould… and pretending they do is how organisations drift into outages, runaway costs or both.

Databases were built for humans. AI agents didn’t get the memo – and they’re already in production.

Inferencing Is a Database Problem Disguised as an AI Problem

I have a habit of becoming interested in technology trends only once they collide with reality. Flash memory wasn’t interesting to me because it was new – it was interesting because it broke long-held assumptions about how databases behaved under load.

Cloud computing wasn’t interesting to me because infrastructure became someone else’s problem. It became interesting when database owners started making uncomfortable compromises just to get revenue-affecting systems to run acceptably in the cloud. Compute was routinely overprovisioned to compensate for storage performance, leading to large bills for resources that were mostly idle. At the same time, “modernisation” began to feel less like an architectural necessity and more like a convenient justification for expensive consultancy services.

And now, just when I thought flashdba had nothing left to say, AI is following the same path.

We’ve Seen This Movie Before

For the last couple of years, most of the attention has been on training. Bigger models, more parameters, more GPUs, massive share prices. That focus made sense because training is visible, centralised and easy to reason about in isolation. But as inferencing starts to move up into the enterprise, something changes.

In the enterprise, inferencing stops being an interesting AI capability and starts becoming part of real business workflows. It gets embedded into customer interactions, operational decisions and automated processes that run continuously, not just when someone pastes a prompt into a chat window. At that point, the constraints change dramatically.

Enterprise inferencing is no longer about what a model knows. It is about what the business knows right now. And that is where things begin to feel very familiar to anyone responsible for systems of record.

Because once inferencing depends on real-time access to authoritative operational data, the centre of gravity shifts away from models and back towards databases. Latency matters. Consistency matters. Concurrency matters. Security boundaries matter. Above all, correctness matters.

This is the point at which inferencing stops looking like an AI problem and starts looking like what it actually is: a database problem, wearing an AI costume.

Inferencing Changes Once It Becomes Operational

While inferencing remains something that sits at the edge of the enterprise, its demands are relatively modest: a delayed response is tolerable… slightly stale data is acceptable. If an answer is occasionally wrong, the consequences are usually limited to a poor user experience rather than a failed business process.

That changes quickly once inferencing becomes operational. When it is embedded directly into business workflows, inferencing is no longer advisory… it becomes participatory. It influences decisions, triggers actions and – increasingly – operates in the same execution path as the systems of record themselves. At that point, inferencing stops consuming convenient snapshots of data and starts demanding access to live context data.

What is Live Context?

By live context, I don’t mean training data, feature stores or yesterday’s replica. I mean current, authoritative operational data, accessed at the point a decision is being made. Data that reflects what is happening in the business right now, not what was true at some earlier point in time. This context is usually scoped to a specific customer, transaction or event and must be retrieved under the same consistency, security and governance constraints as the underlying system of record. In other words, a relational database. Your relational database.

Live Context gravitates towards RDBMS systems of record. It does not appear spontaneously – it is created at the moment a business state changes: when an order is placed, a payment is authorised, an entitlement is updated or a limit is breached, that change becomes real only when the transaction is committed to the RDBMS. Until then, it is provisional.

Analytical platforms can consume that state later, but they do not create it. Feature stores, caches and replicas can approximate it, but they do so after the fact. The only place where the current state of the business definitively exists is inside the operational production databases that process and commit transactions.

As inferencing becomes dependent on live context, it is therefore pulled towards those databases. Not because they are designed for AI workloads, and certainly not because this is desirable, but because they are the source of truth. If an inference is expected to reflect what is true right now, it must, in some form, depend on the same data paths that make the business run.

This is where the tension becomes unavoidable.

Inferencing Is Now A Database Problem

Once inferencing becomes dependent on live context, it inherits the constraints of the systems that provide that context. Performance, concurrency, availability, security and correctness are no longer secondary considerations. They become defining characteristics of whether inferencing can be trusted to operate inside business-critical workflows at all.

This is why enterprise AI initiatives are unlikely to succeed or fail based on model accuracy alone. They will succeed or fail based on how well inferencing workloads coexist with production databases that were never designed, built or costed with AI in mind. At that point, inferencing stops being an AI problem to be delegated elsewhere and becomes a database concern that must be understood, designed for and owned accordingly.

The Biggest Gap In The Clouds? High Performance RDBMS

Over the course of the last few blog posts, we’ve looked at how an increasing number of database workloads are migrating to the cloud, how there is more than one path to get there… and why overprovisioning is one of the biggest challenges to overcome.

We’re talking about business-critical application workloads here: big, complex, demanding, mission-critical, sensitive, performance-hungry… When on-prem, they are almost certainly running on dedicated, high-end infrastructure. And that’s a potential issue when you then migrate them to run on “someone else’s computer“.

As we’ve discussed before, the cloud is really a big pool of discrete resources and services, all of which are available on demand. You want a managed PostgreSQL instance? Click! It’s yours. You want three hundred virtual machines on which you can install your own software? Clickety-click! Off you go. If you’ve got the budget, the cloud has got a way for you to spend it. But underneath it all, whether you are using PaaS databases supplied by the cloud provider or installing the database software on IaaS systems, you are sharing that infrastructure – and the available performance – with the rest of the world.

Cloud Outcomes: Optimization versus Modernization

For some database workloads moving to the cloud, the modernization path will be the best fit, which means they will likely move to Platform-as-a-Service solutions where the day-to-day management of the database, operating system and infrastructure is taken care of by the cloud provider. Some examples of this path: on-prem SQL Server databases moving to Azure SQL Managed Instances; Oracle Databases moving to AWS’s Oracle RDS solution, etc.

But there is usually a certain class of database workload which doesn’t easily fit into these pre-packaged PaaS solutions: the big, the complex, the gnarly… the monsters of your data centre. And they inevitably end up in Infrastructure-as-a-Service… or stuck on-prem. For customers choosing the IaaS route (the “optimization path” in cloud-speak), the cloud provider manages the infrastructure but the customer is still responsible for the database and operating system.

Obviously, IaaS has a higher management overhead than PaaS, but often the journey to IaaS is simpler (essentially more of a lift and shift approach), while PaaS solutions often require a more complex migration. Especially with some cloud providers, where the recommended PaaS solution is actually a different database product entirely (for example, Oracle customers moving to Google Cloud or Microsoft Azure will be recommended by those cloud providers to move to Cloud SQL and Managed PostgreSQL respectively).

I/O Performance Is The Biggest Challenge

My view is that PaaS solutions are the best path for all appropriate workloads, but there will always be some outliers which need to move to IaaS. Almost by definition, those are the most high-profile, demanding, expensive, revenue-affecting… in fact… the most interesting workloads. And in all the cases I’ve seen, I/O performance has been the limiting factor.

It’s relatively easy to get a lot of compute power in the cloud. But as soon as you start ramping up the amount of data you need to read and write, or demanding that those reads and writes have very fast, predictable response times, you hit problems. In other words, if latency, IOPS or throughput are your metrics of choice, you’d better be ready to start doing unnatural things.

And it’s not necessarily the case that your required level of performance cannot be achieved. Often, it’s more correct to say that your required levels of performance cannot be achieved at an acceptable cost. Because it turns out that the following statement is just as true in the cloud as it ever was on-prem:

Performance and Cost are two sides of the same coin…

This is why I believe that the biggest gap in the cloud providers’ product portfolios today is in the area of high performance relational databases: primarily Oracle Database and Microsoft SQL Server. The PaaS solutions are designed for the average workloads, not the high-end. A complex database running on, for example, Oracle Exadata will struggle to run on a vanilla IaaS deployment – while the refactoring required to take that database and migrate it to Managed PostreSQL is almost unimaginable.

Cloud Compromises: Constrained and Optimized CPUs

Imagine the scenario where you wonder into a clothing store to buy a t-shirt. You find a design you like in size “Medium” but it’s too tight (I guess #lockdown has been unkind to us all…) so you ask for the next size up. But when it arrives, you notice something bizarre: the “Large” is not only wider and longer, it also has an extra arm hole. Yes, there are enough holes for three arms as well as your head. Even more bizarrely, the “XL” size has four sets of sleeves, while the “Small” has only one and the “XS” none at all!

Surprisingly, this analogy is very applicable to cloud computing, where properties like compute power, memory, network bandwidth, capacity and performance are often tied together. As we saw in the previous post, a requirement for a certain amount of read I/O Operations Per Second (IOPS) can result in the need to overprovision unwanted capacity and possibly even unnecessary amounts of compute power.

But there is one situation where this causes extra levels of pain: when the workload in question is database software which is licensable by CPU cores (e.g. Oracle Database, Microsoft SQL Server).

To extend the opening analogy into total surrealism, imagine that the above clothing store exists in a state which collects a Sleeve Tax of %100 of the item value per sleeve. Now, your chosen t-shirt might be $40 but the Medium size will cost you $120, the Large $160 and the XXXXXL (suitable for octopods) a massive $360.

Luckily, the cloud providers have a way to help you out here. But it kind of sucks…

Constrained / Optimized VM Sizes

If you need large amounts of memory or I/O, the chances are you will have to pick a VM type which has a larger number of cores. But if you don’t want to buy databases licenses for these additional cores (because you don’t need the extra CPU power), you can choose to restrict the VM instance so that it only uses a subset of the total available cores. This is similar to the concept of logical partitioning which you may already have used on prem. Here are two examples of this practice from the big hyperscalers:

Microsoft Azure: Constrained vCPU capable VM sizes

Amazon Web Services: Introducing Optimize CPUs for Amazon EC2 Instances

As you can see, Microsoft and AWS have different names for this concept, but the idea is the same. You provision, let’s say, a 128 vCPU instance and then you restrict it to only using, for example, 32 vCPUs. Boom – you’ve dropped your database license requirement to 25% of the total number of vCPUs. Ok so you only get the compute performance of 25% too, but that’s still a big win on the license cost… right?

Well yes but…

There’s a snag. You still have to pay the full cost of the virtual machine despite only using a fraction of its resources. The monthly cost from the cloud provider is the same as if you were using the whole machine!

To quote Amazon (emphasis mine):

Please note that CPU optimized instances will have the same price as full-sized EC2 instances of the same size.

Or to quote the slightly longer version from Microsoft (emphasis mine):

The licensing fees charged for SQL Server or Oracle are constrained to the new vCPU count, and other products should be charged based on the new vCPU count. This results in a 50% to 75% increase in the ratio of the VM specs to active (billable) vCPUs. These new VM sizes allow customer workloads to use the same memory, storage, and I/O bandwidth while optimizing their software licensing cost. At this time, the compute cost, which includes OS licensing, remains the same one as the original size.

It’s great to be able to avoid the (potentially astronomical) cost of unnecessary database licences, but this is still a massive compromise – and the cost will add up over each month you are billed for compute cores that you literally cannot use. Again, this is the public cloud demonstrating that inefficiency and overprovisioning are to be accepted as a way of life.

Surely there must be a better way?

Spoiler alert: there IS a better way

Overprovisioning: The Curse Of The Cloud

I want you to imagine that you check in to a nice hotel. You’ve had a good day and you feel like treating yourself, so you decide to order breakfast in your room for the following morning. Why not? You fill out the menu checkboxes… Let’s see now: granola, toast, coffee, some fruit. Maybe a juice. That will do nicely.

You hang the menu on the door outside, but later a knock at the door brings bad news: You can only order a maximum of three items for breakfast. What? That’s crazy… but no amount of arguing will change their rules. Yet you really don’t want to choose just three of your five items. So what do you do? The answer is simple: you pay for a second hotel room so you can order a second breakfast.

Welcome to the world of overprovisioning.

Overprovisioning = Inefficiency

Overprovisioning is the act of deploying – and paying for – resources you don’t need, usually as a compromise to get enough of some other resource. It’s a technical challenge which results in a commercial or financial penalty. More simply, it’s just inefficiency.

The history of Information Technology is full of examples of this as well as technologies to overcome it: virtualization is a solution designed to overcome the inefficiency of deploying multiple physical servers; containerisation overcomes the inefficiency of virtualising a complete operating system many times… it’s all about being more efficient so you don’t have to pay for resources you don’t really need.

In the cloud, the biggest source of overprovisioning is the way that cloud resources like compute, memory, network bandwidth, storage capacity and performance are packaged together. If you need one of these in abundance, the chances are you will need to pay for more of the others regardless of whether they are required or not.

Overprovisioning = Compromise

As an example, at the time of writing, Google Cloud Platform’s pd-balanced block storage options provide 6 read IOPS and 6 write IOPS per GB of capacity:

* Persistent disk IOPS and throughput performance depends on disk size, instance vCPU count, and I/O block size, among other factors.

Consider a 1TB database with a reasonable requirement of 30,000 read IOPS during peak load. To build a solution capable of this, 5000GB (i.e. 5TB) of capacity would need to be provisioned… meaning 80% of the capacity is wasted!

Worse still, the “Read IOPS per instance” row of the table tells us that some of the available GCP instance types may not be able to hit our 30,000 requirement, meaning we may have to (over)provision a larger virtual machine type and pay for cores and RAM that aren’t necessary (by the way, I’m not picking on GCP here, this is common to all public clouds).

But the real sucker punch is that, if this database is licensed by CPU cores (e.g. Oracle, SQL Server) and we are having to overprovision CPU cores to get the required IOPS numbers, we now have to pay for additional, unwanted – and very expensive – database licenses.

Overprovisioning = Overpaying

Let’s not imagine that this is a new phenomenon. If you’ve ever over-specced a server in your data centre (me), if you’ve ever convinced your boss that you need the Enterprise Edition of something because you thought it would be better for your career prospects (also me), or if you’ve ever spent £350 on a thermal imaging camera just so you can win an argument about whether you need a new front door (I neither admit nor deny this) then you have been overprovisioning.

It’s just that the whole nature of cloud computing, with it’s self-service, on-demand, limitlessly-scalable charateristics make it so easy to overprovision things all the time. So while the amounts may seem small when shown on the cloud provider’s Price per hour list, when you multiply them by the number of VMs, the number of regions and the number of hours in a year, they start to look massive on your bill.

And when you consider the knock on effects on database licensing, things really get painful. But let’s save that for the next blog post…

Choosing The Right Path To The Cloud

What happens when customers with on-prem databases decide they want to embrace the public cloud? If you’ve been following the story so far, it is my assertion that most of “the easy stuff” has already moved to the cloud: backups, websites, test/dev suites, videos of cats etc. We are now in the next phase of enterprise cloud adoption, where all the difficult, complex, gnarly stuff is being considered – and that usually means business-critical databases.

The guys at IBM Cloud have a name for this: they call it “Chapter Two“. In fact here’s a quote from page one of a recent IBM annual report (emphasis added by me):

“… the most challenging and complex work of these digital transformations still lies ahead. We call this work ‘Chapter 2,’ in which our clients modernize and move their mission-critical workloads to the cloud, and infuse AI deep into the decision-making workflows of their business.”

It seems that IBM knows it missed out on Chapter 1, but is determined to have a different result for this second, complex wave of cloud transformation. It has a long way to go to catch up, though, because Microsoft and AWS are dominating this market right now – while Oracle and Google are both racing hard to build their own shares of this massive opportunity.

Regardless of which public cloud is being considered, the question for most customers isn’t so much “Who?” as the more thorny issue of “How?”

So with that in mind, let’s have a look at the three main approaches to moving complex database applications into the public cloud.

Three Journeys To The Public Cloud

When it comes to existing, business-critical database-based applications, there are three high-level methods to consider when moving to the cloud. Yes that’s right, migrating to the public cloud is as easy as 1, 2, 3…

1. Optimize: The ‘Lift and Shift’ Approach [IaaS]

The chances are that your on-prem database is going to be one of the usual suspects: Oracle, SQL Server, Postgres, mySQL, DB/2 etc. And as probabilities have it, that database is more than likely running on Linux or Windows (if you are still running on big iron UNIX or – heaven help you – some kind of mainframe, you can leave now please) and there’s a fair chance you have a virtualization layer in there too. So just pick it up and wazz it into an Infrastructure-as-a-Service (IaaS) offering, will you? Quit fooling around reading this, you could have done it by now.

Welcome to lift and shift. You have now immediately realized the benefits of the cloud: all that on-prem hardware has been turned off and junked, the Capex bills have been replaced by monthly Opex costs to the cloud vendor and the DBAs have been rebadged as Site Reliability Engineers (SREs) and given new, cooler t-shirts to wear. Somebody give the CIO a bonus!

Of course, life is never this simple and there are inevitable pros and cons. IaaS still has to be managed by your own operations teams (which is expensive), databases licenses, where applicable, still have to be managed (and are expensive) and those cloud infrastructure bills are looking awfully large. Did somebody say the cloud was going to save money?

Performance is a problem too. The Dream Of The Cloud™️ is to have infinite scale of resources on demand, but your architecture was designed for on-prem and simply cannot take advantage of cloud scalability. When the DBAs (sorry, SREs) deployed this in the cloud, they had to choose between architecting for the average workload – which means performance sucks at peak times – or architecting for the max, which is way more expensive. Inevitably, the compromise fell on the side of cost and so the result is that application latency is high, user experience is low and nobody dares to run any analytical workloads for fear of taking the whole platform down.

Maybe there’s an alternative?

2. Modernize: Managed Database Services [PaaS]

Every cloud vendor has managed database offerings – in fact, most have a plethora of different offerings. Microsoft, for example, has Azure SQL Database as well as Cosmos DB. Oracle has the managed version of its eponymous database, Google has Cloud SQL and AWS has so many database services that there aren’t enough electrons on the internet to list them all. So why choose managed databases?

The Dream Of The Cloud™️ is to rid your business of all the low-level drudgery that comes with running IT infrastructure, so that your operations staff can rebadge themselves as DevOps and spend their time on more valuable activities, like drinking artisan coffees or breaking CI/CD pipelines. IaaS doesn’t really deliver on that dream, but PaaS gets a lot closer. Now, the cloud vendor takes care of much more drudgery and also – for licensable database products – manages the licenses so that you only pay for what you consume.

Of course, life is never this simple and there are inevitable pros and cons. Managed database services can be notoriously expensive if you need all the enterprise features you took for granted on-prem, while performance can often be problematic. Remember that managed SQL databases are designed for the average workload and not the peak, so if your system is an edge case in any way – if you’re not running with the pack – it won’t be a perfect fit. Maybe far from perfect.

Another potential issue is that many business-critical database applications are full of business logic. Think of Oracle database with PL/SQL packages written by developers long since retired. Can that be easily migrated into Managed Postgres on Azure, or Cloud SQL on GCP? Maybe the code calls UTL_FILE to write files which are then sent elsewhere using UTL_TCP. Try feeding that code into an automatic migration service.

Managed Databases are a great solution for the hundreds of boring databases you may have on-prem. Imagine never having to patch stuff again! But for anything even remotely unusual, or anything that regularly causes you pain, the chances are slim that PaaS will be the right fit.

Of course, there is another option…

3. Transform: Refactor to Cloud Native

Ahh, the path of the truly enlightened! Rip up your existing applications and rewrite everything to be cloud native. Fill out a bingo card with words like microservices, containers, serverless, Kubernetes and mutable infrastructure; tick them off one by one as your DevOps team write the whole application in Rust. Move to open source quasi-database platforms like Postgres, Cassandra and Elasticsearch.

Boom! You have now achieved The Dream Of The Cloud™️ which means that your application is truly distributed, scalable and has virtually no performance limits. I sound like I’m being sarcastic, but I’m really not – I have recently been working with customers who have built environments exactly like this and I could not be more impressed with the results (although these were “born-in-the-cloud” companies who were building new application stacks, rather than toiling with the technical debt of “legacy” on-prem apps). It’s the future.

But guess what? Life is never this simple and there are inevitable pros and cons. If you are starting with the baggage of an on-prem deployment which needs to be migrated, this is quite clearly the most complex and time-consuming option. It’s a proper migration project – and everybody in IT has a story about a long-term migration project which ran over time, over budget and ultimately didn’t deliver on its starting goals. Also, it may require specialist skills which your organisation doesn’t have. Do you really want to engage a team of consultants and pay them on a time and materials basis?

No matter which way you look at it, this option is the most expensive and carries the most risk.

So Which Option Is Best?

To state the obvious, there is no best option and everything has to be evaluated on a case by case basis. It makes sense to look at the anticipated lifetime of the application in question, because if it’s only going to be around for another couple of years, why expend the effort of rewriting anything? Just lift and shift, or use PaaS if possible. But most important of all, keep in mind that the options above don’t have to be mutually exclusive. It’s possible to lift and shift multiple applications to achieve an immediate goal of reducing your on-prem data centre footprint, then consider a smaller selection of those for further adaptation to PaaS. It’s also possible to move into the cloud using IaaS or PaaS while, at the same time, starting a longer-term project to refactor to cloud native.

In summary, there is no perfect journey to the cloud. The bigger and more complex the application/database, the more you’ll have to compromise on the expected result. But, after all, when was that not the case in Enterprise IT?

The Battle For Your Databases

There’s a battle going on right now between all of the public cloud vendors – a war in the clouds. And you might be surprised to hear what they are fighting over… They are fighting over you. Or, more specifically, your business-critical databases.

Everybody has something in the cloud these days. On a personal level, we are all keeping our photos, our music and our emails in the cloud. Corporations have followed suit: email, document collaboration and workflow, backups, websites… Almost everything is in the cloud. Almost.

The Big Scary Stuff That Nobody Wants To Move

Pretty much every company with an on-prem presence will have one or more relational databases underpinning their critical applications. Oracle Database, Microsoft SQL Server, PostgreSQL, DB/2 (the forgotten database of yesteryear: it’s still out there, but nobody likes to talk about it), MySQL… these products support mission critical applications like CRM, ERM, e-commerce, all those SAP modules that I can never remember the names of… And in each industry vertical, there are critical systems: healthcare has Electronic Patient Records, retail has its warehouse management platforms, finance has all manner of systems labelled Do Not Touch.

These workloads are the last bastion of on-prem, the final stand of the privately-managed data centre. And just like mainframes, on-prem may never completely die, but we should expect to see it fade away this decade. The challenge, though, is the inertia caused by such massive amounts of complexity and the associated risk of disturbing it. I have witnessed DBA teams who draw lots over which unfortunate will have to log on to “that database”, the one in the corner that nobody understands or wants to touch when it’s working ok. So how are they going to migrate that entire thing into AWS or Azure? Everybody knows a story about an eighteen-month migration project that overran budget by 1000% and then failed, right?

The View From The Clouds

So you may ask, if all this complex, gnarly stuff is full of risk, why do the hyperscalers want it? The answer is, because this is the biggest game left on the hunting ground. These vast technology stacks are the crown jewels of on-prem data estates. If you are Cloud Vendor A, there are some important reasons why you really want to capture this workload into your cloud:

Big applications and databases require a large recurring spend on premium cloud infrastructure
Customers are used to spending large amounts of money to run these services
The surrounding application ecosystem offers potential for the upsell of further cloud services (analytics, AI, business intelligence etc)
Once that workload comes into your cloud, it’s probably never leaving. In other words, it’s a long-term guaranteed revenue stream.

The last point is especially important: vendors use the term sticky to describe workloads like this. The effort of migrating all that sensitive, critical data and all that impenetrable business logic (written ten years ago by developers who have long since moved on) means you are never going to want to do this more than once. Once it’s in, it’s in.

A Massive Anchor

Working with one of the hyperscalers, I have heard these databases described as anchor workloads (credit: Kellyn Pot’vin Gorman) because they are what holds back the migration of large, juicy and complex environments into the public cloud. Like the biggest beast on the savannah, they are the hardest to take down… but a successful capture means everybody gets to eat until they are full.

So if this is you – if you are in fact a massive anchor – it’s probably worth keeping this in mind. Migrating your complex, challenging workload to the public cloud might seem like a mammoth task from your perspective, but to the hyperscalers you are the goose that lays the golden egg. And they can’t wait to get cracking.

Side note: I originally planned to call this post “Cloud Wars”, but I discovered that my former Oracle colleague, the inestimable Bob Evans, had beaten me to it…

How To Look Stupid (Part #612)

Now is the winter of our discontent. But rather than dwell on what a terrible year 2020 has been, I thought I’d make my final post of the year something more positive… so I am going to look back on one of the (many) times I made a fool of myself, in the hope that 2021 will give me the chance to do so again.

When Computers Go Bad

In the late 1990s, I was fresh out of university and working in my first job, for a small company (5 people!) at London’s Heathrow Airport, as a developer and database admin. We provided cargo handling software for all of the big airlines and freight companies. And on this particular day, “Dave”* at Air Canada had a problem with his system.

My company’s software managed the customs clearance of all inbound air freight for most of the airport. In order for inbound freight to leave the secure warehouses on a truck, this software (which, for Air Canada, ran on their main HPUX server) would send a message to the central HM Customs computer and then, upon receiving clearance, print out an official “air waybill” document. The waybill was legal proof that goods had clearance to leave the warehouse: no waybill = no clearance = no freight.

An hour ago, Dave had called in with a major problem: goods were being cleared by customs, but no waybills were bring printed. Air Canada now had a queue of lorries backed up at the warehouse and a crew that couldn’t do any work. There was nothing wrong with the printers, it was our software. Fix it, Dave begged us. Fix it now!

When DBAs Go Rogue

A senior colleague of mine, Denis**, was working on the problem and trying to test a fix on our lab system. He was also dialled in to Air Canada’s production system, on which our software ran – a crucial fact which turned out to be very important.

So when he called through to me from the server room to say, “Hey could you reboot the lab box?” I wondered over to his desktop and typed the magic reboot command on the first root window I found. Hey, one terminal session looks like another, right?

“Are you going to reboot it?” called Denis.

“I already have,” I yelled back, mildly irritated.

Denis stuck his head out of the door and stared at me, puzzled. I was then able to watch a whole range of emotions pass over his face: confusion changed to comprehension which in turn became outright horror.

I had just hard rebooted Air Canada’s entire UNIX platform with no warning to them at all.

Knowing When To Own Up

It took them a little while for Air Canada to realise what (or who) had happened to them. Remember, this was the 1990s, so big iron UNIX systems took about 15-30 mins to restart – and everybody was connected via dumb terminals which would have just suddenly gone blank.

Fred was a DBA until he accidentally truncated the wrong table

I mainly spent this time in purgatory, thinking about alternative careers, planning my new life in a Tibetan monastery or hoping for a natural disaster to divert attention.

But eventually, my desk phone rang and our receptionist said, “Dave from Air Canada wants to speak to you”.

I can vividly remember the dry mouth, my sweaty palms holding the phone, my voice about three octaves too high.

“Yes?” I stammered.

“I don’t know what you’ve done,” said Dave, “but all the waybills are coming out again now. Thanks very much!”

It’s important, I think, to be honest in these situations. But not that honest. So I let Dave get back to his busy job and made a mental note to confess to what had really happened some time within the next 25 years. And then I filed that next to the other mental note – the one about never, ever typing reboot without triple checking which system you are connected to.

Aspirations for 2021

When I look back at this story – and the many other times in my career when I made myself look stupid – I am grateful for the fact that things turned out ok. The whole year 2020 has felt like an elongated version of the purgatory I experienced above. But, as anybody who has ever rebooted a 1990’s-era big iron UNIX server will attest, the login window only appears about ten seconds after you’ve finally admitted to yourself that it’s never coming back.