Evolution of the DBA

In the previous post, I looked at Gartner’s recent assertion that 75% of databases will be deployed to the cloud by 2022 – and that the cloud is now the default platform for managing data.

The massive shift to the public cloud has a lot of implications, many of which have been written about at length over the last few years. But one question I don’t think has been asked enough is: what does this mean for the poor, beleaguered database administrator? Let’s start with a look at the journey DBAs have been since “the old days”.

DBA 1.0: The (Good) Old Days

Data centres used to contain four distinct tribes of beings living in semi-peaceful co-existence: SysAdmins, DBAs, Network Admins and Storage Admins: Four groups of specialists, each with a distinct skillset and a fairly delineated boundary of responsibility. I say four, it was really three – as everybody who remembers this era will attest, Network Admins were actually mythical creatures who never inhabited their desks; historical evidence now suggests that they were actually just a simple script which automatically closed any ticket with the phrase “No problems were found with the network”.

The database administrator occupied a unique position in this family, because they lived further up in the application stack and so dealt with developers and application owners, business users and sometimes – whisper it – those wondrous beings, the “end users”. Conveniently, this made the DBA the perfect person to blame for almost any problem at any layer in the stack. Application slow? Must be a database problem. Query taking too long? MUST be a database problem. Never mind that the database server doesn’t have enough memory the developers have no concept of how to code in SQL and the storage system is a RAID5 bag of spanners running on spinning rust… it’s always a database problem. And we know it’s not a networking problem because it says here that “No problems were found with the network”.

One outcome of this “unique” position was that many DBAs had to learn skills outside of their core profession (networking, Linux or Windows admin skills, SQL tuning, PL/SQL decoding, hostage negotiation etc). I’d love to say this thirst for knowledge was due to professional pride, but the best DBAs I ever met simply learned these skills so they could prove they weren’t in the wrong and thus get an easier life. “Oh you think your SQL runs slow because of my database huh? Well if you rewrote it like this, it runs in 10% of the time and doesn’t make all the lights go dim in the data centre, you imbecile…”

DBA 2.0: The IT Generalist

As the data centre evolved and new technologies such as Virtualization, NoSQL, Hadoop and the Cloud became prevalent, the clearly defined roles of yesteryear started to become blurred. In the last decade, we saw the rise of a new creature in the data centre: The IT Generalist. Of course, this is mainly just another way of saying DBA with Extra Responsibilities (but no extra pay). It is now commonplace for DBAs to be managing a multitude of different technologies outside of the traditional RDBMS: many DBAs are managing, at least at some level, VMware clusters or other virtualization platforms; I know DBAs who have had tangles with firewalls and software-defined networking… I have even met a large number of DBAs who admin their All-Flash storage arrays (simpler than the old fashioned disk array, after all).

As a side note, anyone with the job title of “Oracle DBA” also found themselves lumbered with managing any technology which was Oracle-badged – and that’s a lot of stuff. Fusion Middleware, Oracle Linux, Weblogic, Oracle ZFS Appliance, anything running under Automatic Storage Management, even Java! The list goes on… how long before somebody gets a ticket because Tik Tok isn’t working properly?

Larry Ellison might have famously said he wants to get rid of the DBA, but the reality is that the DBA role has just become even more wide-ranging.

DBA 3.0: The Cloud DevOps DBA

Fast forward to 2020, the DBA is now managing applications running on databases which run in containers on virtual machines in the cloud, probably deployed via some sort of infrastructure-as-code implementation. Hey, the dream of the modern IT organisation is to achieve some utopian level of automation – and it’s the DBA who has the most practice of automating cross-function tasks; they’ve been trying to do it for years just for an easier life. (Note how the dream of “an easier life” motivates so much of DBA behaviour!)

Of course, everything is now DevOps too… right? If you aren’t DevOps, you aren’t in the gang. Remember when everything had to be agile? But, when you scratched the surface, “agile” was just a way of saying “we haven’t documented any of this”. Well, DevOps has taken over from agile as the buzz word of choice. And the literal translation of “DevOps” is “we still didn’t document anything but also we aren’t going to follow any kind of change control procedures or put any of these code releases through anything more than the most primitive of testing routines, so good luck”.

But in this long evolutionary journey, there is one thing that DBAs have never been exposed to … until today. Cost. As a DBA, you may have had to argue for more powerful servers, faster CPUs, more database processor licenses, cost options (“I need the Tuning Pack, damnit!”), but the cloud is a different ball game. A DBA building a database in the public cloud is making decisions which have a direct affect on the (quite possibly massive) monthly bill from AWS / Azure / GCP / Oracle Cloud / other vendor of choice. This is what I wanted to look at in this post before I got massively carried away.

DBAs of the World, Unite!

I’ll be honest, I didn’t intend this post to become some sort of DBA Manifesto, but once I started typing I couldn’t stop. Blogging is like that sometimes. In the next post, we’ll delve a bit deeper into the future of DBAs and angle on the cloud costs. In the meantime, let’s summarise:

Everybody knows that the DBA is the humble, hard-working hero of Enterprise IT: dedicated and underpaid, overburdened and undertrained, blamed for everything and thanked for nothing… the DBA really is the Morlock of the data centre, working long nights and hard weekends to keep all those wonderful, spoilt ~~Eloi~~ end users happy*. If you are a DBA, give yourself a pat on the back for surviving this evolutionary journey. If you’re a SysAdmin, be honest: you guys need to buy your DBAs a drink now and then. And if you are a Network Admin: stick to the script.

* If the Morlock and Eloi references aren’t working for you, read this.

Databases Now Live In The Cloud

I recently stumbled across a tech news post which surprised me so much I nearly dropped my mojito. The headline of this article screamed:

Gartner Says the Future of the Database Market Is the Cloud

Now I know what you are thinking… the first two words probably put your cynicism antenna into overdrive. And as for the rest, well duh! You could make a case for any headline which reads “The Future of ____________ is the Cloud”. Databases, Artificial Intelligence, Retail, I.T., video streaming, the global economy… But stick with me, because it gets more interesting:

On-Premises DBMS Revenue Continues to Decrease as DBMS Market Shifts to the Cloud

Yeah, not yet. That’s just a predictable sub-heading, I admit. But now we get to the meat of the article – and it’s the very first sentence which turns everything upside down:

By 2022, 75% of all databases will be deployed or migrated to a cloud platform, with only 5% ever considered for repatriation to on-premises, according to Gartner, Inc.

Boom! By the year 2022, 75% of all databases will be in the cloud! Even with the cloud so ubiquitous these days, that number caused me some surprise.

Also, I have so many questions about this:

Does “a cloud platform” mean the public cloud? One would assume so but the word “public” doesn’t appear anywhere in the article.
Does “all databases” include RDBMS, NoSQL, key-value stores, what? Does it include Microsoft Access?
Is the “75%” measured by the number of individual databases, by capacity, by cost, by the number of instances or by the number of down-trodden DBAs who are trying to survive yet another monumental shift in their roles?
How do databases perform in the public cloud?

Now, I’m writing this in mid-2020, in the middle of the global COVID19 pandemic. The article, which is a year old and so pre-COVID19, makes the prediction that this will come true within the next two years. It doesn’t allow for the possibility of a total meltdown of society or the likelihood that the human race will be replaced by Amazon robots within that timeframe. But, on the assumption that we aren’t all eating out of trash cans by then, I think the four questions above need to be addressed.

Questions 1, 2 and 3 appear to be the domain of the authors of this Gartner report. But question 4 opens up a whole new area for investigation – and that will be the topic of this next set of blogs. But let’s finish reading the Gartner notes first, because there’s more:

“Cloud is now the default platform for managing data”

One of the report’s authors, long-serving and influential Gartner analyst Merv Adrian, wrote an accompanying blog post in which he makes the assertion that “cloud is now the default platform for managing data”.

And just to make sure nobody misunderstands the strength of this claim, he follows it up with the following, even stronger, remark:

On-premises is the past, and only legacy compatibility or special requirements should keep you there.

Now, there will be people who read this who immediately dismiss it as either obvious (“we’re already in the cloud”) or gross exaggeration (“we aren’t leaving our data centre anytime soon”) – such is the fate of the analyst. But I think this is pretty big. Perhaps the biggest shift of the last few decades, in fact.

Why This Is A Big Deal

The move from mainframes to client/server put more power in the hands of the end users; the move to mobile devices freed us from the constraints of physical locations; the move to virtualization released us from the costs and constraints of big iron; but the move to the cloud is something which carries far greater consequences.

After all, the cloud offers many well-known benefits: almost infinite scalability and flexibility, immunity to geographical constraints, costs which are based on usage (instead of up-front capital expenditure), and a massive ecosystem of prebuilt platforms and services.

And all you have to give up in return is complete control of your data.

Oh and maybe also the predictability of your I.T. costs – remember in the old days of cell phones, when you never exactly knew what your bill would look like at the end of the month? Yeah, like that, but with more zeroes on the end.

Over to Merv to provide the final summary (emphasis is mine):

The message in our research is simple – on-premises is the new legacy. Cloud is the future. All organizations, big and small, will be using the cloud in increasing amounts. While it is still possible and probable that larger organizations will maintain on-premises systems, increasingly these will be hybrid in nature, supporting both cloud and on-premises.

The two questions I’m going to be asking next are:

What does this shift to the cloud mean for the unrecognised but true hero of the data center, the DBA?
If we are going to be building or migrating all of our databases to the cloud, how do we address the ever-critical question of database performance?

Link to Source Article from Gartner

Link to Merv Adrian blog post

Don’t Call It A Comeback

Ok, look. I know what I said before: I retired the jersey. But like all of the best superheroes, I’ve been forced to come out of retirement and face a fresh challenge… maybe my biggest challenge yet.

Back in 2012, I started this blog at the dawn of a new technology in the data centre: flash memory, also known as solid state storage. My aim was to fight ignorance and misinformation by shining the light of truth upon the facts of storage. Yes, I just used the phrase “the light of truth”, get over it, this is serious. Over five years and more than 200 blog posts, I oversaw the emergence of flash as the dominant storage technology for tier one workloads (basically, databases plus other less interesting stuff). I’m not claiming 100% of the credit here, other people clearly contributed, but it’s fair to say* that without me you would all still be using hard disk drives and putting up with >10ms latencies. Thus I retired to my beach house, secure in the knowledge that my legend was cemented into history.

But then, one day, everything changed…

Everybody knows that Information Technology moves in phases, waves and cycles. Mainframes, client/server, three-tier architectures, virtualization, NoSQL, cloud… every technology seems to get its moment in the sun…. much like me recently, relaxing by the pool with a well-earned mojito. And it just so happened that on this particular day, while waiting for a refill, I stumbled across a tech news article which planted the seed of a new idea… a new vision of the future… a new mission for the old avenger.

It’s time to pull on the costume and give the world the superhero it needs, not the superhero it wants…

* It’s actually not fair to say that at all, but it’s been a while since I last blogged so I have a lot of hyperbole to get off my chest.