What Makes a Great Database Engineer?

I recently wrote a job description for a Staff Database Engineer. Sharing the job description with a co-worker for review, his feedback was, “Database engineers with this level of experience drive Saabs.”
Now, all I know about Saabs is that they went out of business. At first I thought, “Is he saying that these people are nostalgic for Swedish made cars? That can’t be right.”
I asked for an explanation and he told me, “Saabs are safe.”
And there we have it; if I had to name the most important quality of a database engineer, it would be their proclivity toward Safety.
In my experience, the most effective database engineers do drive Saabs. They check their mirrors, tire pressure, and lights before every drive. They also require that everyone wear seatbelts and keep their hands at 10 and 2.
At times, such overzealous caution can wear thin on one’s patience. After all, we have a lot to do! We can’t do a 10 point check before every database change, right?
Wrong. We can, we should, we must.
“Brad, isn’t this time consuming?” — I’ll get to that in a minute.
Why do we have to do all these checks? Because the data for our applications is akin to car passengers. We love our data. We want to know our data will be safe. Airbags, seatbelts, proper tire pressure— these are the bare minimum.
Ok, that’s enough about cars. What does it actually mean for a database engineer to be “safe”? Thankfully, there are a myriad of ways to ensure safe database practices. Here are a few questions which I believe are essential to database operations:
Is it safe? How long is the operation expected to take? Has this operation been tested in a production-like environment? What happens if we have to kill it while it’s running? What is our rollback plan?
Safety can be time-consuming, which brings me to the second most important quality of a great database engineer: Automation.
At scale, and in a fast paced environment, we can’t simply have database engineers who sit in the basement and run manual operations when they’re needed, babysitting them along the way to ensure the system stays operational.
A great database engineer has stellar software development skills, with the ability to automate away the historically manual nature of running production databases. Virtually every aspect of our database infrastructure and interaction should be automated.
Another way of saying it: It isn’t enough to have someone inspect our car when we want it to be safe, we need someone who can take that Saab-esque safety and automate it throughout our entire platform. This engineer is not just a Saab driver, they’re a Saab engineer. They can build a keychain that scans your car for all those safety checks when you approach it, flashing red warning lights if there is any remote danger.
Not only can they build such tools, they require it. Which brings me to the third and final quality I will cover for great database engineers: Dedication.
It takes extreme dedication to operational excellence to be a great database engineer. In virtually every organization I have worked in, DBAs and DBEs have the extreme weight of every software engineering team’s deliverables on their shoulders.
In a fast-paced and high-growth environment, that often means being asked to approve database changes that don’t meet (or haven’t yet met) the safety criteria— e.g. “We have this database change that has to go to production at 6pm today, can you take a look?”
It isn’t that other engineers are bad, it’s just that engineering orgs usually don’t adopt more rigorous operational best practices until years after they have started their explosive growth.
It can be very difficult to advocate for caution when a VP, Director, Product Manager, and 6 Engineers are all saying, “We already bought a billboard and our customers expect it. We simply have to push this database change.”
Great database engineers know how to gracefully navigate these situations. Sometimes an engineer may adequately communicate risks as an SME and negotiate a delay until proper testing can occur, or rally together everyone and formulate a last minute plan to ensure the database change passes safety checks before the deadline. Regardless of the method, it most certainly requires good storytelling skills.
