Hillary Clinton and the complex system

Most of my friends are voting for Bernie Sanders. His rhetoric reminds me strongly of Obama’s in the lead up to the 2008 election — we’re hearing the things we’ve been saying to each other all along spoken from the mouth of someone with worldwide attention. It’s intoxicating.

I hear two main criticisms of Hillary. The first, and most unfortunate, is the subtle and constant dog-whistle of sexist word choice to disregard her competence as a leader and bureaucrat. This is when someone claims she’s unlikeable or heartless or somehow doesn’t feel the human emotions the rest of us do. It’s a quiet suggestion that she’s less maternal than she should be. I’m horrified to hear my forward-thinking, social justice-aware cohort laugh politely when she’s called a ‘hag’ or allow put-downs that would never be applied to a man.

The patriarchal criticisms involving gender expectations are only half of the argument against her. The other is that she’s corrupt as a by-product of working within the existing system and because she chooses her platform and her words as a means to effect her own plans rather than from some authentic political purism. The sexist critique is unacceptable but few people dispute her pragmatism — Hillary is willing to make small changes to an existing, imperfect system while working in a large moral gray area. And this quality is precisely why I’m voting for her.

I’ve been building software systems for a dozen or so years now and each year the projects become more complex. When I was a junior engineer I put my effort into helping clients create new projects from scratch or rewrite existing ones in the hot new technology of the day. Most of the projects were small enough I could fit the whole thing in my head and reason about the entire project. Later in my career that kind of work is increasingly rare because I’m working with teams of tens or hundreds of other engineers on projects that have been incrementally built over years. One might think it’s the same fundamental work — a house painter paints the same way whether they’re working on a doghouse or a mansion, right? — but changing a complex, existing system is a separate discipline to changing something small.

The first few years of my career this is how I approached my work: I’d build something good from scratch or if there was already a system in place I’d replace the bad parts with good parts. Tautologically, this seems like a good approach. This is roughly how I’d like to effect governmental change and it’s why Bernie’s platform resonates so strongly with me. It’s pretty hard to argue against the logic, actually: Here’s the whole system that exists, here’s the bad stuff, let’s get rid of that and replace it with something better. And while we’re there let’s add more good stuff we’re going to want.

This approach works perfectly on very small systems but fails rather spectacularly on large ones. And by “small” I mean that you must be able to obliterate the whole thing and rebuild it by yourself and the new system fully satisfies whatever needs the original system satisfied. As an example, consider if you have a slow, ugly spell-checker program that you’re tired of working with. Somebody comes along and rearchitects it so it’s beautiful, uses the latest programming idioms, and runs twice as fast but doesn’t catch as many grammar mistakes. This upgrade is actually a downgrade. The next person to fix it will see that it doesn’t fix bad grammar properly, they will rearchitect it again and they, too, are going to fail in some brand new way.

There are some small systems that actually are manageable enough to be replaced entirely but we humans have a tendency to tightly couple little things together such that changing any one of them means changing a bunch of them. Which means we don’t have small, contained system — we actually have a single large system made up of many conjoined parts.

The U.S. government is one of the best examples of this. We have three separate branches of the federal government but as I type this the election funding laws set by congress are preventing the Republican members of the Senate from nominating a new Supreme Court judge nominated by the executive branch. The tight interplay between these three pieces — some of the most explicitly separate pieces of our governmental system — has resulted in a temporary deadlock that’s preventing the whole federal government from doing its work.

The junior software engineer approach to this problem would be to either fix one of the pieces as if it were isolated (e.g. making a new rule that court nominees must be voted on in a timely manner) or to rearchitect the whole system and install a pristine new democratic socialist government. Until a few years ago this would have made perfect sense to me.

But in 2013 and 2014 I had the questionable honor of owning one of the world’s largest MySQL databases. MySQL is an open source data store that Facebook and Google and thousands of other organizations use to store information and process transactions. A normal database would hold a maximum of about 200 gigabytes and would perform about 100–200 queries per second. So a hundred people every second could retrieve a list of something interesting. That’s something at a size where if I wanted to totally replace it with something new I could do it. I could move it to the cloud, I could swap out the underlying software, or I could add or remove responsibilities to it and I’m reasonably confident that I’d be successful.

But the database I owned was not at all normal because it held 50x as much data (close to ten terabytes) and served almost 1,000x as many requests every second. To support this we bought a special hard drive that could read and write data so fast it was like writing it to your computer’s memory. This single hard drive cost over ten thousand dollars. Due to the massive size of the dataset we needed extra servers to serve as read-only machines with copies of the data so instead of one hard drive we had fourteen. But we needed redundancy of that whole set in case the datacenter that database lived in lost power — so we bought another fourteen of those drives and ran them in a datacenter on the other side of the country.

When I inherited this legacy system I first engaged the same reflexes I’d used earlier in my career: Find the bad parts and replace them with good parts. Or, if the whole system is bad then replace the whole thing with a good one. But I wasn’t fixing a little thing that somebody had built in a hurry — this was a large system built by hundreds of people over five years years that supported millions of small businesses all day. Making any mistake in changing this system would be cataclysmic.

The only way to fix a complex system is to get a large number of people who have intimate knowledge of it to slowly change it in the same direction. In my case this meant a dozen senior engineers working full time for two years carefully excising parts of this massive database into other places and performing core hygiene improvements on the system so that future excisions could be made even easier. As of the time I write this that old database is still doing most of what it used to do but with a lot less data, a lot less responsibility for critical issues, and is organized such that nobody will have to worry about it for several years.

I was a Bernie Sanders supporter until I thought about what his approach would be to the hairiest problems I’ve faced in my career. I love his rhetoric and I want the world that he imagines to become real but I think he believes that one can just replace the bad with good. I can easily imagine him proposing changes that address the worst parts of our governmental system but that have secondary or tertiary effects he doesn’t control for that make the system as a whole less effective. Should we fix the role of banks in the economy? Absolutely. Should we do it by limiting their size or limiting the salary and bonuses their employees can receive or by making an even stronger version of Glass-Steagall? I’m not sure, but I know the solution isn’t simple and I don’t trust any leader who suggests it is.

I don’t find Hillary Clinton very likable. She doesn’t seem to be interested in being my friend or winning my heart. She does seem interested in winning my vote though and she’s earning it by demonstrating a sophisticated knowledge of the United States government. Not the ideal one that exists only in civics textbooks and in the minds of historical reenactors but the actual, living system made up of millions of little parts that meets the needs of hundreds of millions of people each day. You can’t just replace the bad with the good in our government. Let’s say you want to cut the cost of the military in half. Brilliant, let’s do that. That would fix so many problems and we could spend it on education and even create a federal, fully-funded pre-Kindergarten childcare program. But you can’t just tell the joint chiefs that you’re giving them half the money next year. Even if you had total discretionary power over the budget (which the president does not) you’d still have to go look at where the money is going and decide if you’d rather spend less maintaining the Air Force, less paying veterans’ healthcare bills, less on multi-year research programs, or some other line items. You can’t just fire every other employee in the military or cut all the (very low) salaries in half and expect the system to keep working.

I don’t expect Hillary Clinton and I will ever meet or, if we were to, I don’t think she’d give a damn about me or my values or my opinions. And I don’t care because she was our Secretary of State, a U.S. Senator, she’s already lived in the White House for eight years, and I know of no other leader with such a depth of knowledge about the incredibly complex system that we’ve built to support our country.

I wasn’t the first person to own that massive database I mentioned earlier. Everyone before me tried the Bernie approach to fixing it and failed so poorly that not only was the system still unfixed but a narrative developed that it could not be fixed. And the first two times I tried to fix it I failed somewhat dramatically. Each attempt failed to take into account that as bad as all of the bad parts were the thing mostly worked most of the time and we couldn’t afford to have it stop doing what it did even for a moment. In the end the real solution was slower, more expensive, and less satisfying — it was just lots of tiny, measured steps in the same direction performed by many people over a long time. And this is what I believe Hillary is uniquely able to do.

Hillary has won the endorsements of seemingly every group that can endorse a presidential candidate. Congresspeople, mayors, governors, unions, state senators, diplomats, industry execs, lobbyists — they’re all behind her. My millenial friends see this as corruption (particularly that she has such support of lobbyists) but I see something else. Because if she can get empowered people moving in the same direction she can make real improvements to this complex system we all accidentally built together.