How To Destroy Everything
We can rebuild it. We have the technology.
By definition, all technology is created while standing on the shoulders of giants, simply because building anything sufficiently advanced requires the use of existing tools. Software and apps are upgraded more than they’re installed, and progress usually leaves a wake of changelogs and patches, tiny attempts to turn whatever already exists into something a little better, ideally without causing a visible ripple on the surface. It’s a tricky balancing act — you must improve it without breaking it.
For the most part, at least. Drupal, a popular open-source software project used to build complex web sites, has adopted the unusual philosophy of ignoring backwards compatibility. It is always in the process of being totally rewritten. For a Drupal site, there’s no such thing as an upgrade — every milestone release of the application essentially requires a complete rewrite of all the sites built with it, and its users are conditioned to treat consistency between one version and the next as a happy accident, predictability that counterintuitively cannot actually be expected. This, the theory goes, allows Drupal to improve in dramatic leaps. All changes are seismic.
This is a bizarre approach to accomplishing anything, Groundhog Day with aspirations, so project lead Dries Buytaert and I convened for an interview about how he thinks about change in general. Here it is, edited for clarity, insofar as we ever have any of that in this life.
When was it that you decided to start developing Drupal in this way, and what prompted it?
I started Drupal because I wanted to build a website just for myself — like, this was an internet message board that I used to exchange messages with my friends at the dorm. We would talk about we want to grab dinner, we would share jokes, talk about stuff. Very quickly that message board evolved into an experimental platform, meaning I would add more features. This was at a time where I felt you could sort of wrap your arms around the web. I think there was maybe half a million websites or something in the world, and so what I would do is I would look at emerging trends, would implement them, add them to my message board, and that would force me to re-write my message board multiple times over to accommodate all these new things. The very first version of Drupal shipped with RSS. This was at a time when RSS was being invented. It wasn’t an official standard yet. I would see these things being developed by others and I would then just implement them in my platform. I had to rebuild it from scratch multiple times. I was so obsessed about getting the architecture and the psychology right that I couldn’t imagine preserving backwards compatibility, because in my mind that meant I had to drag along a lot of historical baggage. You have to keep all that code around. It has to be loaded to memory. It has to be parsed. I felt it didn’t make sense to keep layers of backwards compatibility, because I wanted Drupal to be pure. I want it to be perfect. I wanted it to be architected in the right way, and so I wanted to strip it from anything that wasn’t necessary. My mission in the early days was to keep it fast and small and clean and on the leading edge of technology. I focused completely and utterly on that — fewer lines, more elegant. The only way to do that was to break backwards compatibility.
You pointed at two different forces there. On the one hand, you have your own desire for architectural elegance within Drupal, but on the other hand you also have external forces like RSS. Those are not mutually exclusive, but they’re two different ways of looking at the same problem.
If we hadn’t re-invented ourselves many times over, a lot of innovations would just not have happened. Like if I didn’t make Drupal modular at some point — when we started, we didn’t have modules or plug ins or extensions. Introducing the module system was a rewrite in many ways, but without that we wouldn’t have a community of hundreds of thousands of people that are building modules. That would not have happened if we didn’t have an architecture that allowed that kind of participation and collaboration. Re-architecting enables us to embrace external market forces, but also enables us to innovate internally.
When you say “market forces,” obviously Drupal is part of the technology world and you want people using it because it’s good for the project, but that’s also in opposition to the terms you used earlier — I think you said you wanted Drupal to be “pure.” There’s probably a version of Drupal that would be easier to use, easier to set up, but wouldn’t be architecturally pure.
Yeah, the tensions are interesting. I like that Drupal is architecturally pure, but for me that may have been the biggest thing in early days, but today it’s more about making sure Drupal stays competitive and relevant in the bigger market. That means we have to evolve ourselves and we have to break things to stay relevant. Mobile didn’t exist when Drupal started. We had to flip things around, and today Drupal 8 is a mobile-first platform.
We’ve used the phrase “architecturally pure” without ever really explaining it. What does that mean to you?
There are many dimensions to the answer. It means finding the right balance, or trying to optimize different attributes from performance to maintainability of the code to the readability of the code, how easy it is for people to understand the software. You have all these things you want to optimize for. Usability is another one. Resilience is another one, I would say. It’s about making trade offs between these things. Some of it is maybe subjective, but other parts are also objective. To some extent it is following best practices, but in software engineering there is always an element which is more about style and elegance.
“Style” is an interesting word to use here. The way most people use it, there’s an implication that it’s frivolous, right? To an extent that could be said of code style: it’s a way of writing things that doesn’t necessarily impact the execution, but it does the impact the way the whole thing fits together. How does style fit into this approach to change? Is it the sort of thing where you’re able to change style freely, because the expectation is that people will follow along and change everything? Or does the style of the day change the way the software can iterate? With this pace of change, you are free to chase different styles in a way that most projects are not.
Sometimes we discover a more beautiful way of writing something — it’s simpler or it’s more light, I don’t know. It’s kind of hard to describe, but it just feels like a better way of doing it than we did it before. Then we apply that as a pattern across the rest of the code base. It’s actually one of the things that the Drupal community is really good at: if they see something that they like, a lot of people can help apply the style or the new pattern or whatever you want to call it. Sometimes that does break things too. We do change it, because otherwise the only changes you would make I guess are the ones that are absolutely necessary, right? We do more than what’s just absolutely necessary. It’s very subjective. In a way it’s like poetry. You can say “this is great poetry” or “this is bad poetry.” You can’t articulate that in rules. I’m not an expert in poetry, but there’s no rules that this poetry gets ten points, and this piece of poetry gets only two points, because it doesn’t match a specific formula or description or certain rules apply to it. People can gravitate towards it and a lots of people can agree this is beautiful poem, but that other thing is not a beautiful poem. It’s based on a feeling. That same kind of thing exists in code. Sometimes you write an algorithm and you say, “Wow, it’s beautiful.”
There’s this notion of expressivity in code, the degree to which you have to abstract the thought in your head in order to implement it, how many changes does the initial thought have to go through? It’s not necessarily in opposition to other goals, but it is a different goal. Let’s say there’s something that would be very expressive, but it’s less optimized — how do you approach making a decision like that?
I guess it’s a trade-off, and there’s no right or wrong. Some of it is data driven, like we will benchmark things, but a big part is just guts, what we feel makes the most sense given all of the different dimensions.
There’s a very large community of people and contributors. Is there an extent to which this kind of iteration can’t be done by consensus? Are there limits to how much of a consensus you can look for?
Yeah, we have some guidelines, I guess. For example, security is always going to be more important than pretty much anything else, right? We won’t make trade-offs there.
That’s the first thing you’ve mentioned that basically pre-empts everything else. With every release you’re zigging and zagging and you’re able to totally change everything, and you have this totally weird trajectory through software and engineering and architecture. There’s this philosophy of change, but at the same time you’ve got something like security, which is like a beacon in the distance, and you always have to fly towards that. Is there anything else that has that sort of role?
There’s two dimensions. One dimension I think would be… let’s call it “attributes.” Security trumps scalability and scalability trumps performance, these things typically trump architectural purity. You can kind of order these things, you can prioritize. Then the second dimension, we have content creators, less technical people, people that manage the content, we have developers, people that build modules and extensions to Drupal, and we have HTML designers. You have to make trade offs sometimes, because sometimes when you make something better for one user, you make it worse for another user.
It sounds like you’re sort of explaining striking a balance between hierarchical values and non-hierarchical values: security always has to be at the top, but when weighing two different types of users, they’re both human beings whose feelings matter. How do you manage the relationship between those two types of concerns, and when there’s a new concern that appears, how do you determine whether it’s a hierarchical concern or a non-hierarchical concern?
Sometimes there’s no question about it whatsoever. Let’s say you go to a restaurant and you order fish. You don’t care which truck brought the fish to the restaurant, do you? Sometimes it’s as obvious as that. You have to do what’s right for the operator, the fish is fresh and what have you, but we don’t care what truck gets the fish to the restaurant. Everybody understands that, even the truck driver. Maybe it’s not a great analogy.
How do you approach organizing the things you want to get done, your own goals and plans? The analogy that springs to mind here is the difference between a shopping list when you’re going to the grocery store versus a bucket list of things you want to accomplish before you die. These are both lists of things to do.
Everybody can do what they want to do and they can bring things to the table without any kind of top-down planning or that kind of stuff. It’s almost Darwinian. We see that often in Drupal, where we have maybe five different ways to do an image gallery for our websites, and people build these things and then compete. Eventually what tends to happen is either it makes sense for there to be competing solutions or it doesn’t, and then one of them kind of dies. Survival of the fittest. Actually starting with mutation, if you will, where there can be multiple different versions of something, variants or mutations or what have you. Over time either multiple species survive or they don’t, and the best solution will win. It’s a radically different way of innovating, incubating innovation, that is unique to open source, because a commercial organization can’t say “let’s go build five different ways to build image galleries, spin out five different teams and go out and build competing solutions.” They’re resource-constrained in a way that open source isn’t. People think of it typically as inefficient, wasteful, because why would you do things five times? At the same time, it’s a proven model that works. We’ve been able to out innovate our proprietary competitors because we have many more people and can be a little bit more wasteful.
These are two different things, the absorption of new code into Drupal via this Darwinian system, versus kind of the expulsion of old code from Drupal via the catastrophic approach to change. Are things still Darwinian inside Drupal? That implies a certain amount of chaos, honestly — little components fighting or whatever. That’s not necessarily a way to architectural purity.
To some extent, yes. First of all, I don’t think they’re exclusive. When I removed old code from Drupal, it was often because I wanted to build something that’s more flexible, and basically enabled the Darwinian nature of Drupal. We built a platform that allows the collaboration of 32,000 people. That took actually removing code from Drupal, and changing and breaking things. I think they’re very aligned in a way. You enable the absorption of new code by getting rid of old code and by having a willingness to break things. When people can do things many different ways it creates a little bit of chaos, sure, and we try to organize some of that chaos by having guidelines and best practices, but at the same time, the reason that we’ve been successful is because we have never micromanaged people. Nature tells you which bird survives better. What will survive is the stuff that gets adopted.
As the guy who’s responsible for this project and most associated with it, does the thought of just royally fucking up a Drupal release keep you up at night? Do you ever worry about having this widely reviled release, the Jar Jar Binks of Drupal releases, that provokes a backlash, and having that take a personal toll?
No. It honestly doesn’t worry me, because the way we develop things is in complete transparency. We don’t have a secret roadmap or anything. Our process is so transparent that there’s so much validation along the way. It’s not black box development where after a year or two or maybe longer something comes out of the oven and people see it. No, people have been involved from day one, and they’ve been able to share an opinion. It doesn’t mean we can’t get it wrong, but I definitely feel it’s a shared responsibility. The chance of all of us getting it wrong seems very low. While we radically change things, we make big changes, but we try and not to completely break Drupal. In a way it’s like we’re flying a plane, and we’re rebuilding the plane in flight. We don’t land the plane, take it apart and then put it back together, we’re really rebuilding the plane without landing. It’s very rarely that something is malfunctioning to the point that plane crashes; that just wouldn’t work, so we have to make big changes incrementally.
