This Company Went From Shipping Twice a Year to Once a Day

And they did it in less than two years. Jorge Varona explains how.

When Jorge Varona started as the head of engineering for Ipreo’s Private Capital Markets division in September 2016, the development team was deploying once every five months. Now the team, which produces an application for data-driven investment portfolio monitoring, is deploying daily. We called him at his office in New York City to ask him how he did it.

Tell me what you inherited when you walked in the door at Ipreo.
It was a monolithic application, with 7 million lines of code, which is a lot of code for something that’s only been around for eight years. There was 3 million lines of backend code, 2-point-something million lines of java script. It required eight hours of downtime on a Saturday, twice a year, to deploy it. The entire process around it was very waterfall-y. There wasn’t really a notion of product development or product owners. There were one or two product people and an army of business analysts writing a lot of documentation. In terms of the infrastructure, we were deployed to Rackspace, on a single database server with multiple database instances. We’d gone about as far as you could go with vertical scaling.

Did you know all of that before you took the job?
I didn’t know how many lines of code, but I knew the broad strokes.

What attracted you to the job?
Just the idea that, “Holy crap, these guys have a lot of challenges — some that they don’t even know they have. But they’re easy to fix.” There’s no secret sauce. You just need to convince people that it can be done, and then you need to convince people that they need to invest time to do it. That’s it. These were not hard challenges.

It can be difficult to sell the business side on the changes necessary to implement DevOps. So what was your pitch?
Believe it or not, it wasn’t a hard sell. My boss is somewhat technical, so he understands there’s a reinvestment you have to make in software development and technology in general. I think the conversation started with, “We need to go faster. We need to scale this team out.” He didn’t believe it would scale in its current form, and I confirmed that it wouldn’t scale the way they were planning to do it because there were choke points. So the sale became, “I can get us to go faster and mitigate risk by just releasing more frequently.” From a sales perspective, when a client says, “I’d like to see this feature,” you can tell them, “It’s a five-month window.” But if for some reason you miss that five-month window, it’s really hard to go back and say, “You need to wait another five months.”

What about the conversations with your team?
That was much harder. The application was largely built by a third-party contractor. They’re a long-term third-party contractor. My staff right now, of 85 people, 70 of them are contractors. Almost all of them are based in Russia, spread across two locations. You’ve got cultural differences, so it’s hard to explain. You’ve got technology stack biases. Not only was everything built on a Microsoft stack, but it was an old school Microsoft stack. So if you tell them, “We need to move to Git because we’re doing feature branching,” they don’t know how that works. You describe it by asking, “What allows you to do a hot fix very quickly?” You get them thinking, “Well, we’ve isolated the change, we know exactly what’s happened, we can make decisions about the level of testing, because the test surface is much smaller.” So you convince them that everything needs to go hot fix.

How long was it before they came around?
They’re still not all bought in yet. It’s like 75–25. I didn’t get to the 50 percent threshold until the end of last year. This whole project was supposed to be completed by the end of last year, and we just moved everybody over last week. The process is a perfect example of Conway’s Law.

How so?
When I started, the application was a big ball of mud, built by a big ball of developers. So our experiment was to try a reverse Conway. Even though we had a really large, one-big-deployment solution, we split the teams into ten different teams and told them they owned certain areas and told them to start carving their code out and making sure they have independent, deployable pipelines. That took a year, because you still have a product to put out. You can’t just stop conducting business during development. You have to continue to develop and carve out time to make these changes — basically pay down your technical debt. So we spent most of 2017 splitting things apart so that people were less scared about deploying their pieces of code. And now the application is about 25 different deployable applications, all talking together.

It’s interesting that you got instant buy-in from the business side but ran into resistance from your developers; usually it’s the other way around. How did you deal with that?
I’ve actually done this a couple times, so every time someone gave me a what if, I had a really quick response for it. I can respond to the technical questions really quickly. The business questions are always a balancing act. I was fortunate that the business was ready for change and understood the value proposition more quickly. I understand how fortunate I am to have that. It’s a great reason to be here.

Tell me about a point when you felt like you’d gotten over the hump or you’d picked up enough momentum to feel good about the transition.
I still don’t feel 100 percent about it. Though we should be further along than we currently are, I’m confident that we will get there.

So what’s next? What’s the next long pole?
Deployment pipelines are still about an hour. Actually, it’s an hour to four hours, by the time you commit code and all of the tests run and an environment is created. We need to be much faster than that. Part of that is our allocation of environments, part of that is that the gauntlet it runs through isn’t really optimized. So there’s some pipeline optimization and some process changes that need to happen. The mandate was that there will be no more manual testers in this organization by the end of the year. I’m not going to have an army of people to do repetitive tests. Robots do that. I need people to write robots.

Want to ship software faster to production? With its weeklong Navigator program, sodo will bring technical subject matter experts to you to help you identify your team’s skills and set goals for a software delivery transformation. https://l.shipondayone.com/nav.html