Software Origins of DoD Software Factories with Enrique Oti
Listen on iTunes, Spotify, or your favorite podcast app.
On this episode of Defense Unicorns, Enrique Oti joins Rob to discuss the origins of Department of Defense Software Factories and his strategy of innovation used to build Kessel Run, using a strategy opposite to most innovators, instead of keeping their heads low and unseen, they made noise, told as many leaders as possible and eventually went all the way to the top for support. Enrique Oti is a retired Air Force Colonel of 23 years and the founding director of Kessel Run, an acquisition and operations unit that rapidly deploys modern software capabilities for Air Force programs, as well as completing many operations in Cyber Warfare and intelligence. He also co-founded the Defense Innovation Unit (DIU) in Silicon Valley.
Defense Unicorns, a Podcast, is hosted by Robert Slaughter, Founder and CEO of Defense Unicorns.
Guests this week include:
Enrique Otis, retired Air Force Colonel of 23 years, founding director of Kessel Run, and CTO at Second Front System
Highlights from Defense Unicorns, a Podcast:
This interview has been lightly edited for length and clarity.
(00:23) What is Kessel Run and how it started
(2:56) Choosing military members
(12:31) Building talent
(18:55) Navigating Risk
(30:06) Overcoming low points
(36:08) Second Front
(40:20) Advice for the listeners
ROB: One of the first questions I have for you is, Can you describe for the audience what Kessel Run is?
ENRIQUE: Kessel Run is the term being used by the Department of Defense for a software factory. I don’t like to think of it as a software factory, but really, Kessel Run is an organization. It’s a military unit that actually builds, deploys, and runs software on behalf of the Air Force. When we stood it up, it had a very specific mission: to support air operations centers. So it’s the idea that, rather than just building software over a period of years to point it at somebody in the field, you actually use modern, agile methodologies and modern DevOps principles, where the team that’s building the software is continuously iterating with the users, deploying it to the field, and running it along with the user. So that was Kessel Run. I think right now, it’s probably 1200–1300 people, a mixture of military, civilian, and contractors building a wide variety of software products.
Before Kessel Run, when we actually started the program, I was working at the Defense Innovation Unit Experimental (DIUX) out in Silicon Valley. I helped co-found DIUX back in the 2015 timeframe; I ran the Air Force team out there. What eventually became Kessel Run was actually just one of many little products we tried to see what we could do differently. Our innovation in the early days of DIUX wasn’t terribly focused; we were trying a lot of different things and conducting a lot of different experiments. What turned into Kessel Run was one of those early experiments that just happened to be successful.
Prior to being called Kessel Run, the project didn’t really have a name. Internally, we called it MDOC, which stands for Multi Domain Operations Center. We chose that term because we thought it would sell pretty well in the building because that was the term that was in the Air Force future operating concept that had come out earlier, I think in the 2015 timeframe. And we’re like, Ooh, there’s this whole MDOC concept of a future operating center; why don’t we try to build one of those? So, internally, at DIUX, we called it MDOC, but our first product name was Jigsaw. I think a lot of people know us from that first product that we rolled out, Jigsaw.
ROB: One of the things that I love about the military software factories is the inclusion of active-duty military members' writing software. What did it take to get active-duty members to stop working on whatever they were supposed to be working on and head out to help and support you?
ENRIQUE: When we first decided at DIUX that we wanted to experiment with building software and try a different way, like agile software development practices and DevOps principles, we really weren’t thinking that we were going to stand up a software factory. That was actually not part of the original design; the original design was for us to learn how to do it and demonstrate there are better ways of building software, with the idea that the Air Force should know those better ways. Again, this is all about innovation and experimentation in the early days. When I looked at how software was being built in the DOD, it had really long timelines and was being done by defense contractors. So I said, What’s the opposite of defense contractors in traditional defense facilities? Well, let’s go, airmen only; nothing gets civilians or contracts. If I’m going to prove a use case, I’ve got to show the extreme: I grab only airmen, stick them in civilian clothes, and stick them in downtown San Francisco. Let’s go the complete opposite of what someone in the military is expected to look like, and let’s try that.
When I was looking for airmen, it was initially an email from myself to a few other officers I knew around the Air Force, and then I got the Air Force A6 staff to forward it around to all the cyber squadron commanders. And I had — I think it was 47 names come back at me, both officers and enlisted; blue-suit, active duty. I went through with a software developer I knew because, truth in advertising, I am not a software developer; I was a history major in college. We dug through all these records, probably interviewed about a dozen, and we chose six — three officers and three enlisted.
ROB: Talk to me about those six airmen at the time. What were they like? What were their backgrounds?
ENRIQUE: Sure. I won’t use names. What were they like? They were motivated. Part of it is that I didn’t pick the right people, but it turned out fine. The reason I didn’t pick the right people is that I didn’t know what I needed. All I knew was that I needed like six people; I was told that’s about the right size for a team. Then we worked at the company that did the enabling for us, called Pivotal Labs, which is now part of VMware. We show up the first day, and they’re like, Hey, who’s your designer? And I’m like, First of all, what does the designer do? I don’t have a designer. I have a bunch of people who write code. So I tried to force one of my airmen — a Master Sergeant at the time, I believe — to be a designer because his portfolio was the best-looking for it. It turns out he’s not a great designer. He’s a great front-end developer — actually, a great full-stack developer. He was like, “No, I’m not going to do it.” Well, I don’t have many options. So we just looked around the room, and I literally just picked a dude. I’m like, “I’m sorry, I have to have a designer. You’re my designer.” The same thing happened with the product manager. I told one of the software developers, “You seem like a pretty good leader. I grabbed one of my officers and was like, “You’re gonna be a product manager.” Because I didn’t pick a product manager, I didn’t know I needed a product manager. When you talk about experimentation and just going with what you got, I just went with what I got. I had six people. I told them, “We’re going to use you for the best you can do, even though it’s not what you signed up for.” And they had a great attitude about it.
They are militaristic, not in a bad way, but when you grab a bunch of military people, and you’re hanging out together, you act militaristic, which is kind of awkward sitting in a downtown San Francisco office with a bunch of less than-military companies working chairs over. So it was a fun dynamic, but I actually think everybody kind of loved it. They thought we were weird, that we were the military guys in the back corner. We thought all of them were kind of weird because they have purple hair and everything, while we all have shaved heads. It’s such a different mentality and cultural mix, but I think that’s what made it work — forcing us out of our comfort zones into this environment. Actually, I think it made the process and the product much better.
ROB: Do you believe that the active duty and civilian workforce in the federal government and in the DOD lacks the talent to write good software?
ENRIQUE: No, not at all. I think we have the talent, and even if we don’t, we have people who have the mindset and the capacity to learn to develop it. The problem is not talent. The problem is structure, policy, and guidance as to how to build software and why we build software. So we have the talent. I’ve never believed it was a talent issue, but we’ve proved that. I actually took over the Kessel Run program office, and we took people into the program office and made them designers and PMs (program managers). We actually made some software developers by sending them to boot camps. We have the talent; what we don’t have is the understanding that allows us to make good decisions about how to use that talent.
ROB: Okay, then talk to me a little bit about that talent aspect of things. Of course, in the military, federal government, and civilian, everything’s about your job code. Are there enough software engineers working across the federal government?
ENRIQUE: I’ll stick with the Air Force first; I don’t want to speak to the federal government or the DOD. The big answer is no; there’s not enough. Let’s parse that a bit because nuance really matters. Let’s look at the Air Force. We have hundreds of software developers, but we used to have thousands; it used to be an officer career field, and it used to be a very large enlisted career field. Now it’s a small enlisted career field, and there’s no officer equivalent. Active duty, we have probably 500, or a little more, coded software developers. The real question is how many of them are doing real software development. The answer is a very small number; a lot is being used for other things that are not slinging code; they’re doing quality control and management functions. They’re working on people’s websites, and it’s a different skill set between building a PA (page authority) website versus writing a full-scope application. So, I think the issue is that we have a lot of people who can code because it’s their career field but aren’t being used appropriately.
In addition, there are a lot of people all over the DOD in a bunch of different career fields that also know how to code, but because of the way we set up the personnel system and the way we set up primary duty assignments or secretary of duty assignments, we don’t give them the opportunity to use that skill set to be software developers within their own organizations. I think that’s something that we really should address, and a lot of people are trying to do that, but I don’t think we’re quite there yet as an organization on how to properly use people that can code and put them into an environment that enables them to actually deliver code. It’s just playing around until it’s in production, so we have to find ways to help them get their software into production.
ROB: So pre-Kessel run, even during Kessel run, or possibly even during the M-DOC days, was it ever at risk of just getting shut down?
ENRIQUE: There was not a lot of risk in the early days, in the first few months, because we did it out in California. We did it under the authority of DIUX, for which we work directly for SecDef, and we didn’t actually interface with the program. We deployed the stuff out to the Middle East, and the initial iteration did not actually require technical integrations with the weapon system. So when we first did it, it was actually not bad; there was no real pushback. It’s when we started building the second application because we had some publicity on the first one. So we started putting this up on the second application, and that one’s a little more closely tied to the weapon system. We now have at least one member of the program office working with us; we have parts of the program office advocating this methodology, and the rest of the program operations are going, “Who are you, and who said you’re allowed to do this?” So there was definite tension. It was in that transition at the end of March, where Jigsaw gets deployed before we actually officially stand up Kessel run by name at the end of April that there was that tension of “oh my gosh, they’re just going to shut us down.” We were actually very worried about that. We’ve got a product live, and now we’re going to be shut down because now it has attention. Now that people have seen it, they are talking about it. So suddenly, the antibodies come up.
So it was part of that four- to six-week period where we were really concerned about what that future was going to hold. But again, by bringing everybody out to the valley and taking it in a collaborative, cooperative manner versus saying, “This is the right way to do it; you guys are wrong,” and doing it more of an educational methodology, we actually were able to gain supporters and kind of keep ourselves, but yeah, there was some worry in there.
ROB: Describe for me your journey as Kessel Run went from DIUX and a good idea into a large organization. What was, in your opinion, the lowest point for you?
ENRIQUE: I’ll tell you, right now is actually a pretty low point. That’s kind of strange to say because I’ve been out for two years, but seeing where the community is going right now in terms of Kessel Run explicitly, not talking software development to the government writ large in the duty of software development writ large, I think is on a good path. My personal view is that Kessel Run is meandering a bit and is not going in the direction we need to go. I think we’re going back more toward stovepiping: among operations, among development, and among acquisitions. We built the organization to get all that stuff together in one area where we have a single authority to execute all aspects of what it takes to build a weapon system.
I think what you’re seeing now is the same tension to rip it back apart into traditional models to fit into traditional structures. So it’s actually kind of a low point right now to see what’s happening. During my time at Kessel Run, I’ll be honest, every day had a high point and a low point at any given moment. I think we’re doing amazing work, and I have probably the best airmen — writ large airmen or military contracts — I have the best of the Air Force. I fully believe that. They’re the most motivated, creative people you can imagine, but at the same time, I’m just constantly fighting for budget. I’m fighting for authority. I’m fighting to explain to people why those airmen should even be there as organizations try to pull them back to their core career fields. So I don’t think there’s a single explicit low point, but there are a lot of low points throughout the day, a lot of frustration throughout the day, and a lot of different topics.
ROB: So, talk to us a little bit about how you bounce from those low points. Especially in terms of giving folks recommendations, if you’re going to bounce between them multiple times in a day, you probably have a pretty good strategy. Just listening in terms of any advice you might have for folks about how they can push through when the bureaucracy is overbearing when they should have authority yet don’t have authority, when they know what’s right but aren’t allowed to do what’s right.
ENRIQUE: The greatest way to overcome those negative low points is success; success trumps everything. Let’s say you build an organization with the ultimate culture. You can imagine that if you don’t actually succeed, that culture starts to fray over time. It’s human nature. You thrive on accomplishments; success actually matters. So one of the things you can do is look back at any moment in time.
When I was at Kessel Run, I could look back on where we were a month ago, six months ago, two years ago, and five years ago. You mentioned that cloud article I wrote in 2015. I look back on that article (which, by the way, I’m in the process of writing an update on where I think we’ve gone since then), and I see some things that have not changed since 2015. There is some stuff you look at and go, “Oh my gosh, the world was different; what I wrote doesn’t even make sense anymore,” because you can imagine that old world. So I think part of bouncing back is looking back and seeing where you’ve actually been, where you’ve made accomplishments, and being proud of those. Be proud of what you’ve done. I think that’s a huge aspect of it.
The other part of it is: don’t take it personally. People will make it personal. There are people in the Pentagon who, I think, made it very personal. And I know that because either they said it to my face or I heard about it from others; they made it personal. It shouldn’t be personal. At the end of the day, all of us have the intent of enhancing national security. All of us have the intent of enabling the warfighter, but we may have different opinions on how to do that. So different opinions are not about making it personal. It’s not about making it emotional. A difference of opinion means that if someone does not agree with you, you do not yell at them. It doesn’t mean go stab them in the back. It doesn’t mean you get angry at it. What it means is that you need to think about what your underlying positions are. You think about how you’re enumerating those and figure out how to sell it better, how to convince someone, or how to educate somebody. I think people hit those low points in innovation — I have at multiple times — because they take it personally instead of buckling down and actually figuring out how they can do things better, which kind of leads to a third area of what they can do. Find someone you can talk to.
There is a difference, and you need to understand the person you’re talking to. If it’s just one person you’re going to talk to, and the two of you are just going to bitch and complain, you bring each other down and become more and more negative. That’s not helpful. You see this continuous downward spiral of negativity, where people now become an echo chamber among themselves, saying things like, “We’re the best; we’re doing it right; everyone else is wrong.” That’s the wrong kind of person to tell. You need to find a person you can talk to and who you can speak openly to, but the next sentence they’re going to have is, “Well, why don’t we try this? Or maybe this is the approach we should take.” and they’re looking for solutions. You need a sounding board for your negativity; get it off your chest, but that has to end with solutions, not with more negativity. It is hard to find those people. There were times in my career where I got trapped mentally, sitting around, calling people up on the phone, basically just complaining, which has never helped anybody. I see it a lot in our innovation ecosystem. I see a lot of people who just like to complain about everything, but they’re not doing the hard work of being solutionary and solving the problems. Everything is solvable, but you’ve got to be willing to do the work and find and hold a positive attitude that you can solve it. So.
ROB: You’ve transitioned from the service; you’re now working at a small startup called Second Front as the CTO. Do you want to talk to us a little bit about what you’re doing with Second Front?
ENRIQUE: Second Front is something I’ve been thinking about for quite a while. We have done a great job with software innovation in the DOD, focusing on: how does the government build better software. And how does it build it faster? What we have not quite done is figure out how to get the government to adopt commercial software faster. There are few efforts out there, honestly, and I think P1 has laid an incredible foundation for thinking through how to rapidly accredit and onboard software, but at the end of the day, there are still a lot of issues in terms of scalability. There are process issues as well as policy issues. So a lot of what I’ve been thinking about here is, How do we make it easy? How do we reduce friction? There should not be friction for commercial companies to get into the DOD. At the same time, there should not be friction for a government program officer, crediting official, or evaluator securing that software.
So what we’ve done is build a platform that’s really, we’ll call it a DevSecOps platform, but really the focus is on security and operations. How do we run high-quality and secure Day 2 operations on behalf of commercial entities to allow them to be accredited and operating inside the DOD’s environment? That’s what we’ve built — a product called Game Warden. And that’s really kind of the focus of it.
ROB: If Game Warden is successful, what does that mean to folks across the DOD or federal government? What does success look like?
ENRIQUE: I think what success initially looks like is competition. That’s how I’m going to know we’re successful. As soon as you have someone trying to compete with it, you know it was right, and that’s not a bad thing. I’m a massive believer in competition and free markets. I think competition is great because, at the end of the day, what are we trying to do? We’re trying to make sure that our government users have the best quality technology. There is no one person, organization, or company that could scale to do that. It just cannot be done. It’s going to take an ecosystem because there are a lot of nuances on how to do it. Some are going to go through something like Platform One, some will go through something like Cloud One, and I think some will continue to use FedRAMP. Some will come through someone like us who can actually operate it on their behalf in a commercial manner. So success from a user standpoint is when somebody puts a piece of software into a contract and is able to use it live in their operational environment within days because we’ve solved the accreditation challenge. So that’s what I think success looks like.
From the commercial side, it’s different. So it’s not like a multi-party, multi-market system here. What does it look like when we talk about the DOD and competition with China, and how do we establish a new defense innovation base to enable the warfighter? Well, then, success looks like a small startup that’s venture-backed and can get their products into the hands of a warfighter and make revenue so they can continue to improve that product. Rather than having their company basically die during that multi-year valley of death, then either the company collapses, or the company leaves government marketing and says, “Forget it; we’ll leave that to defense contractors. We’ll do commercial work only”, so there is an entire other avenue of success.
We, as a nation, have built a defense innovation base. It’s competitive with China because we no longer have software security accreditation as a barrier to getting access to the best quality technology. So that’s kind of what I’m seeing. I really believe what we’re building is game-changing. It will change the dynamic for both the commercial markets and for DOD users.
ROB: What advice and recommendations do you have for the listeners right now who are going through their own innovation struggles and trying to accomplish their own missions and objectives, potentially on a smaller scale or possibly growing?
ENRIQUE: A couple of things: First of all, you’re going to have to put in the hard work of getting top cover. You can’t avoid that. The greatest idea will lose out to a mediocre one if the mediocre idea is being backed by a program office. This is the reality of it. So if you are an innovator, you are going to have to do the hard work of getting leadership cover. If it’s at the unit level, maybe the link matters top cover. If you’re in a program office, it’s going to be like your senior material leader, a PEO, but you have to get top cover. There are so many things we can do to change the system. Let’s accept the fact that this is the system you have. How do you excel? How do you exceed inside that system? Because your success is what allows you to change. So stop complaining about the system, execute inside of it, and use those wins every step of the way to change the system.
That’s kind of like the first level of bias for innovators. The second one is, don’t stop innovating, but there are going to be setbacks. We had a product on which we pivoted. I thought it was a good product; I thought it had utility, but it didn’t have a product-market fit. So we pivot; there’s nothing wrong with that. I’ve had some other ideas inside the DOD that aren’t as great. That’s fine; it’s okay to not have successes. But when you don’t have success, we need to understand why you failed to understand how to do better the next time. Sometimes it’s out of your control, and sometimes it’s in your control, but either way, don’t stop innovating; keep trying. And at the end of the day, if you know who your user is and what they need, and you can find and fix that pain point, you’re probably on the right track. Don’t just innovate to solve your own problem.
Everybody thinks that they’re the smartest person in the world, and “I got this great idea.” Well, if you haven’t validated that with people who are going to use whatever you’re doing — process, product application, whatever it is — then you’re just being self-serving, and that’s not really innovation. Innovation has to be humble. So if everything you’re doing is self-serving, then you’re wrong. You have to do something that helps your users.
Subscribe to Email Option?
Keep up with the industry’s sharpest minds right from your inbox.
Want to hear more? Tune in to the podcast episode, where we cover lots more.
You can also subscribe to Defense Unicorns, a Podcast on iTunes, Spotify, or your favorite podcast app.
Thanks,
The Defense Unicorns Team
