Contractor Nava PBC’s San Francisco Office in the early days — Source

How healthcare.gov made good federal contractors possible

Healthcare.gov was the biggest government technology failure in recent memory. Only six people signed up successfully on the first day; millions of other people were unable to apply for health insurance. However, this grand disaster produced at least one positive outcome: healthcare.gov made possible the rise of non-traditional government IT vendors that offered high-quality products comparable to those found in the private sector. Healthcare.gov’s massive failure created two new contractors: Nava PBC and Ad Hoc.

Healthcare.gov catalyzed a wave of technical talent to join government, showed government the need for innovative contractors, and gave Ad Hoc and Nava legitimacy. This was a unique moment. Unless a similarly spectacular disaster occurs in the future, we are unlikely to see nontraditional vendors achieve the scale and success of Nava PBC and Ad Hoc Team.

Welcome to Washington DC

Healthcare.gov’s very public failure was a call to action for mission-driven, talented technologists. All the sudden, government needed them. Rohan Bhobe, Kalvin Wang, and Sha Hwang — who would later start Nava — worked on an anonymous social network at Kanjoya, a long-distance rideshare company that went through Y Combinator, and a company that made animated cards, respectively. The three, among several others, received a call from the White House: Healthcare.gov is broken, and we need you on the rescue team.

“I was like, ‘Is this a joke?’ ” Hwang says. “Because I’d only met her once for coffee. And she said, ‘No, it’s not a joke. Let’s talk right now.’ ”
Sha Hwang, co-founder of Nava, on being recruited by rescue team member Jini Kim—Source

For the first time in a long time, technologists were welcome in government. There were few, if any, employees with technical skills. For the most part, government contracted out work instead of building in-house; this meant government lacked the manpower or know-how to develop good software. Henry Chao — healthcare.gov’s project manager — recounts that policymakers often ignored the few technical people in government and didn’t include them in key conversations. The “coders” existed only to implement policy asks.

In this huge shift, technologists were not only welcome, but were the heroes. Instead of updating websites or maintaining legacy systems, a technically talented public servant could help make healthcare available for every American resident. This huge mission and possibility to touch the lives of every American allowed government to compete with Silicon Valley, offering a powerful mission instead of the ludicrous pay and benefits of the Bay Area. Curiously, working at QSSI or CGI Federal — winners of the biggest healthcare.gov contracts—also offered the opportunity to work on such a large-scale project. The healthcare.gov rescue team, however, had two key additional benefits: the ability to make major decisions without dealing with government bureaucracy or political repercussions, and the romanticism of saving government from its huge mistake.

On the rescue team

Once Rohan, Kalvin, and the rest of the team were within government, they began to gain the experience they would later use to launch a new breed of federal government contractors.

First, they learned how desperately government contractors needed their product, software, and design skills. With poor project management practices such as waterfall software development — in which one part of the software must be build before the next can begin — and constantly changing requirements from policymakers, contractors were set up for failure. At the same time, the misaligned incentives within the procurement system led to poor contractor selection and contracting practices. The cost-reimbursement contract vehicle selected, for instance, allowed a contractor to bill $209 million for one of the contracts meant to cost $56 million. In another example, the contractors working on the site neglected to implement any tools for monitoring the health of the site.

“There was no place to look to find out whether the site was up today or not except CNN, which was literally how we found out about problems a good part of the time in the beginning.”
Rescue team member — Source

Rescue team members realized they could do better, for less. With 1–2 engineers over six months, Nava built their Scalable Login System that saved Centers for Medicaid and Medicare Services (CMS) $65M+/year and could handle a load test of over a billion users.

Nava PBC achieved “7,754 requests per second sustained for one hour with acceptable latency and zero errors.” — Source

Second, the rescue team members gained invaluable experience. They learned to work within existing government rules and regulations, influence relevant stakeholders build with — rather than for — government. Many argue that healthcare.gov failed because of unclear policies set by legislators and policymakers, rather than government contractor incompetence. Those on the rescue team realized this and fought hard for executive leadership buy-in; for example, team member Ryan Panchadsaram remembers that the rescue team set the expectations early that leadership had to answer their questions within three hours. Since then, Nava and Ad Hoc have been sure to engage leadership of the contracting agencies when working with them, whether they build out vets.gov or an API for CMS’s Quality Payment Program.

Just as important, team members got a realistic sense of what it was like to build in government. Innovating within the bureaucratic, political behemoth would be the hardest work they’d ever do, and they had to be prepared. In the past, the civic tech community was characterized by bright-eyed, highly-talented technologists who had an “arrogance about tech”:

The problem is that when we invite people to civic tech meetings, we are necessarily leading with our framework (“we love civic tech and we think we can use it to solve your problems!”) rather than simply being present with them.
Daniel X. O'Neil, Ad Hoc’s Director of Product Development & Business Strategy— Source

This savior complex often leads to high-tech but unnecessary solutions; technologists then leave when they learn how hard it is to build needed products with government. On the rescue team, however, members gained a realistic understanding of the trials of government work and learned the skills necessary for overcoming them. They would later bring this experience, humility, and open mindedness to their work as federal contractors.

A shift in government purchasing

While healthcare.gov brought Rohan, Kalvin, and team to government, more importantly, it made government more receptive to technologists’ help. 94% of large federal IT projects fail, but from the perspective of a government employee, “Nobody ever got fired for buying IBM.” Healthcare.gov was the first time when this was not the case. Federal purchasing processes are by no means good now, but in order for Nava and Ad Hoc to win the contracts following healthcare.gov’s rescue, at least some contracts became:

Smaller—healthcare.gov’s massive contract sizes excluded small and innovative companies that offered high-quality products from bidding. Only well-capitalized companies can afford the insurance to be an actual contractor, hire the sales team to schmooze procurement officers and respond with bids that were hundreds of pages long, and have the work history necessary to make risk-averse government staff feel comfortable. As a result, small contractors could win business only by becoming subcontractors to one of the massive Oracles or Deloittes.

Some may say that federal government awards a significant proportion of contracts to be won by small businesses, but dollar spend does not map to the same proportion. Prior to healthcare.gov (and even today), federal government’s $86 billion IT budget per year was largely spent on maintaining legacy systems (~80%!) and massive contracts with Deloitte, SAP, Oracle, and the like (data from usaspending.gov suggests that Oracle has won $1.3T in contracts from federal agencies since FY 2008). After healthcare.gov’s failure, agencies saw the need for smaller contracts awarded to better vendors. Now, we see Ad Hoc winning a $12 million contract from the Department of Veterans Affairs (VA) to build vets.gov. In the past, contractors hoping to work on such a large software project would be drowned out by IT titans vying for hundred million dollar RFPs.

More of a partnership — by encoding all of the Affordable Care Act legislation within procurement code into a Request for Proposal (RFP) for healthcare.gov, CMS built an RFP that was 800 pages long. Given the massive administrative lift required for each RFP, only procurement officers with the expertise and experience issue RFPs while other public servants are happy to have the complexity abstracted away from them. As a result, few with technical skills or subject matter expertise are involved in writing the RFP. Further, the common practice of waterfall software development meant that public servants rarely saw the software until the contractor had largely completed building the software. Healthcare.gov showed public servants the importance of being very involved from the beginning. To build good software for government, vendors and government staff learned, the two sides needed to work together.

Since, contractor-government collaboration has refined and become more common. For reasons similar to those for the rise of Nava and Ad Hoc, healthcare.gov spawned the US Digital Service (USDS) and 18F, tech teams within government. USDS teams, Nava, and Ad Hoc continue to work closely to this day. The contractors also continue to work closely with agency staff to co-develop excellent products. These close relationships build up trust between government staff, Ad Hoc, and Nava, which are more than likely considered when procurement officers decide future contract winners. It is thus no surprise that Ad Hoc and Nava are contractors only at agencies with strong USDS teams.

Less prescriptive — although written by a largely nontechnical team, the healthcare.gov RFP, like most government RFPs, outlined requirements in great detail. In addition to outlining specific features even if it wasn’t clear that users wanted or needed them, the RFP prescribed that contractors must use Marklogic — an outdated database software that was rarely used in the private sector. As such, the winning contractors had limited room to innovate within the narrow constraints. Instead, contractors were set up to fail as they built features that were not necessarily best for the final product.

Page 78 of healthcare.gov contract HHSM-500–2007–00015I, compelling the winner to buy certain software licenses—Source

Contracts now are beginning to give more flexibility to the contractor to experiment with products best for the end user. The RFP that Ad Hoc won to build vets.gov specifies little more than for the vendor to “create an alpha prototype and conduct facilitated demos to elicit feedback for improvements to the design.” While many in government fear(ed) that this agile mentality gives too much freedom to the contractor, it has given Ad Hoc and Nava the ability to shine when executing on contracts, further bolstering their reputations.

Awarded based on demonstrated skills—contracts in government are known to be awarded based on ability to tell about skills, rather than actual ability. Because of procurement’s complexity for a contract the size of healthcare.gov’s, CMS knew the competitive bidding process could take a year, if not longer. Healthcare.gov had to begin building soon, however, so the agency selected an expedited contract vehicle — Infinite Delivery, Indefinite Quantity (IDIQ) contracts on GSA IT Schedule 70 — in which only pre-vetted contractors could participate. However, the way in which agencies pre-vetted contractors and then ultimately selected them largely depended on how well they responded to the RFP. Once the government IT project failed and was then fixed, government staff saw how a good response didn’t map to quality execution and that excellent execution was possible; thus began an effort to assess vendors in other ways.

Nava tells of the VA’s 72 hour coding exercise in which teams were assessed by “production-ready code, not a demo.” The hackathon-esque event was meant for vendors to show their process and demonstrate that they were able to execute on skills similar to those found on the contract. Similarly, a CMS RFP asked for prototypes that in-house engineers, designers, and product managers would assess, rather than the traditional hundred page response commonly asked for by RFPs.

Because of this new importance of demonstrated skills, healthcare.gov gave Nava and Ad Hoc the legitimacy to be a prime contractor, or one that contracts directly with federal government agencies. Typically, before a small company can successfully bid on a federal contract, it must sacrifice a significant percentage of the contract as overhead to another prime and work as a subcontractor for a couple of years (and even then, the company will likely still not win!). The founders of Nava and Ad Hoc, however, could point to their time on the healthcare.gov rescue team as evidence of ability to execute. Once they left the team, it was relatively easy for them to continue their work as prime contractors on a CMS contract. From there, they could point to their work on that contract as they bid on contracts at the VA and beyond.

Beyond healthcare.gov

Ad Hoc and Nava are beginning to expand beyond the CMS contracts that came out of the healthcare.gov debacle, but they still deeply rely on their time on the rescue team. Even much of their talent comes from USDS and 18F (including Nava advisory board member Mikey Dickerson, former head of the USDS), which started because of healthcare.gov. A quick glance on their websites reveal the same sentiment; their work on healthcare.gov is one of their biggest sources of pride:

Nava’s website
Ad Hoc’s website

This is not to criticize their dependence on the success of healthcare.gov’s rescue team. The founders’ work on the team was an invaluable victory for government technology, making health care available to millions of Americans and bringing many technologists to the public sector. Further, the CMS contracts are still ongoing and much of the team is still working on healthcare.gov. Most Americans have heard of healthcare.gov’s failure, and it makes sense to capitalize on the infamy. But, an unsettling feeling persists. Again, 94% of large federal IT projects are unsuccessful, but none prior to healthcare.gov spurred a wave of govtech or created innovative federal prime contractors that produce high-quality software at the scale. Had healthcare.gov’s failure been slightly less publicly disastrous, we wouldn’t have Nava or Ad Hoc.

While Nava and Ad Hoc are growing quickly and are making huge changes in government, their influence has largely been contained. At best, agencies with USDS members trust them, and, at worst, a small set of teams at CMS and VA know of them. We need more quality contractors winning large federal IT RFPs. In all likelihood, however, we likely will not see another Nava or Ad Hoc without another catastrophe and subsequent media swarm.

Until then, the best we can hope for are procurement workarounds championed by innovative public servants. With interesting experiments in micropurchasing and public challenges, maybe we will begin to see a meaningful cultural change within government. Hopefully, we’ll one day see well-written RFPs, procurement officers who can distinguish between good and bad vendors, and better collaboration between vendors and government staff during software development.


Athena Kan is a junior at Harvard studying computer science while taking a ton of classes at Harvard Kennedy School. She’s working on bringing young technical talent into government at Coding it Forward and is now looking to use software to streamline procurement. Follow her on Twitter at kan_academy.