The 10 Most Surprising Things I Learned Working in the Government
I worked at the United States Digital Service for 2.5 years after spending the first part of my career in startups. As someone new to the federal government, there were a lot of surprises both initially and after several years there.
I left my job over the winter, and since then I’ve found myself sharing some of the same odd anecdotes over and over.
Here are 10 of the most surprising things I saw or experienced in that time.
As you read the following, please know that I (mostly) loved my time in government. Checkout my post The Best and Hardest Job You’ll Ever Have, which talks the types of impact you can achieve only in this setting.
1. Ethics are taken very seriously
On my first day at the United States Digital Service (USDS), I went out to a drink with colleagues. I was surprised to find out that there wasn’t one bar tab for the group— I had to buy my own beer. The ethics guidelines in government are really strict. To avoid the appearance of improper influence over a senior colleague (which could be seen as a bribe), everyone pays their own way. Surprisingly (if you’re new to the DC-area), nearly all restaurants are used to this and are happy to provide individual checks. It’s often the default, and yes it’s OK if you split a $100 check 10 ways.
Oh, and let’s say you’re at a vendor’s office and are offered a coffee. There’s typically a jar to pay $0.50 for that cup o’ joe so that you’re not taking a gift from a vendor.
Further, there are also lots of rules (and clearances from the ethics office) to attend a conference. If you are offered a discount for tickets or travel (even if you’re speaking!), there’s more scrutiny. It got to the point where I didn’t even try to travel for the government any more. That’s not ideal — how is the government supposed to stay in touch with what the industry is doing? How are ideas to be exchanged?
2. It’s easier to spend $1M than $10,001
Want to buy a new desktop computer with all the bells and whistles? If it costs more then $10,000 (the limit was even lower, at $3,300 my whole time in government) or you want to buy several of them, then you’re not allowed to charge it on a credit card. You’ve hit the “micro-purchase threshold” and need to go through a multi-month competitive process of finding a vendor. And at this point, the agency will probably identify this as an enterprise-wide need. That makes the contract bigger and more attractive for vendors to bid on, which drags out the process even further.
It’s also quite hard to find someone inside the agency to do one of these “micro-purchases.” But when it comes to bigger spending, people know how to contract for millions of dollars without hesitation. It’s normal for a contract to be millions if not hundreds of millions of dollars. At that threshold, there are lots of guardrails put in place (review boards, departmental reviews, OMB reviews), that give folks a sense of security through a drawn out process. I’ve seen people much more comfortable spending $10s of millions buying something they vaguely understand than <$1M on a very specific product because they trust in the process.
3. It’s much easier to save $10M+ than you would think
I have a couple stories from my two years that are very similar to the one that my colleagues from the USDS team at the VA wrote about (it’s a great article, go read it!). Sometimes, by being at the right meeting, you can quickly rule out a poor technical proposal that was attractive to other folks by virtue of containing the right buzz words. Or you can explain that no, really, it’s not normal to spend $2M+/year on data center hosting for a low-traffic website. Often, though, it takes a lot more time and energy, especially where there’s more potential impact. For instance, convincing an agency that they should be using a SaaS product instead of trying to customize some “off the shelf” product might take a year but saves $10s of millions or more.
Why is it easier than you’d think? Because, as I describe in the next one, there are so few technical voices in the room…
4. So much work is done by vendors
There’s some variation, but many agencies employ only a handful of software developers. Instead, the IT org outsources most of the product development and implementation lifecycle to vendors. That means lots of PMs and middle management that can quickly balloon a contract. I’ve been told that the agency average ratio of federal employees to contractor employees on a IT project at CMS is 17:1. So if there are 1,000 people working on healthcare.gov (for instance), then there are only 58 feds working on the project. In my experience, many projects have a ratio closer to 40:1 (as best as I can remember, healthcare.gov seems to have closer to 25 feds involved in the IT build).
This means that there’s little oversight, little transparency, and vendors have a lot of leeway (this can be bad—see #9). A project often grows a large program team, whereas a few developers could do the same work. And it’s pretty common for a pure software project to have fewer than 50% of the team be developers.
Sidenote to entreprenuers: If you want to make a lot of money building a company, there’s probably no better way than to win a few government contracts. The numbers are huge (often as big as a Series A round) and you don’t dilute ownership. I’m fairly confident that a team of five experienced tech industry developers could be more productive than a team of 30 from most federal IT vendors — just think about those margins. I’m happy to introduce you to some folks that could provide advice — our country could use more good product teams in government software.
5. How laws and regulation really work
So congress passes a law, then what? The law might include a lot of money, timelines, checkpoints, etc. I worked on implementing parts of MACRA (a Medicare modernization bill) to which congress allocated $200+ million to implement and set several deadlines (sidenote: laws are like version control — they apply diffs to other laws, which is cool but means it can be really hard to figure out what the current law is since you only have the diffs and not the current state). Next, the Department responsible for implementing goes through the regulatory process.
For MACRA, the law is 95 pages. The regulation, which describes how all the finer details will work, is developed by policy makers — at the agency and at the White House (e.g. the Domestic Policy Council) — and lots of lawyers. After several months, it is posted as a “proposed rule.” This is often much bigger than the law: MACRA’s proposed rule is over 400 pages. At that point, citizens and companies have 30 or 60 or 90 days to provide comments. Rather than updating the text of the rule, we again go through a “diff” process, in which the government responds to the comments and will apply changes to the original rule in their responses. The final, year one rule for MACRA ended up being 824 pages once all these comments were included.
Why does it end up so long? Because there’s very little compromise. Rather, regulation is filled with carve outs and exceptions for different groups. It’s a noble idea—regulators use their discression to prevent undo hardships on certain groups. The problem, though, is that this typically done without input from the operations and product teams who have to build the IT systems to run the program. When you have 824 pages, many dedicated to edge cases (the carve outs and exceptions), a regulation can end up too complex to implement. Many people who were there claim this is the underlying cause of healthcare.gov’s initial failure—the business rules were too complex.
6. Outdated laws and regulation get in the way
Regulations are rarely revoked (last year, the Office of Management and Budget finally revoked a rule related to reporting on Y2k. No seriously.). Laws, which take an act of Congress, are changed even less frequently. So things tend to get out dated pretty fast.
Case and point is the Paperwork Reduction Act. With its noble purpose of preventing duplicate or cumbersome data capture, it has the unfortunate side effect of making it hard to change a form on a government website. In many cases, you must go through a public comment period to even do so. So if you found better phrasing during user testing or want to break one input box into two, you better be ready for a 3–6 month turnaround.
7. Compliance, compliance, compliance
As you may have guessed, there’s a lot of compliance. So you want to use a SaaS product? Well, is it FedRAMPed? Wait a second, is your crypto library FIPS-140 validated by a government lab? What FISMA level is your system? Does your documentation describe how you deal with a fire in the data center? Your password must be exactly 8 alphanumeric characters!
I’m not sure I really understood what compliance was before this job. I hadn’t been deep in a regulated industry like healthcare or finance. Coworkers have been in those positions, though, and they tell me that the government likes to add a whole new layer of paperwork and complexity. We do many of these things not because we have to, but because it’s the way we’ve always done it. It’s a lot easier to add new reviews and rules than it is to remove them.
Compliance gets in the way all the time, and it makes software less secure (for example: unpatched versions of OpenSSL are often used because newer releases aren’t FIPS-140 validated). Like regulation, it becomes outdated quickly and nearly everyone is afraid to deviate from what the compliance manual says. And if you do convince the right people, you better be ready for lots of paperwork and last minute cold feet.
8. Scheduled Maintenance is, unfortunately, a thing
It’s an accepted practice, written into contracts. I am so fired up about this one, I wrote a whole post on it.
9. There’s a whole set of technology companies that are government only
If you drive on the highway through northern Virginia, you’ll see lots of buildings with company names on them. The first time I did that drive, I had deja vu of making the trip before. I realized that I had made a very similar trip through silicon valley, but here I was in an alternate universe where all the companies had been replaced with ones I didn’t know (with some exceptions — IBM, Oracle and other big names are both places).
I always thought it was smart for the government to bring in technology companies to do the work — after all, industry is much better at building software than the government. In my mind, the government was relying on industry to mimic the success of large tech companies like Google or Microsoft. But now, I find it misleading when someone states that “industry knows best” because the government is often contracting with a subset of the industry that doesn’t overlap with the modern, technology and product focussed part of the industry that does know best.
Instead, large defense contractors whose expertise is launching satelites or building weapons systems are running data centers. When it comes to software, these vendors aren’t able to match silicon valley salaries/perks/lifestyle, so you end up with junior developers who don’t know the basics like unit testing. Program executives are more worried about going through the paces to make themselves look good on a compliance worksheet and to get their contract renewed.
10. Oh my, the acronyms
There are so many acronyms in government. Every agency is an acronym. Every process and team and tool and… the list goes on. And acronyms aren’t only nouns — you can inexplicably Freedom of Information Act (FOIA, pronounce foy-ya) something. An acronym (GS, which I don’t know what that means) determines your pay scale.
Still don’t believe me? The Centers for Medicare and Medicaid Services has a glossary of over 4,400 acronyms.
It’s so true that Acronyms Seriously Suck.
