Is Government in the Business of Software
Being a hard-core developer who cares mostly about software, often for the sake of software itself, and having worked at a local government organization for the past 8 years, this is a question that I’ve pondered often enough. On the one hand there is a running philosophical theme in later years, and through all generations of programmers, that programming is the new literacy, that since software permeates every aspect of our lives, programming amounts to expressing knowledge just like writing does. Therefore deciding whether to write software based on what business you are in would be as silly as deciding if you should be writing in English to conduct said business. On the other hand, programming still requires a fair amount of technical expertise and engineering skills and is therefore rather difficult and costly to cultivate alongside of what would be the core competencies of an organization. This is a dilemma that surprisingly nobody pays attention to. Surely, decisions are made when the problem presents itself, but those decisions are nothing more than practical resolutions of short term constraints. For smaller governments, being a software-competent organization is never on the table as a long term strategy. Even worse, software is considered a burden, a necessary evil not only by computer averse personnel, but by the IT department itself. I’ve come to the conclusion that the price paid for that attitude is much higher than one would reasonably suspect. These are my thoughts in support of an alternative.
Supporting vs. Vital Systems
This same dilemma of in-house software versus outsourcing is faced by any type of organization and the analysis usually makes the distinction between “infrastructure software” and “business software” with the latter encapsulating business knowledge. To invest in the creation of custom infrastructure software, a very strong argument has to be made and usually such software is purchased or an open-source variant adopted. But when it comes to business software, the decision becomes less clear cut. A recent essay by Philip Armour in the Communications of the ACM offers a very down-to-earth analysis of the trade-offs with several interesting insights (* the essay is pay walled unfortunately, but I am paraphrasing here what’s important for my points). Instead of the usual distinction of infrastructure vs. business, he talks about supporting vs. vital systems. A supporting system is one needed to support the business, but is not constitutive of the business itself. A vital system is the operationalized knowledge of what the organization does, that which justifies its existence. That’s a more useful perspective than the infrastructure vs. business divide as it shifts the analysis towards what differentiates one organization from another. Mr. Armour then goes on to argue that companies are well advised to avoid building in house supportive systems, but to also avoid outsourcing vital systems. The last part is anti-orthodoxy — we are supposed to outsource as a cost-cutting measure, regardless of the nature of the system, right? That’s the traditional view of software as a cost, a service you consume just like you need to buy or rent physical space to operate. The problem with that view is that software has become the best, most accurate (and sometimes only!) expression of business knowledge that an organization possesses. Hence Armour’s point: vital knowledge is the operationalized knowledge of what the company does for a living, why it exists. This knowledge, how it is obtained, how it is built upon and, most importantly, how it is operationalized (in the software form) is the most critical asset for many companies. Organizations need to own this knowledge in all its forms, especially the executable ones. But many companies do not realize this. The key point here is really a radical shift in attitude towards in-house development: rather than doing it as a last recourse, when other options are deemed non viable, it should be the first option to consider.
The Government Case
The argument in the previous section implicitly alludes to the notion of a competitive advantage because Armour’s target audience is corporate IT executives. But what about government organizations? In governmental contexts, competition is a motivator only to a limited extent because a successful government bureaucracy is hard to measure. There are certain competition playing fields such as constituent satisfaction as expressed in elections (or protests), various independent bodies evaluating government service performance, or IT departments and distributing awards. But those are hardly what IT projects in governments are ever after. More importantly, there isn’t any intellectual property worth protecting in a government business process (how civil servants do things), not to mention that at least in modern democracies such things are public information by law. On a different but related subject the whole movement of running government-as-a-business has created confusion within the ranks of public officials who, pressed to create private sector style efficiency, forgot that they don’t live in a corporate world and should abide by a different code of conduct — in particular, the spirit of secrecy doesn’t have a place in government (with the usual disclaimer of security issues). So no, making the case from the perspective of a business competing in a market doesn’t work for governments as it does not square well with the values of openness and transparency.
Yet, there is a universal appeal for in-house development in government agencies large enough to afford IT departments with programming staff. The rationale is still very similar to the corporate case, but at a deeper level: it has to do again with the fact that knowledge resides in software. The only difference is that in governments knowledge ownership is not exclusive — an agency’s performance is never relative to other agencies (as can be the case with companies competing for market share) and therefore its operational knowledge is not subject to intellectual property impulses. However, this does not imply by any sensible logic that a government agency can safely part ways with its competencies by replacing them with software black boxes. Software should be there to assist and streamline business process rather than the other way around. It frequently happens with closed and inaccessible software packages that business processes are established as workarounds to software limitations (and no, I don’t think bad software as a job creator is a good defense here!). Whenever knowledge resides in people, those people should be part of the organization, not outsourcing entities. In other words, it is still the case for government organizations that knowledge is the most critical asset which they need to own…in all its forms. I will offer further justification of this position below, but first let me address a superficial economic argument against in-house software in governments.
Software Procurement Economics
At a first glance, one is tempted to assume that governments are all offering nearly identical services and correspondingly all have the same software needs. Following that assumption, it is only natural to make the argument that those common needs lead to a marketplace for private industry products to compete, just like any other goods. Whenever there is a specific need, peculiar to an agency, the marketplace of service providers again lowers the cost because each provider competes for the business by offering the best price/quality ratio. This logic is perfectly valid for regular material goods, from office supplies to garbage trucks, or even building construction. But in the present context it breaks down for two reasons: (1) this is about software and (2) this is about government.
Perhaps the simplest way to see what makes software different from other goods is an analogy: is it easier to dismiss a knowledge worker who provides value through their ability to navigate a complex organization involving multiple conflicting parties or to fire a Chaplin-esque assembly line worker? Any government vital system, as per Armour’s definition, will be encapsulating and enacting business processes. Because software is the embodiment of business knowledge and because no business process is fixed in time, any software system adopted is subject to a perpetual evolution cycle. Therefore every procurement is a long-term strategic investment even if one refuses to see it as such. Now, we could categorize business processes and their corresponding system functions as unique to the agency or standard for all agencies and we could then argue how market forces should optimize cost/quality for a given system. But that would be ignoring the nature of the beast and what happens in practice. In the current state of affairs, a private corporation steps in, builds a product or as is also frequently the case adapts an existing solution to be more “government-like”, and sells it to all agencies in need. Operations unique to an agency are presumably rare, until the agency is lucky enough to run into a competent business analyst or until a painful implementation roadblock is reached. To solve the problem, first the agency pays for customization. Then engineering idealism dictates that the system should have been flexible enough to accommodate the agency’s special requirement in the first place. Therefore the next version of the system incorporates that single agency’s special needs into the core product, raising complexity, raising costs across the board, exponentially. Granted, when an improvement applies to everybody then everybody is happy. Except it’s a perverse phenomenon that I call “reverse consulting” — when the software provider, after an inadequate delivery, gets paid to acquire domain knowledge that they protect as private henceforth and go to sell elsewhere.
Furthermore, governments are not in the business of making money, but of spending money. Profit is never a motivator and consequently there is very little pressure to save money that you already have. Except at the time of crisis and draconian budget cutting measures, like we saw post the 2008 recession. There are excessive rules and procedures when it comes to procurement and a vendor approval process which effectively shuts the door for the vast majority of potential market players. Note that much of the extra constraints imposed on vendors are there precisely to deal with the fact that key operating procedures are being affected by an external entity. Moreover, the software systems are generally specialized to government even if some functional subsets have wider applicability. Therefore the potential users of any given system are much fewer than for an analogous system targeting private enterprise customers.
All of the above aspects lead to the non-trivial, yet unavoidable conclusion that market forces in governmental software systems are practically non-existent. A vendor with a lock in position only has to ensure that the cost of switching to another vendor is always higher than the maintenance cost of their system. A vendor will often claim that each client is essentially sharing the cost of software development with other clients instead of absorbing it all in house. Sharing the cost is good on paper, but the problem is that the entity determining the cost and administering the sharing is a single self-interested private entity whose sole purpose of existence is making that cost as high as possible. Once the lock in is established, the entity has free reign in managing the cost, the only question that remains is how many resources are available, how much is the government willing to pay, and any hope for free market dynamics and an objective value determination of the services provided are gone. This goes on a lot at government agencies because money is abundant and accountability is pro forma due to lack of expertise. One deep reason could well be that the cost of software is only marginal compared to the cost of, say, public transportation or operating a fire department. But my guess is that true numbers are hard to come by because software costs are not labeled as such, they are buried under archaic budget codes alongside other expenditures.
Unintended Consequences of Outsourcing
So, setting costs aside, what are the consequences of outsourcing for a government in terms of performing its duties? First, to repeat, software is unlike the typical physical assets acquired by an organization — it is knowledge and therefore much more alive. For example, we tend to use the term maintenance to describe the activities involved in keeping software working, but that is a misnomer. Strictly speaking maintenance is about keeping an artifact in the exact same state in order to prevent it from deteriorating. But software does not deteriorate. In the software world, we call maintenance the adaptation of the software to environmental changes. And environmental changes encompass the evolution of business processes, a multifaceted evolution to be sure — periodic reorganizations of the workforce, upgrades of dependent software, new automation opportunities, new regulations, re-engineering, new initiatives and what not. In other words, the organizational knowledge is a live, moving target and so are the vital systems of an organization. With an outsourced setup such maintenance happens only through contract customization which holds the agency hostage to cost, time frames and whatever other constraints the outsourcing company imposes because it has established a lock in.
Next, outsourcing software means outsourcing knowledge. It may not appear obvious how that is so, since the people doing the human portion of whatever service is being provided are still employed by the agency. True, but that’s only a superficial assurance. For one thing, the trend is for automated tools to take over more and more tasks. And even with automation already in place, technical expertise is nonetheless required, the most important one being the ability to make a judgement on the value of a software product. When no software professional taking the interests of the agency are available, gullible career bureaucrats of which there is no shortage at decision making positions, will go with the most charming vendor. Too many hugely consequential technical decisions are made at the executive level where technical merits are not even examined. Even if one refrains from the cynical position that government employees operate under too loose financial restrictions, how could one expect them to assess the merit or more importantly the monetary value of a piece of software? When a big company comes to a software professional with an estimate of $200,000 for what is otherwise conservatively estimated as 2 weeks of work, they get called on it. When they do that to a glorified Excel clerk promoted to a department director as a reward for their discipline and dedication, they just pay up or bow down and show empty pockets. Because all technical expertise and a substantial portion of the business knowledge is essentially deferred to the outsourcing entity, they rightfully consider themselves as the ultimate authority on the subject matter. Whether they do a good job or not is a matter of luck — sometimes they do, other times they don’t. But the ensuing arrogance on the part of private companies is appalling.
This is not about demonizing the private, profit-oriented enterprises. There is nothing inherently dishonest or ill intentioned in vendors’ behavior. However, the dynamics in this relationship between private and public sectors are not optimal, to say the least. Here are a few of the feedback loops responsible for the undesirable state of affairs, First, there is the spiraling complexity of a system that must meet the diverse requirements of many niche customers — all companies strive to make their product or service universally appealing so as to cover the largest possible market. Breaking a product into smaller independent pieces that inter-operate well with other software, a best practice evident to any software architect worthy of their title, is not in the interest of a software producer. The complexity becomes unbearable, which pushes the costs so high that the promised cost sharing is long gone. Moreover, release cycles and upgrades have to follow the planning of the software provider resulting in tremendous loss of agility in the face of changes. Then there is the vicious cycle in which a government agency, understanding well that a commitment to a particular vendor is long term, risky and with few affordances for change down the road, will go through an extensive RFP selection process that favors large bureaucratic counterparts. To be competitive in such bidding process, companies have to invest a lot in support infrastructure, investments which must be recouped by milking the agencies that end up buying in. One can argue that this is a necessary evil since a government’s priority is stability and not maximal efficiency. This is a valid argument, but I object that with modern communication technologies and open-source software practices, one can have the best of both worlds.
To sum up, the danger of outsourcing vital system stems from the ultimate loss of competence and ensuing degraded performance of an organization all with highly questionable financial benefits. In Armour’s word: this outsourcing approach amounts to an abdication of the task of extending and operationalizing their vital business knowledge…this is sad and it is dangerous. In their pursuit of a low-cost “solution”or perhaps in pursuit of an easy management life, these [organizations] may be surrendering their key assets and their futures.
The Alternative: Government As Its Own Open-Source Community
So what’s the alternative you might ask? That every government agency bears the cost and effort of writing its own software? Clearly this doesn’t make much sense either. As it often happens when an economic arrangement is over time discovered to be lame, cutting the middle man solves the problem. The middle man here being for-profit enterprises that manage the production and costs of vital software systems shared between multiple agencies with closely overlapping needs.
Open source software is already penetrating at all levels of government. There are many initiatives promoting the adoption of open source with the government not only being a passive consumer, but a contributor. A prominent example is the White House new U.S.Digital Service 18F task force, launched more or less as a response to the healthcare.gov debacle. For the systems falling into the supporting category, this is a great trend. On the other hand, vital governmental systems being rather niche software, there aren’t any major efforts to tackle them following an open source model. Why? Because of the wrong and outdated perception that a government is not “in the business of software”.
How about agencies employing qualified programmers, well versed in modern technologies and keeping abreast and writing the code to drive government business in an open manner, collaborating with other agencies, sharing expertise, sharing the workforce. That is true cost cutting and the best possible strategic long term investment. There are countless open source projects with well established practices within their respective communities demonstrating the success of this approach. An open source government community, G2G would build its own set of policies, its own mode of governance of projects if need be. But the benefits seem unquestionable as there will not be vendor lock ins anymore with the ensuing abuses.
Government operational software will be free, easily modifiable, easily adaptable. On staff development translates to an order of magnitude more agility for absorbing changes or fixing problems. Bug fixes and upgrades are done immediately, even within a single day. Software will play its role as a perfect vehicle for knowledge sharing and business process improvement throughout the whole life cycle. Rather perversely, knowledge sharing currently occurs mainly through private companies commenting on their experiences from one sales gig to another. Yet software is the one thing agencies that are completely disconnected because of geographical distance could collaborate on and it will be a tremendous opportunity for collaboration not only for the software itself, but about nearly every aspect of their day-to-day operations. Because software is knowledge, the consensus reaching process in open-source development will automatically trigger a much tighter knowledge sharing and in a much more practical, meaningful way. More importantly, because knowledge remains within the organization with programmers being on staff for the most part, project continuity is always ensured which is critical for long term cost control as well as for avoiding the past mistakes.
Besides, having a good team of programmers on board is liberating, it’s enabling. There are all sorts of little projects, ideas from this official with an obsession to provide service X or that bureaucrat dreaming of a tool that does Y, none of which is covered by off-the-shelf solutions. Unfortunately, currently a development team is a rare commodity precisely because of the cultural bias that government is not in the business of software. There are probably pockets of resistance (figuratively speaking) here and there, but most likely they are small and are not able to ensure continuity in that direction. As soon as an effort weakens, the large commercial entities or the commercial closed source spin-offs from what was originally non-profit efforts move in. Yes, it’s so much easier and fun to sit back and be courted by vendors. It takes off both work and responsibility off one’s back. And it takes a fair dose of courage and idealism on the part of decision makers not to do that.
Jump starting such a cultural change is clearly an ambitious project. But at least it should be talked about. At Miami-Dade County, we organized a small conference four years ago, consisting mainly of local neighboring agencies — the 2010 ShareGov Symposium. The idea is simple: we’ve written some in house code, it works pretty well, let’s give it to others and let’s collaborate in improvements and maintenance so everybody benefits. Admittedly, it does take a lot of time and effort to create a thriving open source project from a piece of internal code however well it might work. And not all projects will survive. But it’s worth it. It can also be a great motivator for employees when they see how their work is serving people outside the confines of their agency. A notable outcome of the efforts at Miami-Dade is one of the biggest and most successful in house projects done at that agency, the CRM system driving the 311 call center and a wide variety of departmental services. It is live and operational, it is on Github and waiting for attention. And there are others in the pipeline.
Going back to the motivation for an open source G2G vision, a key argument here is that allowing private interests to express themselves, to operate in the context of government is already in and of itself a form of corruption. Corruption in the sense of distorting the dynamics, the purpose and ultimately the function of the organization. Private interest is self-interest that will strive for secrecy and seek to establish a monopoly in order to maximize profit. That is great when market forces operate effectively. But self-interest can also be a great motivator for government. There is no reason public agencies should be outsourcing vital software systems — not only is this against fundamental principles of what government is about, but it does not make sense economically. In other words, the point is not ideological, but a pragmatic one as I hope to have convinced you. The perception that government is not good at technical innovation or at accomplishing challenging engineering projects may or may not be historically justified. The belief that government cannot and should not do such things is a counter-productive, self-fulfilling ideology.