Open Source for New York City
A couple of weeks ago, I was asked if I would testify in support of two bills being put in front of the New York City Council (NYC’s legislative body), both around use of open source software in the city government.
I was honored to be testifying alongside people and organizations that I deeply respect. Of everything said in the two-and-a-half hour hearing, the testimony that stood out was from Karen Sandler, Executive Director of the Software Freedom Conservancy. The full text is here, but I wanted to pull out one particular section:
I am deeply familiar with the dangers of proprietary software. I have hypertrophic cardiomyopathy (I have a big heart) and have an implanted medical device with software that I cannot review or work with my healthcare professionals to modify. I rely on one company to ensure its safety and hope that they provide the life-critical updates I need. I have no real choice because there is no free and open source software defibrillator. I wonder every day if I will get an inappropriate shock or have my device fail through inaction. I live with proprietary software in my body, knowing that it has vulnerabilities I can do nothing about.
While there is no way I can compete with a story like that, here is the written testimony that I submitted (lightly edited):
Dear Chairperson and Members of the Committee on Contracts-
Thank you for the opportunity to testify to the Committee in support of two important bills:
I am a software developer who has been living and working in NYC for six years, and in that time have been actively engaged with the developer and civic technology communities. I currently work for 18F, a 100% open source team in the federal government, though this testimony is being done in my personal capacity. In and outside of my job, I am a maintainer of dozens of open source projects, and actively contribute to and promote open source across a variety of programming languages. I should make clear that I am not a lawyer nor an expert in procurement or policy, but believe strongly in the value of open source and effectiveness of public services. I will present information as clearly and honestly as possible, and include references where applicable.
I am largely in favor of both bills, though have specific feedback on how each could be improved. In this testimony, I will discuss the general benefits of open source, particularly in the context of government, and then address each bill individually.
Benefits of open source
For those that may not be familiar, “open source software” (OSS) means that the code for a system is made publicly available, and that it can (generally) be read, used, modified, and redistributed without royalties or restrictions. See Int. 0366–2014 §6–401(b) for the formal definition.
Interestingly, this bill is a perfect example of permissive licensing in action: the definition for “open source software” in §6–401(b)(1–10) is pulled directly from the Open Source Initiative (OSI) web site. Normally, copyright laws would apply to content published on the web, so text could only be reused with express consent of (and often a fee paid to) the copyright holder. In this case, however, OSI’s site footer specifies that:
Opensource.org site content is licensed under a Creative Commons Attribution 4.0 International License.
Therefore, the content can be reused and repurposed without the knowledge or consent of OSI. While the bill should reference the OSI to be compliant with the license, it’s important to note that this bill could not exist (as written) without the existence of permissive licenses for content. The sponsors of the bill didn’t need to come up with a definition of “open source software” from scratch — they could take the canonical definition that has been developed and refined since 1998 by top experts in the field.
Now, imagine a similar scenario, but applied to procurement. A piece of software may be purchased for one agency in government, but without carefully written contracts, use of that same software by another group or agency could require buying it again. These multiple procurements are a huge waste of both time and taxpayer dollars, but are all too common across governments of all levels.
Imagine instead that the software carried an open source license. If the software is custom-built, only one procurement would be required, and if it’s available “off-the-shelf”, the entire procurement process could be skipped. Any other agency in any government could then benefit from the existence of that software, skipping their redundant procurement cycles. There is also greater freedom to experiment with what software works best for a given problem, as there are lower financial/legal barriers to evaluate the systems. Agencies can also be more flexible in changing systems when it suits their needs, rather than feeling compelled to wait out the duration of a license.
Additionally, open source software means a reduction in vendor lock-in, because the source code can be read and modified by anyone, including a new contractor. Therefore, money spent on software contracts can be better spent in supporting and improving the software, rather than paying over and over for the same thing.
A common misconception about open source is that the security of the system is sacrificed by making the code publicly available. This argument is known as “security through obscurity”, and is specifically addressed (and refuted) by the National Institute of Science and Technology:
System security should not depend on the secrecy of the implementation or its components.
Relatedly, the Department of Defense issued a memo in 2009 stating:
The continuous and broad peer-review enabled by publicly available source code supports software reliability and security efforts through the identification and elimination of defects that might otherwise go unrecognized by a more limited core development team.
In other words, source code on popular open source projects is monitored by many people, meaning it is often more secure than its proprietary equivalent.
It’s important to note that source code being public is completely separate from data being public — an open source project is no more likely to reveal sensitive information than one that is closed source.
Open source software is widely leveraged in the technology industry. Among the top million most popular web sites:
Aside from what has been mentioned above, open source software (particularly projects with strong communities) has the following benefits:
- Issues that have been previously encountered are often documented on public sites, meaning that solutions can be found without relying on a support contract.
- Agencies can benefit from upgrades made by the community “upstream”, without the need for paid upgrades. Conversely, any improvements made by the city will benefit the users of the software, and thus contribute to the greater good.
- If a piece of software needs to be modified to suit an agency’s needs, the code is immediately available to do so, and there is no risk of violating the terms of a license.
- Because data formats used by open source (and the code used to generate them) are almost always open, there is generally better access to and portability of data than in proprietary systems.
- As noted in the preamble to Int. 0366–2014, the availability of the code means greater transparency and ease of auditing.
As mentioned before, I am largely in support of this bill. In this section, I will go over areas where it can be improved.
Open vs. closed source
The introduction states that free/open source software provides the following:
- Availability, succession, and permanence of the city’s data
- “Interoperability through adherence to open, platform-neutral standards”
- Dictate “how, and for how long, the city may use the software it has acquired”
- Ensuring that software doesn’t, “in addition to its stated function, also transmit data to, or allows control and modification of its systems by, parties outside of the city’s control”
Though these desirable conditions are easier to achieve through open source, it’s important to note that these requirements could be written into contracts for proprietary systems. Therefore, I believe that these points should be included in the text of the bill itself as requirements for all software procured by the city, but separate them from the stated benefits/requirements of the license.
Off-the-shelf vs. custom software
One thing I noticed in this bill is the lack of distinction between off-the-shelf vs. custom software. Off-the-shelf software (e.g. LibreOffice, a word processing program) can generally be downloaded and used as-is, whereas custom software needs to be developed specifically for the city/agency (e.g. nyc.gov).
While I clearly have strong feelings about the benefits of open source, I am also pragmatic about the tradeoffs. For example, use of “cloud” software (e.g. Amazon Web Services or Google Apps) can greatly reduce the operational burden/costs of the city, despite the fact that those systems are rarely open source and don’t necessarily provide the same level of control over data as self-deployed solutions would. In such cases, the benefits of using proprietary products may outweigh the fact that they are not open source.
My suggestion would be that each software need be evaluated on a case-by-case basis for its effectiveness in completing the stated objectives, with the following (possibly weighted?) considerations:
- Open source
- Available off-the-shelf
- Commodity software (i.e. is a similar offering available from multiple providers, and would the switching cost be relatively low?)
- Large community
- Comprehensive documentation
- Closed source
- Needs custom development/maintenance
- Offering unique to that provider
- Needs to be deployed by/for the city
Additional suggested requirements
The following requirements should be added to the bill explicitly:
- Any software that is custom-developed for or by the city should be released under a license approved by the Open Source Initiative, if not dedicated to the public domain through something like CC0.
- Any contract for custom software should require transfer any copyright in the work to the government.
- The public release of custom software should include all design documents and documentation pertaining to the project.
- All software should use open, standard, and well-documented data transfer and storage formats. There should be no restriction on the ability to read/write/convert to/from these formats.
- All custom software should be developed in the open from the start, and allow/foster participation/collaboration with the broader government community and the public.
This last point is in contrast to the “throw it over the wall” approach to open source, where the code is developed behind closed doors, and only publicly released once, or at infrequent intervals. This all but guarantees that the project will receive no outside contributions, and may contribute to fragmentation around the project community. More information at Civic Commons.
One open question: does there need to be any language in the bill pertaining to patents? This is an issue that can often complicate the use of open source software.
The part of this bill that jumped out the most was the following in §6–403(a):
coordinate with jurisdictions outside of the city of new york regarding the procurement of software
I don’t know of any initiative like this from any other government — I think it will be the most impactful piece of the entire bill, and also the most difficult to accomplish.
The original press release for the Civic Commons Act associated the bill with Code for America’s Civic Commons platform. Unfortunately, the marketplace software that it was presumably referring to no longer seems to be hosted at Commons.CodeForAmerica.org. This sort of turnover is unfortunately common for community projects without a corporate or institutional backer, and is an example of why a bill should not be tied to a specific piece of software. It may be useful to detach the idea of “a…FOSS portal…to facilitate collaborative software purchasing” from Civic Commons specifically.
That being said, wiki.civiccommons.org is a wonderful resource, which would largely benefit from use and contribution by the city government. These kinds of open guides are practically zero-cost to run and maintain, but can be hugely beneficial.
In general, this bill may place too much emphasis on the implementation details (e.g. building a portal), rather than setting desired outcomes and allowing the program office to identify the best solution. There may be research that went into this bill that I’m not aware of, but the user-centered approach would be to ask the question, “what are the biggest obstacles for procurement specialists and/or program offices to adopt open source software?” The answer may be discoverability of existing options, which a portal would solve, though it’s worth investigating if there are other large hurdles.
The text is somewhat confusing in whether “Civic Commons software source code” refers to code for the software listed on the site, or the code for the portal itself. For example, while the portal provides a listing of open source software for agencies to consider, is the intention that it will also provide hosting and collaboration features for that code? There is no shortage of platforms that solve this need well (in descending order of popularity: GitHub, BitBucket, and GitLab), and any custom/standalone solution will decrease the likelihood of outside contributions. I don’t believe there is any benefit in trying to reinvent the wheel here. I would recommend providing guidance on tools, but ultimately allow agencies to determine which best suit their needs.
The following points should also be addressed more directly in the text of the bill:
- Point out that the agency in charge of developing the portal will also be responsible for maintenance.
- The agency responsible for the platform should encourage community engagement for the projects and overall platform, with city and external government users, as well as the public.
- The agency (and the platform itself) should encourage modern/forward-thinking development workflows for the software listed on the platform.
- Ensure there are clear instructions about how to build/deploy each piece of software.
- Making it clear where to go with support questions, feedback, contributions, etc.
- Using package management, build tools, containerization, and other strategies for making deployment as easy as possible.
- What software is required to be listed on the portal, if any?
Open source software can be an enormous benefit to governments of all sizes, and adoption on the scale of New York City could make a huge difference in the open source ecosystem. There are countless software collaboration opportunities with other governments, companies, and the public, all while improving the efficiency and efficacy of NYC. I applaud the Council for considering these important bills, and hope they pass with the relatively minor changes suggested in this letter. I am happy to be a resource if you have any further questions. Thanks again for your time.