Developing Badge-worthy Standards
When the Zephyr software badging team was formed six months ago, we had two qualities to help get us started — the desire to applaud folks who are building useful software tools in the transport modeling space + the enthusiasm to set up a reasonable and transparent process to “badge” their efforts. As David Ory posted earlier, the idea of the badging initiative was to create a low-stakes incentive program to reward anyone who builds useful and robust open-source software.
- But what is useful and robust open-source software?
- What does badge-worthy actually mean?
- How does the mission of Zephyr relate to useful software tools?
- Is any handy little tool badge-worthy?
- How mature does a software project need to be to “deserve” a badge?
Our team debated these questions and others via a series of exciting and enjoyable conference calls, and are now pleased to present our badge-worthy standards below and the badge application form.
Stay tuned as we test the badging process in order to verify that it is thoughtful and comprehensive. The Zephyr badgers are excited to begin reviewing and learning more about your open source tools and look forward to collaborating.
Badge-Worthy Standards
The goals of the Zephyr Software Badge process are to recognize the excellent software that exists in our community and to encourage those efforts towards collaborative progress that benefits the community. To qualify for a badge, the project sponsor needs to answer the following questions to our satisfaction. Each question is described in more detail below.
- Is the Software Useful to the Zephyr Community?
- Does the Software Contribute to a Common Problem Space or Benchmark in a Manner that Encourages Community Progress?
- Is the Software Easy to Use?
Is the Software Useful to the Zephyr Community?
The software badging process is sponsored by Zephyr and is intended to serve the Zephyr community. As such, the badgers must first determine if the software under consideration for a badge is useful to our community. Useful software is generally understood to contribute some kind of solution to a common problem faced by members of the community. This will be determined via the badge application form, to be completed by the project sponsor and then reviewed by at least three badgers. Rejected projects will receive a written response detailing the reasons for rejection; accepted projects will be notified of acceptance and invited to move to the next question.
Does the Software Contribute to a Common Problem Space or Benchmark in a Manner that Encourages Community Progress?
A key goal of the badge program is to encourage our community towards collaboration since we believe it results in better software solutions. As such software efforts must be oriented towards a common problem space or benchmark to receive a badge. This idea is best illustrated via some examples:
- Software that infers home location from mobile location data. This software could demonstrate its efficacy using a standard set of mobile location data. If a standard data set has not been established, which is likely in many cases, the project sponsors should propose a standard and make the data available to interested collaborators. The software does not need to perform better on any objective measure to be badge-worthy, but it must identify the standard data set so results can be verified.
- Population synthesizer software. This software could demonstrate its efficacy using a standard set of control and microdata, as well as a standard set of performance metrics so results can be verified and more easily understood. If such standards have not yet been established, the project sponsor should propose them and make the data available to interested collaborators.
- Interoperability standards. Much of the work in the Zephyr Community relies on standard data elements, such as networks and matrices, that use bespoke data formats. Interoperability standards, such as a standard matrix format or tools to read/write the standard matrix format, could demonstrate their efficacy by being able to be passed from one software package to another, to another, …, thereby making a round trip while maintaining its integrity.
The determination of this criteria will be made by the badgers. Project sponsors must complete the badge application form for consideration. Rejected projects will receive a written response detailing the reasons for rejection; accepted projects will be notified of acceptance and invited to move to the next question.
Is the Software Easy to Use?
The Zephyr Community believes software that is poorly documented and/or is not consistent with good software practices is rarely useful. As such, the final assessment of the subject project prior to receiving a Zephyr Badge is of the software’s documentation and coding practices. Projects will be asked to briefly describe their approach to the following:
- User and Developer Documentation — for example, installation instructions, system requirements, API documentation, examples/vignettes.
- Testing and Verification — for example, benchmark performance, examples, unit tests, coverage statistics, verification, continuous integration.
- Contribution Guidelines — for example, style guides, pull request etiquette, interface standards, development roadmap, algorithm documentation/methods description.
For open source projects, the badgers will review the project repository to assess the consistency between the stated and realized practices. At this time, we are only considering applications for open source projects; we may consider closed source projects at a later point and welcome any comments on this topic.
The determination of this criteria will be made by the badgers. Project sponsors must complete the badge application form for consideration. Rejected projects will receive a written response detailing the reasons for rejection; accepted projects will be notified of acceptance and provided a link to their software badge to display proudly on their open source repository and/or product page.
The Zephyr Foundation is a venue through which leading researchers and practitioners seek to develop and implement travel analysis methods and tools that are demonstrably more valuable, credible, transparent, tractable, reproducible, and usable in order to support public policy decision-making that is inclusive and promotes equitable outcomes, shared prosperity, and sustainability. Contact us if you’re interested in software badges, or learning more about other Zephyr projects.
