How we released our first Open Source project
Some weeks ago, we released our first (bigger) Open Source project sda-dropwizard-commons on GitHub. Here we want to present how the project started, why we released it as Open Source and what challenges we faced while doing it.
Here at SDA SE, we use Dropwizard a lot for our microservices. Dropwizard is a small microservice framework built on top of Java EE libraries like JAX-RS.
Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.
Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.
While building our services using Dropwizard we noticed that it missed some features we expect from a microservices framework in a cloud-native environment. Most of the time this isn’t a big issue, as a lot of third-party solutions for components like MongoDB or Kafka are available. While the official Kafka library wasn’t available back when we started, other libraries didn’t fit our needs or were unmaintained. Some of Dropwizard technology choices don’t fit our tech stack. One of these choices is Dropwizard Metrics. While Dropwizard Metrics is a great library, we decided to go with Prometheus for our monitoring stack, so we had to replace it. Other technologies that we want to use are missing from Dropwizard, like OIDC support, authorization using Open Policy Agent (a topic my colleague will describe in a future post), an S3 client, or an OpenAPI integration. Some of our platform requirements are too special to find them in a framework, like trace headers or additional health checks. The default configuration of Dropwizard doesn’t match our security requirements and changing them in every service is cumbersome. Same goes for overriding dependencies with newer versions to respond to CVEs.
While one could argue now, that Dropwizard might be a bad choice as a microservice framework in our situation, we think that it still solves a lot of our problems and one would run into similar issues with other frameworks.
The birth
Based on this situation it didn’t took long, till some small internally libraries popped up to fill these gaps. At first that worked well, but then we run into issues that also a lot of third-party extensions to Dropwizard have: Version conflicts, out of date code, and a lot of work to maintain all the separate libraries.
So we started to combine our efforts into a single and central repository, with the goal to keep away version conflicts and provide a consistent library. This library was first developed by an internal platform team that provided components and framework for other teams. But later we restructured our teams to focus more on different business units and evolve the platform together. Instead of a dedicated team, a group of cross-team maintainers took the lead and turned it into an inner source project. A community with members from different teams used and extended the framework.
We have an open relationship with our partners and customers, which provides them a lot of access to our internal tools and frameworks. We didn’t officially announced the project, but they liked our framework and started to use it. This happened quite early, without us even noticing. However there were some obstacles for them when using our library. While all developers at our partners and customers have access to the build artifacts, most of them are unable to access and contribute to the source code or read the documentation.
To let our customers and partners participate in the project we decided early in the project that it would be good to provide it as Open Source. That would also match our company mantra: “we are open”. But from there it was still a long way to go…
The road to Open Source
Having the intention to Open Source a project doesn’t mean you can do it. First you have to convince stakeholders and get their approvals. We also had long discussions with our legal department about the license to choose. In the end we selected the simple MIT license. Once the conditions were specified, we developers could start our work. We made a plan what we have to do before we can publish the project:
- Check if the project contains only source code that we want to publish: Does the project contain any security relevant components that we don’t want to publish?
- Check our repository for secrets and clean them up if necessary. A tool like truffleHog is a great help for this task.
- Ensure that we meet the obligations brought in by used Open Source libraries: We use fossa for this.
- Update documentation: Include a security policy, license, and contribution guidelines.
- Make the CI pipeline public: Previously we used our internal build tools for the project, which are not accessible to the public. For security reasons, we don’t want to execute untrusted on code on this environment. Now we migrated the project to GitHub Actions.
- Create a new GitHub repository: We don’t want to publish the internal and private discussions of developers on issues and PRs.
Once we completed these tasks, we could flip the switch: sda-dropwizard-commons is finally public!
The future
Now that the sda-dropwizard-commons is public we look ahead to its’ future.
We still have some minor tasks to complete to make the project truly available to the public. Right now, build artifacts are only available to our internal users, as well as partners and customers. In the next step we plan to publish it on Maven Central, to make it much easier for external developers to use our framework.
Releasing it doesn’t mean that the development is completed. A big task will be the migration to Dropwizard 2.0, as it will require some bigger changes. However, we see this as an important step to keep up with new features and latest security fixes. In addition we want to move to OpenAPI 3 and add distributed tracing using Open Telemetry. We guess that our library users will come up with many more ideas that can be included.
So take a look at our project on GitHub, maybe it can help you or you are interested to contribute.
