Designing Software for Developers: Building a Library
While most software engineers develop applications or products, the Architecture team at DraftKings has a very different focus: we develop core libraries utilized by other engineers to create those products. These libraries provide common or cross-cutting functionality such as fault tolerance, authentication, logging, and database operations, to name a few.
Our team has to consider many different aspects when building a new library. Foremost, we need the library to be easy to use. At DraftKings, developer efficiency is critical to successfully launching new products and features. We also need to focus on flexibility and writing APIs that are pluggable into any microservice. Significant planning is spent on evaluating the different use cases of a library across various teams. Lastly, we need to validate no production incidents occur from library releases. The Architecture team adopts various methods of testing and monitoring to ensure a smooth rollout of our software across all services.
Recently, the Architecture team began updating our existing MySQL driver to a more modern open source solution. Through this migration we have experienced many challenges that are valuable learnings for anyone writing software for other engineers.
Easy to Use, Hard to Abuse
Self-Documenting by Naming
One of our primary goals in producing libraries is to ensure developers understand how to adopt our new functionality quickly and correctly. Following industry best practices goes a long way towards enabling that.
One important aspect is the naming of library APIs. While naming is something we pay attention to in application code, it becomes critical when building a shared library. The engineers utilizing the libraries are not the ones who helped develop them. If a function is named in a confusing way, a developer could misuse the component or skip using it altogether.
With our .NET Core/Standard libraries, we lean on Microsoft naming recommendations, for example, naming an async method. We also closely follow patterns from open source libraries. We make sure all of our libraries follow similar naming conventions to create a consistent experience across our software.
Intelligent Defaults
Another strategy we use is determining the most common use cases of our library and designing our APIs around those. One example of this was our legacy bulk SQL inserter. We investigated how developers used it and built the new bulk loader to those specs. We still provide developers with intuitive ways to override legacy defaults, but now have an easy to adopt, drop-in replacement.
As you can see from the two examples below, it is not always necessary to specify options when batch inserting data. We noticed that in the majority of use cases the developer intends to replace duplicate rows. We also do not want to require setting a table name and column names if we can simply infer those from the model passed to our bulk inserter.
Complex API
Simple API
Bend Don’t Break - Writing Flexible Code
The Architecture team leverages several strategies for building flexible APIs, which allow developers to configure our libraries depending on their needs, as well as industry best practices.
The Power of Attributes and Reflection
.NET attribute tags and Java annotations provide a very powerful way for developers to configure how the API will consume their model. In the case of our MySQL library, we use attribute tags in order to allow developers to specify the table name columns and set properties for our API to ignore. This way developers can pass objects to our API that do not need to exactly conform to their table structure.
It also provides an easy way to avoid configuration if your model exactly matches the table you are inserting into. If no configurations are set via attributes, our library uses reflection to get the class and property names, which it then uses to build an insert statement.
Extension Method Configuration
A common practice in ASP .NET Core is to allow for configuration through fluent extension methods. This allows for setting up in what feels like a more natural way.
By utilizing extension methods, developers can configure any combination of options for our library components that are relevant to them. It also uses verbose method names for the extension so the developers understand exactly what options they are configuring.
Preparing for Real World Use
Testing for Performance
Our MySQL library is integrated into almost every microservice at DraftKings. One major goal was to ensure our changes do not have adverse effects on overall service performance. We split our testing across core functionality for both async and sync operations. Some of our services run synchronously, due to limitations from the old driver only supporting synchronous operations. The preferred implementation with our new library is to use async across the service, however, we still need to support legacy uses for some period of time. We set up testing to validate that sync and async operations on the new driver performed at least as well as the existing one. After running these tests, we were able to prove the new driver did not negatively affect performance, and in some cases significantly improved it.
Performance Testing Functional Components
In order to validate our solution quickly, we tested the new library code in isolation. This helps us to determine the overall performance, while removing any extra variables like latency. To do this we leverage an open source benchmarking tool called BenchmarkDotNet. This allows for writing Unit Test-like code to compare the performance of different methods. When running these tests, we make sure to confirm our new code performs properly up to the maximum our current software needs to perform at in production. For example, the new bulk inserting was validated to perform better than the old version up to 20,000 inserts at a time.
Sampling Across Microservices
Once the performance of the library is validated, the team is able to fully move forward with confidence. Next, we want to make sure the library continues to work when adding it to several microservices. Our team partners with various product and platform engineers to add our new library to their services. We then set up the services in our load environment in parallel with services running the old library. We push equal loads across both sets of microservices to measure the outcome. This allows us to run production-like traffic scenarios and determine if the software still continues to perform as expected.
In order to measure success, we track several metrics across our microservices using both the old and new library. This allows us to compare the performance for several microservices and make sure there are no unexpected performance hits with the new library.
As you can see the latency for database requests is almost 2x lower for this particular microservice. By compiling the results from running load tests against several microservices, we are able to determine the new library will likely be successful upon full rollout.
Collaborating with Teams to Canary to Production
As part of our validation process we partner with engineering teams to canary our library to production using their microservice. This allows us to responsibly roll out new features with a lower risk, if an issue were to occur. In order to canary, we create a release version of a service with our new library. We then deploy that release to a single production instance. We collaborate with the development team to monitor the service and ensure everything looks good before rolling forward. This process is repeated across several teams to gain confidence in our software, before releasing it to the rest of the engineering organization.
Publishing a Library
After finishing library changes, they need to be shared with the engineering organization, so people can begin using them. We utilize a few strategies on the Architecture team to raise awareness about new library features we release. One method we use is to communicate through several engineering Slack channels for every major and minor library release. In addition to this, the team holds bi-weekly tech talks to highlight what the team has been working on. This enables the Architecture team to demo our features and highlight how they can be used in a service. Lastly, we keep up-to-date changelogs of features we release so engineers are able to track when a feature releases.
Supporting Library Use
When rolling out a new library change to a service, the Architecture team will coordinate with the product team to create dashboards to monitor the overall health of the system. This gives both teams insights into the performance of the service and ensures there is no unexpected behavior. For our MySQL library changes, our team builds monitors for each service that integrate with the library to track the initial rollouts.
In the case of an issue arising with the software, we are able to isolate problems by tracking various metrics related to our library. For our MySQL library, the most critical metrics to track are latency, error counts, and new connections opened.
Here we can see a large amount of connections are created and there is a spike in database latency. Our tracking allows us to easily spot issues when they occur and quickly investigate them. We can then create a fix for the issue or determine if there was an unintended use for our library.
How do you Roll out New Changes and Manage Usage
For our libraries the Architecture team provides extended Long Term Support to specific major release versions. LTS means that the greatest minor version of the LTS major version will receive support from the Architecture team for 12 months after the date of that major version’s first release. This allows us to build confidence around using our libraries, without getting bogged down supporting too many versions of our software.
The Architecture team will also mark APIs or general functionality to highlight a deprecation. We use the [Obsolete] Attribute to generate a compiler warning when these APIs are used. These will continue to work and be maintained by the Architecture team as long as they are present in a supported version of the library, with the expectation that they will be removed in the next major version release. By setting these guidelines we are able to continue to build new features and adapt our software components to stay up to date with modern technology, without continuing to allow legacy software support to slow the team down.
Where to Go from Here?
Next time you are developing a software component, think about the ways it may be used by other developers in the future. You may want to build it as part of a software library if it is not part of your core business logic. By ensuring the library is easy to use, flexible, and scalable you can increase the likelihood it is reusable across multiple services and teams. Writing library code provides the challenge of deeply learning new technologies. This enables you to develop performant and scalable software, which empowers engineers around you to leverage those capabilities rather than recreating them. Get the ball rolling by building your first software library or expanding your existing set of common libraries. Your team will thank you for it later.
