Simplifying Your Tech Stack: How a Service Catalog Can Optimize Your Workflow
Introduction
Does this scenario sound familiar? You are a developer working in a software company that produces a multitude of microservices, each in its own stack, maintained by multiple teams. You’re tasked with developing a new service to perform some business functionality that depends on another existing service.
You’ve gathered all the requirements and have sketched out your design and now you begin to scaffold your implementation. First, you realize your service needs to perform data mapping to match the contract of an external API. You start looking for the owner of that API, who maintains the underlying microservice and its contracts. After some time, scouring various sources (Confluence, Google drive, Markdown documents, Slack, etc.) you may find something that is relevant but outdated.
You start to wonder if there is a better way to look up particulars of a service (i.e. owners, dependencies, API definitions, documentation), perhaps all within a single unified view that’s easily accessible. Furthermore, what if this centralized place allowed you to also track certain metrics and enforce certain standards for every service within an organization?
At SSENSE, we are often faced with these challenges so we devised a way to address them in a manner that would improve the overall developer journey as well as provide insights to our microservice ecosystem at a higher level, without having to rely on endless outdated spreadsheets. The solution: create a centralized service catalog that serves as a “one-stop-shop” for all our system inquiries.
What is a Centralized Service Catalog?
In a nutshell, a centralized service catalog provides a single source of truth for all services and resources within a software organization. Typically, it includes detailed information about each service, such as description, availability, performance metrics, dependencies, as well as documentation about their usage, all under one digital roof.
But why is having all this information in one central place so important? The benefits of a centralized service catalog are numerous, but perhaps the most important is that it allows organizations to manage their services more effectively and efficiently. By having a single point of reference for all services, developers and stakeholders can easily access the information they need to make informed decisions about which services to use and how to use them.
Other benefits include:
- Improved Transparency/Visibility: Users can easily view information on a particular service such as their API definition, availability, cost, and track whether it adheres to certain standards and internal policies (e.g. has proper monitoring setup, follows certain conventions, migration tracking, etc.)
- Faster Incident Resolution: Knowing who owns and maintains a particular service amid a plethora of microservices deployed within an organization, especially during an incident, can be quite daunting. Having that information readily available within a service catalog can greatly improve the time it takes to resolve an issue.
- Streamlined Onboarding: Aggregating documentation in one place can also help new employees and users get up to speed quickly by providing clear descriptions of the usage for each service or component. This can reduce the learning curve for new employees and improve their productivity.
- Better Cost Management: By providing detailed cost information for each service, a centralized service catalog enables organizations to better manage their IT budgets and make more informed decisions about where to invest their resources.
Building a centralized service catalog from scratch can be time-consuming and complex. Luckily, there are several readily available platforms that can help you build your service catalog with great ease and minimal effort.
At the moment, these solutions are largely different and choosing the right one depends on what you are expecting from your catalog.
Looking for the Best Option
During our research we evaluated the following four options that provide service catalog features:
Backstage
Built by Spotify, Backstage is an open-source platform, which means that those working on it can either contribute to the project or customize parts of it to best suit their team’s needs. This approach opens the door to a variety of technical and business-oriented features, while keeping ownership of the content in the hands of the developers.
Moreover, it offers a modern user interface, easy-to-use plugins and integrations, and a good set of features out of the box.
Datadog Service Catalog
SSENSE uses Datadog extensively. In the last year, they released their service catalog that offers acute visibility on the logs, allowing us to see the relationships between the different services that have been deployed. Adding an integration with their whole portfolio of services (dashboards, tracing), and automatic detection of the infrastructure dependencies, makes for a catalog with a solid set of features. However, most of them are destined for a market that has the technical knowledge to make sense of the information they are offering.
AWS Service Catalog
Another alternative is AWS Service Catalog, which offers a centralized place for organizations to manage and distribute IT services. It allows for easy configuration and management of services, and integrates well with other AWS tools. Like the previous one, it is also necessary to have a good level of technical skills to understand the benefits of such a solution.
OpsLevel
This is a Software as a Service (SaaS) solution that has many similarities with Backstage. It offers integration with multiple services, including an interesting one with Slack. When dealing with incidents, this platform can provide relevant information in the channel so that the team on call can quickly identify the issue or the person to contact for a fast resolution. The downside is that it is a closed source software and subscribing will generate costs.
At the end of our research, we decided to go with Backstage because of its ease of customization, extensibility via its plugin integration, and the fact that it allows owners to manage their own catalog entries within their repositories. We have been using it for a few months now and find that it is an efficient solution.
Lessons Learned
Building an internal developer platform based on Backstage taught us a few important lessons.
The first takeaway was the importance of focusing on a simplistic user-centric experience.
Enabling too many features without a solid foundation and a well thought out cohesive experience can degrade the user and development experience. A frequent release of new functionality is easily achievable due to Backstage’s composability system [1]. At the core of the framework are plugins, which empower developers to expand the platform in any way. Additionally, the existing plugin ecosystem is very well developed [2], providing significant functionality for “free”. Combined, these factors drive a lot of excitement and can lead to enabling lots of features.
Another key lesson we learned is the need to thoroughly define how the SSENSE organizational structure maps to Backstage’s catalog functionality early in development. The catalog feature, a core plugin, is responsible for enabling the ability to discover services and it is the foundation for providing relevant information for each of those services. Trying to fit the vision and culture of other organizations, such as Spotify, can cause problems [3]. The issue is that different companies have different structures. Therefore, it was important to spend time exploring and thoroughly understanding the catalog functionality. This involved feedback from architects and staff developers. After the initial implementation, we understood that a lot of the structure was missing. Redefining the catalog is an extremely painful and time-consuming process. Spending extra time and steps to make sure that the definitions fit all use cases within the organization is highly suggested. In our case, the catalog discussions uncovered some structural patterns that were hard to address and it created interesting conversations around standards. These discussions lead to necessary features that ensure these standards are met. We used the open-source Tech Insights [4] plugin to assess the service quality against these standards.
The last important realization was that driving Backstage adoption within teams was more challenging than anticipated. Each squad has its roadmap, and having Backstage integration prioritized by each team was not easy. There are many suggested approaches to help with adoption [5]. At SSENSE, we held several seminars to discuss what the platform has to offer and the roadmap we have in place for it. Another direction we attempted was to attach incentives by providing scores to each service. One of the primary scores evaluates the extent to which a team has integrated with Backstage. These scores create satisfying experiences for teams when achieved and clear visibility for management.
Conclusion
As organizations evolve at an increasingly rapid pace, keeping up with technical stack information can be challenging, not to mention time consuming. Therefore, it is crucial to have a centralized way to index this information.
At SSENSE, we chose Backstage to address our needs. Not only does it have significant out-of-the-box capabilities, but it also offers easy extensibility to support future business requirements.
Even though we are still in the beginning stages of building a centralized place for all of the technical information, we have learned a few lessons. Notably, the importance of understanding your organization’s ubiquitous language and ensuring that it accurately maps to the tool you are using.
[1] https://backstage.io/docs/plugins/composability/
[2] https://backstage.io/plugins
[3] https://github.com/backstage/backstage/issues/14263#issuecomment-1323065085
[4] https://github.com/backstage/backstage/tree/master/plugins/tech-insights
[5] https://backstage.io/docs/overview/adopting/#tactics
Co-authored by Dimcho Karakashev and Christian Belisle
Editorial reviews by Catherine Heim & Mario Bittencourt
Want to work with us? Click here to see all open positions at SSENSE!
