Building a Developer Portal with Backstage (Part 2)
How we adopted Backstage.
Oliver Sand with Dominik Henneke
In the first part, we wrote that Backstage inspired us to build a developer portal, not just a software catalog. We want to present our adoption journey, the features of Backstage that we use, and what we’ve done to make it work for us.
Our Backstage adoption journey
We have been working with Backstage for over a year now. Inspired by various posts about adoption stories from other companies, we decided to share our own journey. Let’s dive into the features we use and have developed with the Backstage community.
Our first steps
In our first pull request, we announced that we are going to adopt Backstage:
We started by getting familiar with Backstage and the community. We learned the architecture of the core and plugins, learned how to work with the Backstage source repository, and learned how to contribute. In this process we created our first PRs to do some small fixes. We created our own Backstage application and did an initial Kubernetes deployment. At the same time, we started to work on the first features.
Where are my APIs?
Rendered API documentation for all our software was our most anticipated feature. Our internal processes around API design involve OpenAPI and AsyncAPI, so we wanted to render them in our minimal viable product. There are already great open source libraries such as Swagger UI React Component or AsyncAPI React Component that we were able to embed.
While there was already the aim to manage API definitions in Backstage, the feature was not yet implemented. But since Backstage is open source, we joined the discussion in the initial issue, clarified the details, and provided a first implementation:
The plugin wasn’t perfect at this early stage, but we iterated and improved it in various follow-up PRs. This also involved contributions from others outside of the SDA SE.
Where are my developer tools?
After completing the first version of our software catalog, we reminded ourselves that Backstage is more than just a catalog — it is a developer portal. Hence, we wanted to give our developers more benefits of using Backstage to catalog their software by integrating their favorite tools.
The community has already provided plugins to integrate with GitHub Actions, Sentry, Jenkins, and many others. But we also use tools that have not been supported yet. So, we created and contributed plugins for SonarQube and FOSSA.
We were also able to motivate another team from our company to develop a plugin for Bitrise (learn more in their blog post).
Where are my docs?
In the first part of this series, we presented our problem of scattered and incomplete documentation of our software components. We solved this for API definitions, but it was still an open problem for the rest of the documentation. Fortunately, Backstage already comes with a nice documentation plugin: TechDocs. Each entity in Backstage can provide its own documentation. And there is also the option to create standalone pages for independent guides. The approach of handling docs-like-code was an internal request even before we heard of Backstage, so it’s a perfect fit.
There are different options to integrate TechDocs. Backstage recommends a flow where the (MkDocs) pages are built in the CI pipeline, but that requires integration by every team. Instead, we used an enhanced basic flow, where Backstage handles the building and publishing of the documentation. This required some contributions. We also took security precautions by building the documentation in a separate environment, isolated from the Backstage backend.
Where are my Software Templates?
We develop microservices and our teams regularly create new services. In the past, creating a new project involved copying an existing service and taking over the good (and bad 😉) parts. Developers had to find and replace all the service names, and delete any code they didn’t need. As you might imagine, it took a few days to get everything right, and misnamed things still showed up months later. For a long time, we wanted to create a blueprint project that anyone could copy from, but we were afraid it would become outdated — and it still involved copying code and renaming things. We also considered using a template-based generation solution to get rid of the manual tasks, but we didn’t know the right tool for that.
When we started looking at Software Templates in Backstage, we learned about Cookiecutter — a Python tool for executing templates based on user inputs. Independently of the launch of Backstage, we experimented with it; and after a short while, all new projects were created using our new Cookiecutter template. We were able to reduce the time to create a new project from several days to just under an hour. However, this still required the developer to install Cookiecutter, download the template, create GitHub repositories, and perform other manual steps.
So, we took the chance to migrate our Cookiecutter template to Backstage. The manual steps are gone, and a developer can set up a new project in just 5 minutes.
Where is my X?
The best catalog is useless if nobody can find anything in it. That’s why Backstage provides an extensible search framework. It can index any data, such as catalog entities, TechDocs content, or even external sources. We use it to index API definitions and an existing internal non-TechDocs documentation page. Thus, it became the universal search for everything tech-related in our organization. OK, we don’t index Confluence, but we want to motivate the teams to move on to better structured docs with TechDocs 😉.
In the first version of the search plugin, it was implemented using Lunr. Unfortunately, our dataset was too large to reasonably fit into memory and the index building degraded the backend performance. So, we decided to add a PostgreSQL based search backend. We figured that our dataset is small enough to be handled by Postgres’ full-text search capabilities.
This also meant that we could avoid the burden of operating a dedicated search backend like ElasticSearch, which we didn’t use in our company yet.
Why is there no better visualization than tables?
Tables and lists are not necessarily the best way to visualize data. Especially if you want to discover services or understand the relations between different entities. There was initial work on a graph to visualize relations, but it wasn’t very powerful yet. We added a graph plugin to visualize all the entities in a catalog and lets you understand how they relate to each other.
Here’s what it looks like:
There is also a dedicated graph explorer that allows you to dive even deeper into the network of entities:
Where is my ecosystem?
In the first part of this series, we introduced how we are creating an ecosystem of ready-to use solutions. With our new software catalog, rendered API documentation, release notes, and TechDocs pages, we now have a central place to find all the details our customers need to work with our products. And the search and graph representation are attractive for discovery. The only building block that was missing was that our customers should have access to all this data. But we can’t give them access to our developer-focused Backstage installation. We need a tool that is more like a marketplace and less like a developer tool — but with all the important information.
We decided to create a separate portal based on Backstage. Teams create the content in our internal portal and decide which parts are ready to be shared with the ecosystem. We collect the Components, APIs, Systems, and Domains, and copy them, together with the corresponding TechDocs pages, into a separate Backstage instance that focuses on third party access. This instance contains only the essential service catalog and discovery features, but skips adding plugins for any tool integration.
If a customer uses an own Backstage instance for their developers, we can also mirror that content into their catalog so they can discover the ecosystem in their own environment. This has the added value of allowing them to establish relations between entities from the ecosystem (e.g., an API) and entities in their own catalog. A future extension might also enable them to share their own solutions to the ecosystem.
This feature is very specific to our organization and is not Open Source. However, it was very easy to build it on top of the Backstage architecture. We use the catalog API to export the entities and we use entity providers and processors to read the data into the other instances. The result is that federated entities are part of the original Backstage plugins, just in a dedicated namespace to avoid name overlap.
Conclusion
Backstage restores order to our ecosystem (as promised on the website). All products and their ownership are visible to everyone in the SDA ecosystem, new services are created in less than five minutes, APIs and services are documented properly, developers have access to the most used tools through a unified interface, and our ecosystem portal provides up-to-date information to our customers. Backstage is the perfect platform to build a tool that really helped us as a company to grow toward our vision.
We have really enjoyed working with the Backstage community and contributing to the project. It has been fun to be part of the development process of such an emerging project — and we plan to continue our contributions in the future. We can really recommend getting involved in the community.
What’s next for you?
We’d like to wrap it up by sharing some tips and tricks for getting started. First of all: be open to the ideas Backstage promotes and see how it fits to your organization. Everyone is welcome in the community — you don’t have to be a large enterprise or become a top contributor to use it. You’re always welcome to report issues, suggest ideas for new plugin or feature, fix issues, and/or contribute new features. Every idea and expectation helps to find a solution that works better for everyone!
A good place to start is with blog posts about adopting Backstage and other Backstage success stories. One thing that we can highly recommend is to focus on a central topic that has a large leverage. Make Backstage a place where developers can find (or do) things they can’t get elsewhere — or at least not as convenient. This might be a quick entry point for new hires or the best experience to create new service. Maybe you have an obscure tool or process that nobody likes to interact with, but you can make it cool and easy to use in Backstage. Once people are excited, you can add more features and explore the other plugins. If you want to get started right away, head over to Getting Started and create your very own Backstage app.
For the community, you can add yourself as an adopter, attend the community meetups, and read or participate in RFC discussions or other GitHub issues. Start small with your code contributions and don’t be afraid to take on larger tasks that you think are valuable. The community and the maintainers are happy to help and move you through the contribution process.
This was the last part of the series; we hope you liked our insights on our journey to Backstage.
If you find this interesting and would like to join us, we are hiring.
