The Developer Experience Team: An Internship
Our great summer interns, Weston Greene & Tim Geissler, had the opportunity to help SailPoint implement backstage.io in our engineering environment. Read about the team’s work with our DevEx team and back stage, followed by Wes and Tim’s thoughts on their experience!
The overall concept of Developer Experience, or DevEx, extends far beyond our team here at SailPoint — large enterprise companies have found many different approaches for tackling the challenge of scale, most often being the adoption of microservices. While end-users and clients will usually never have to encounter these, the role that product engineers play in building these services is crucial. Companies have become more aware that the experience of their engineers while creating, maintaining, and deploying their microservices is tied directly to its ability to scale. Organizations like RedHat and Bloomberg have built their entire infrastructure with an emphasis on their developer-centric focus; increasing efficiency and productivity helps developers across the board and is especially important for newcomers, allowing for seamless onboarding and integration. Fundamentally, the goal of DevEx is to maximize developer velocity, code quality, and standardization of development practices across the organization.
At SailPoint, our service landscape spans hundreds of microservices, all supporting our IdentityNow product. The DevEx team was formed to provide all of our SaaS platforms with a team of developers who can streamline the creation, deployment, and upkeep of our legacy services, and for our new services, while ensuring the codebase is kept clean and well-documented. This provides a seamless experience both for existing engineers and new onboarding engineers alike.
Backstage Developer Portal
Before Backstage.io, creating new SailPoint services — a common task for the Platform DevEx team — required starting a project from scratch, including writing standard boilerplate code to add basic functionality and interoperability with other SailPoint services. This process often took multiple days per new service, which left less development time available to build and deploy the service itself. This inefficiency was compounded by the complexity of organizing the vast number of services which SailPoint’s Platform team oversees. By building Backstage, we wanted to automate the mundane task of creating and cataloging services and enable the DevEx team to spend more time building tools to maximize developer velocity and less time on administration and repetitive boilerplate code.
The inefficiencies which arise from building new microservices are not unique to SailPoint. Spotify manages a vast number of microservices, and as the number of new services increase, the amount of code boilerplate and complexity also increase. Creating a new microservice can become nearly impossible. This lead Spotify to create Backstage, an open-source, developer-focused portal which seeks to improve productivity and overall developer experience.
In the SailPoint DevEx team, Backstage has the potential to become a highly valuable tool to eliminate the need for boilerplate and reduce the complexity of the microservice landscape, along with introducing a simpler process for onboarding new developers and getting them up to speed on the core components of our old and new services. It extends to our entire platform, allowing all of our developers to quickly stand-up any new service, which would be built on top of existing code templates, in a matter of seconds. All services are hosted on the SailPoint GitHub, meaning that the administrative operations that could slow down deployment before are no longer the common blockers they used to be. Backstage provides a seamless and efficient pipeline for enhancing developer experience.
Backstage Project Overview
Backstage.io — Forked from Spotify
Backstage is an open-source project originally developed by Spotify. Excellent documentation at http://backstage.io allowed us to quickly get a local development version of Backstage forked and running, so we could begin building and customizing the project according to our specifications. The project included a NodeJS backend and React frontend, and was extensively modified and extended over the summer to meet the needs of the DevEx team.
Building Software Templates
By the end of development, four different SailPoint templates were written in Go and Java, and based on the atlas-go and atlas-boot libraries to target the most common templating use-cases: RESTful API calls, event-handling, and IdentityNow connections. These templates were designed to incorporate seamlessly into SailPoint’s system of microservices with established development best-practices, and populate into Backstage’s Software Catalog.
Building the Software Catalog
Each template was designed to integrate automatically into the Backstage Software Catalog, with the relevant metadata and ownership relationship mapping. Backstage is also configured to ingest pre-existing service repositories on Github and add them to the catalog. These pre-existing templates can be easily configured to register into the Catalog using a simple .yaml file containing the relevant metadata.
The Docker Container
Build staging for Backstage is included out of the box for developers — the official documentation provides a Dockerfile for building the backend of our application. For Backstage, the backend is responsible for provisioning the frontend to the user, and runs all traffic to the app through a built-in proxy through a custom plugin included with the app by default.
Some of the services we need to run alongside our app, specifically our database, our cache, and our proxy will require their own images. For deployment, we use a build tool called kaniko for standing up our containers, which uses a similar staging process to other tools like docker-compose, which we use for our development staging. A caveat to using kaniko is that we are required to use only a single Dockerfile for building our app, which we will see later would be a constraint on how we can set up our images for all of our separate services.
The NGINX Middleware
Despite being a service which would only be used in SailPoint development contexts, we made sure to integrate several security measures into Backstage. Since the dashboard itself is a frontend application, one of these measures places an NGINX proxy between the traffic from the end-user to the server, and vice versa. NGINX is lightweight and fast, and can be spun up for a production build easily with its own official Docker image.
By default, Backstage serves the frontend to the user through the backend — this is not ideal for placing our desired proxy in front of our web app. Luckily, the process for partitioning the Backstage app into a separate frontend and backend is outlined in the official documentation — however, this method still did not fit our proxy solution, as it does not build any of the backend files alongside the static frontend files. In the end, we used a series of shared volumes between the backend and NGINX containers to communicate the static files and provision the frontend to the end user.
The POSTGRES Database
Backstage defaults to SQLite for both local development and production builds, yet recommends using PostgreSQL for deployment. As we are using our Docker build for our production staging of the app, we decided to use SQLite for our development and debugging purposes, and Postgres in our containerized app.
The Backstage CLI that builds the app uses our app-config.yaml for configuring our local environment, which means we needed to partition our app to use both local and production config files. We created an app-config.prod.yaml file which reads the Postgres config in our Dockerfile, and we left the default SQLite database configuration in the app-config.yaml for development purposes. Ultimately, this gave us flexibility between a fast and easy-to-stand-up database for local builds and a fast and robust database for deployment.
Redis Caching Optimization
Caching with Backstage comes with a default in-memory cache implementation which is fairly barebones, but is still sufficient for the purposes of local development. For production builds, there is straightforward support for both Google Cloud App Engine memcache and Redis. We ultimately chose the latter, as it was already utilized by many other SailPoint micro services.
Backstage uses caching for different purposes, such as storing all of the owners in a SailPoint organization and persisting the contents of the software catalog. A robust Redis cache is appropriate for the deployed app, but for local development, standing up a local in-memory cache is a more suitable solution. Similar to the Postgres process, we configured Backstage to fall back to a memory cache in the app-config.yaml, and delegate to Redis in app-config.prod.yaml.
Backend Parameter Handling
Although most of the app configuration for the Backstage environment is provided through the app-config.yaml, this is ultimately not as secure as we would like for our deployed app. Using this method would place our secrets in plaintext within a text file, and although our app is containerized, it still presents an area of potential vulnerability in the event of a container breach. Ultimately, we want to populate the Backstage environment through the backend, so instead of code reading specifically from our config files, we can leverage a request to read these secrets dynamically and store them in the app directly during startup.
Since our secrets are stored in the SailPoint AWS Parameter Store, we used the Amazon AWS SDK to read these parameters into our local environment and populate them there, rather than insecurely storing and reading them from plaintext configuration files.
To know which parameters to query specifically from the Parameter Store, we read them from a custom config file, integrations.json, and then used this to invoke the SDK. The location of this file changes depending on whether we are using a production build of the app, so we use an environment variable that communicates this to the backend.
CI/CD Deployment
Each Pull Request triggers a Github Actions workflow which installs the necessary dependancies, compiles the TypeScript code using yarn, and builds the entire project. Only branches which build successfully in the workflow are permitted to merge to the main branch.
The Backstage deployment pipeline is written in Jenkins and hosted on a Cloudbees server. The pipeline builds a Docker container using the docker-compose file, pushes the container to SailPoint’s Elastic Container Registry, and then deploys the service to an Amazon ECS cluster.
Outcomes for DevEx and SailPoint
The Backstage project is now a fully-deployed developer portal which services SailPoint’s entire Platform engineering team. The process of creating a new cloud service, which used to take days of coordination between multiple teams, is now automated by Backstage’s Software Templates and occurs in less than a minute. This automation also means that coding best-practices are now standardized across old and new services, and integration into other relevant frameworks and services is automatic. All of SailPoint’s services are now catalogued and mapped by Backstage’s Software Catalog, maximizing organization and increasing development velocity.
Beacon Dashboard Project
Beacon mode is a sophisticated internal tool developed by SailPoint which allows cloud services to be executed and debugged on local development machines. To achieve this, Beacon routes relevant network traffic (Redis, Kafka, RDS .etc) through the local development instance instead of the deployed instance, allowing for seamless service debugging and testing.
To troubleshoot Beacon mode, developers are required to execute manual DynamoDB queries to search for tenants which host the cloud services, and place breakpoints in the service code to check the routing of each API request through Beacon.
With the goal of eliminating the tediousness of this task, we were tasked with building a dashboard to simplify the process of troubleshooting Beacon mode. Ultimately, the project will automatically query the DynamoDB database for the relevant data, return the results through REST API’s, and display the data in a developer portal dashboard. Concurrently, the atlas-health check module will be updated to return relevant Beacon mode data through a health-check API call.
Beacon Queries and REST API’s
Currently, checking the entries present within the SailPoint DynamoDB is a manual process — checking if these entries exist while troubleshooting a service in Beacon requires going into the AWS Console separately from the IDN management portal. To automate this process, inside the atlas-beacon-rest module, we leveraged the Java client for the AWS SDK and automated queries to the necessary tables. We also wrote some REST API endpoints to return this data with the payload back to the IDN management UI to display it directly alongside other Beacon tenant data.
Extending Health Check Field
The atlas-health module needed to be extended to collect data relevant to Beacon mode (including timestamps, hostname, port number, uptime .etc), and return that data in the form of a REST API endpoint as a part of the standard SailPoint Saas health-check system. The Beacon health data was collected primarily through Java System calls and various SailPoint Atlas configuration variables. An accompanying JUnit test was designed to mirror the tests already present for other health-check modules to ensure the Beacon module behaved as expected.
Building an Interface
Finally, the SailPoint IdentityNow Management Portal, which is written in Angular and serves all SailPoint Saas engineering teams, needed to be updated to display the Beacon data returned by the new REST API’s. A new UI module was written to concisely render the Beacon mode troubleshooting data, and was integrated into the deployed Management Portal interface.
The Internship Experience
Tim Geissler: The internship began with a week set aside for us to learn about SailPoint and understand what we would be working on over the summer. During the first few weeks, my main goal was to understand how the organization worked from a DevEx perspective, and how that would translate into day-to-day work in order to successfully complete the Backstage project on time.
If my experience surviving a few semesters of “Zoom University” during the pandemic taught me anything, it’s that organization and communication is key when working remotely. The ‘virtual’ aspect of the internship made onboarding into a new organization challenging at times, but I quickly got up to speed with all of the systems and tools that DevEx (and SailPoint as a whole) use to keep team members in sync and keep collaboration flowing. Over time, our workflow adapted easily to remote and often asynchronous team work.
Almost all of the software DevEx maintains and builds is based on modern microservice architectures, which contrasted considerably from my past experience with traditional, monolithic applications. Because of this, I was faced with a steep learning curve and had to adapt and update my technical understanding of software engineering. Fortunately, I had some fantastic resources within the team who were always available to answer my questions. I soon realized that in order to deliver a successful project on time, quick-thinking and adaptability would be key. Over the course of the Backstage project, I have been exposed to, and required to learn and utilize, countless concepts, tools, and systems. From a personal perspective, the challenge of needing to learn so much, and then to use that knowledge to build a product in a short amount of time, has been immensely gratifying.
By the end of the project, our focus was set firmly on testing and deployment; and I realized how much had been achieved in such a short period of time. The Backstage project was a real-world opportunity to expose myself to every step in the life cycle of a cloud service. As soon as I became comfortable with our current stage of development, that stage was over and we were on to the next one. I was constantly pushed out of my comfort zone, and driven to continuously learn and build. Despite the challenges of working remotely in different timezones, I had gained the real-world experience I was looking for when I began this internship. I realized the importance of organization and communication in order to foster collaboration and teamwork, I saw how crucial continuous learning is to modern software development, and sensed how important an organization’s culture is in ensuring every individual is pushed to achieve their potential. Both as a software development project and a period of personal learning, my time with SailPoint’s DevEx team was a success, and a fantastic experience.
Weston Greene: Onboarding onto the DevEx team made it clear that there was a lot of responsibility placed on us to keep the SailPoint product, and its family of micro services, alive and maintained. On day 1 of my internship, I gained a different perspective on the idea of enterprise: instead of this idea of a single large team of software engineers that constantly update and work on a central codebase, there was a lot more ordered delegation of tasks, and a very structured team of engineers who could add, change, or remove things from a set of services that we are expected to support. And by extension, a different team is expected to support a different set of services, and so on.
The sense of scope I gained made the purpose of bringing Backstage to SailPoint a lot more clear, and the goals of the DevEx team much more outlined. Having only worked on codebases that I maintain myself, whether it involves independent web development for my lab, or personal projects for my own learning, working on a product like Backstage with some end-user importance was a transition from my previous methodology for building software — it became much more necessary to receive honest feedback and to work closely with the team to put this together based on the requirements of the DevEx team.
A large part of the value of working on Backstage was the exposure I got to many of the other micro services that existed within our scope, and that I would need to be aware of for the end users. Early on, I realized the software templates in the dashboard would have to include many services outside of the ones that were maintained just by the DevEx team — there are engineers from different platforms that can also directly benefit from using templates to stand up new services extremely quickly. As the scope of Backstage broadened, so did my exposure to the many other services in the SailPoint organization.
As we got closer to deployment, wrapping up a project was another learning curve I navigated. For all of the projects I worked on for myself, these projects are meant to be ongoing and updated for as long as I have some goal for them. In the timeline of our internship, we would need to get Backstage deployed and in a production instance for all platforms to see and use. This process was very insightful to how enterprise companies deploy their services, which can take a while due to the number of requests in the backlog. Hardening an app for security purposes and containerizing it was also something I did not have any experience with, and knowing how to prepare the app for the production pipeline is a valuable skillset I now have.
The internship has holistically been a rewarding experience, and working within the DevEx team meant that I learned a lot about what it means to make products for other developers that can improve their workflow, rather than for myself or a non-technical customer. Everyone on the team was very welcoming, and I enjoyed talking with each of them both inside and outside of meetings — they challenged us, but also were very approachable when we had questions or clarification. Overall, this showed me that it is possible to work on things that have some importance and actively contribute meaningfully in a large corporate environment, especially here at SailPoint.
