Beacon Mode — how we are solving one of the most common problems in microservices development
Authors: Vasil Shlapkou and Jeff Upton
Building a Scalable Microservice Development Architecture
Microservices have become the industrial standard for SaaS enterprise solutions. A microservice is a small, deployable application which is usually designed to handle one specific business task and handle it extremely well. By controlling the number of microservice instances in your production environment you should be able to horizontally scale your applications as needed.
Microservice development is not an easy task. Legacy monolithic application developers can setup, start and debug applications directly on their workstation, comfortably set breakpoints, check application logs, reproduce issues locally and debug the code.
What if your environment needs to run hundreds or thousands of microservices owned by different teams with different functionality? By implementing an event-based approach, communication between microservices became asynchronous and eventually consistent. Code defects and errors became very difficult to troubleshoot, reproduce and fix.
How can organizations establish reliable and consistent development practices for their development teams when working in a microservices-rich environment? Below is SailPoint’s journey to our current development process.
Docker as a Monolithic Application
Wouldn’t it be nice to have all microservices running in your development environment and be able to set breakpoints in any services you want at any time? Is it possible at all? The answer is yes in theory. If you are packaging your microservices in container images by using container technologies (e.g. Docker) you can run all your services in your local workstation environment with Docker. You will be able to start, stop services, update microservice images, connect with IDE debugger, and follow traditional development practices.
At SailPoint, we started with this approach, using Docker Desktop and local development environments with a full set of microservices and supported infrastructure: Redis, Kafka, MySQL, etc.
It was working great for a small set of microservices. However, over time, the microservice architecture complexity of our SaaS product drastically increased. The major downside is that all services run locally. With the average developer laptop having 16GB of memory, some developers dedicate 10–12GB of RAM to the dev stack (and even then, disable many fringe services). This creates a horrible developer experience where there are not enough system resources left to run all of the other required applications (IDE, Editor, Outlook, Browser, Slack, etc) without creating system pauses and swapping. To combat this, the developers simply ran a subset of the services and mentally filtered out failures that would crop up in the UI and dependent services. This was far from ideal.
Another approach is to use a form of network redirection to send traffic normally destined for your cloud-deployed containers to local container instances. At SailPoint, our microservices are multi-tenant, and developers each have one or more tenants they use to develop and test their work. Simple network redirection falls short when your containers service multiple tenants, yet you want to direct traffic for only a single developer’s tenant to their local workstation.
Introducing Beacon
Faced with these challenges, we focused on the developer experience. Developers want cloud-running production-level microservices for the vast majority of the environment — the services they are not themselves working on at the moment. At the same time, they want the local development experience and tools for the specific microservices they are working on.
At SailPoint, we created our own framework called “Beacon”. Beacon lets developers attract cloud traffic for their specific tenant to their local workstation microservice instance while leaving everything else running in the cloud environment. It works as follows:
- Developers set an environment variable with their test tenant name in their local environment.
- On service startup, the Beacon framework registers the development service. HTTP, Redis, and Kafka traffic for this one tenant flows to the local instance. We call this “Beacon mode”.
- On shutdown (or failure to heartbeat), traffic redirects to the default cloud containers running production code.
The image below shows the state of the development of cloud environments for a Microservice “X” developer (The owner of tenant “C”). All other tenants route to the production-mirror clusters. Microservice “X” requests to tenant “C” will route to the developer local instance.
How Beacon does it
The Beacon framework includes the following sub-modules.
Configuration
When service is started in Beacon mode, it needs access to all of the same configurations that the service in the cloud environment has. This includes JWT keys, database credentials, Redis host information, etc. Our microservices parameters are provided via our platform config service. Our DevOps team keeps all this configuration up to date.
DynamoDB as a “Development Server Registrar”
On startup, the Beacon framework writes a record to DynamoDB with an expiration time. A heartbeat thread is started and updates TTL field in the DynamoDB table. When the record expires, the tenant is reverted back to cloud mode.
Intercepting HTTP Traffic
For routing HTTP traffic, the Beacon framework provides a custom implementation for our Service Locator interface. The shared DynamoDB instance provides a server registration module for Beacon specific implementation, which is able to route all HTTP traffic to the appropriate local development host. We are using the same implementation for inter-process communication and in our API Gateway microservice.
Intercepting Redis Messages
In our platform, we use Redis as a message broker. The platform provides a level of abstraction on top of Redis messaging. The Beacon framework added a custom broker implementation that recognizes some services and tenants running in Beacon mode. It opts out of processing queues that are tied to tenants running in Beacon mode for a particular service while processing queues for non-Beacon mode tenants normally in cloud instances.
Intercepting Kafka Events
In SailPoint we are using an event-based framework which is an abstraction around pub/sub with an implementation based on Kafka — our microservices are not directly using Kafka APIs. As a result, the producers need not know if consumers are in Beacon mode or normal mode.
In a few cases we are using a dedicated topic per tenant. That’s straightforward — we assign all partitions of a dedicated topic to a Beacon-mode instance consumer with a custom partition assignor.
In all other cases we are using a single topic per environment. All tenants are collocated on the same partitions within a topic. Using a custom partitioning strategy, assigning 2 dedicated partitions per Beacon-mode tenant service. The local instance then uses those two partitions only, while non-Beacon tenants use the remaining partitions.
Conclusion
After rolling out Beacon mode across our development organization, we received a lot of positive feedback. Our developers were more than happy. All they need to do is just to clone source code from GitHub, add one extra environment parameter in the IDE, and they can start debugging service code as part of the cloud cluster.
We were also reminded of a few challenges with distributed microservices. VPN speed is very important, especially if your service is doing a lot of heavy lifting with RDS. As we’ve increased the number of developers using Beacon mode, we’ve had to increase the number of Kafka topic partitions as well, for each developer’s services to get their own partitions, which doesn’t exactly match our production settings anymore.
Seeing how well it worked for developers, we are now adding Beacon to our CI/CD pipelines. Each Pull Request Build can run end-to-end tests on a candidate container and test tenant, and capture failures pre-merge. The same dynamic per-tenant routing technology can be applied in production also, to temporarily redirect large workloads for specific tenants to well-resourced containers for individual microservice instances.
Beacon has quickly become our standard way to locally develop and test microservices as part of our larger cloud-native product.
