Building our Midgard “Valkyrie”

In the fast-paced world of software development, the quest for building scalable, resilient, and efficient distributed systems is ever-ongoing. As developers navigate through the complexities of modern architectures, microservices (or Valkyries as we like to call them in the Heimdall platform) serves as a way for us to excise “imperative control in a declarative context”. In this article, we embark on a journey into the realm of Valkyries, looking at how we decided to implement our latest Midgard Valkyrie — a formidable entity crafted on the bedrock of the Heimdall .NET microservice template and intricately woven into the fabric of Backstage via an entity provider, with Kafka orchestrating the symphony of inter-service communication.

Before delving into the intricacies of how we did it, lets take a second to survey the technological landscape that forms the foundation of our new Valkyrie:

Heimdall templates: At the heart of our endeavor lies the Heimdall .NET microservice template, a collection of battle-tested design pattern implementations and abstractions for .NET microservice development in our Heimdall platform. The template offers a plethora of features, including configuration management, authN/authZ, telemetry, dependency injection, health checks and more to allow developers to kickstart their development with ease.

Backstage: At the outermost periphery of the Backstage Software Catalog lie the entity providers, serving as the originators of entities that establish the foundation of the processing hierarchy. Entity providers possess distinctive characteristics: they are instantiated individually via backend code and passed to the catalog builder, often with one provider instance corresponding to a remote system. Entity providers wield the capability to execute detailed updates on the set of entities under their jurisdiction, including full set updates or individual additions and removals.

Kafka: As the backbone of our distributed architecture, Kafka emerges as the quintessential enabler of asynchronous, fault-tolerant communication thanks to its pub/sub messaging model and robust scalability. It serves as the link between the Valkyrie and Backstage to enable asynchronous communicate, to achieve loose coupling and horizontal scalability.

Putting it all together

One of the critical aspects of microservices architecture is communication between services. Traditional approaches involve direct API calls between services, which can lead to tight coupling and scalability challenges. To overcome these issues, the entity processor, combined with Kafka as an event broker, offers a decoupled and scalable communication mechanism between our “Valkyrie” and Backstage, which leaves us with something like this:

The flexibility of the Backstage entity provider architecture offers an intuitive interface for handling domain-specific operations, such as CRUD (Create, Read, Update, Delete) on entities in the catalog. By integrating the Midgard Valkyrie integration events, which are emitted upon succesful execution of its Scaffolder API, with the Backstage entity provider our Midgard capability as a whole achieves a higher degree of flexibility and scalability between the subsystems that comprises its inner architecture.

Furthermore when a CRUD operation is performed on an entity within the Backstage subsystem, such as creating a new user or updating a product, an event is published to a corresponding Kafka topic that is processed by the Valkyries worker.

The practical steps we took to build our Valkyrie where as follows:

1. Setup: First we cloned the Heimdall Templates repository for .NET microservices from NovoNordisk-OpenSource GitHub.
2. Customization: Then we customized the template according to our project requirements, including configuration settings, database connections, and service endpoints.
3. Business Logic: With all the scaffolding done we proceeded to implement our business logic within the domain, focusing on the core functionality of our application.
4. Infrastrucute Logic: Having composed our application we could then proceed to implement infrastructure and host assemblies, which in turn was containerized and orchestrated for local development using docker compose.
5. Integration with Backstage: Lastly we forked Backstage and implemented a custom backend module to facilitate interoperability with our Heimdall platform. Starting with the initial Midgard entity provider.

Conclusion

The integration of the Valkyrie into our Midgard capability not only demonstrates the prowess of modern microservice architecture but also underscores the importance of decoupled communication for scalability and flexibility. By embracing event-driven architecture facilitated by Kafka, coupled with the versatility of the Backstage entity provider architecture, the Valkyrie strikes a harmonious balance, in a world of trade-offs, where domain-specific operations and system-wide scalability seamlessly coexist. Through meticulous setup, customization, and integration steps, the development journey showcases a paradigm shift towards building distributed systems for our Heimdall platform that are not only resilient and scalable but also adaptable to evolving business needs in the dynamic landscape of software development.