Elevating Microservices Testing and Development using DynamicEnv

Welcome back to our ongoing journey with DynamicEnv at Riskified. In my previous post, I talked about how DynamicEnv makes managing our large number of services and their needs for testing and development simpler.

In this post, I’ll show you how DynamicEnv has been a game-changer in our development and testing at Riskified. You’ll discover how we’ve enhanced our processes, particularly through efficient, simultaneous testing.

I’ll also delve into how we’ve smartly reused resources, a strategy that has led to significant cost reductions. By the end of this post, you’ll not only understand the mechanics of DynamicEnv but also how these strategies could be applied to your software development projects, potentially leading to similar improvements in efficiency and cost savings.

DynamicEnv Walkthrough

To see how DynamicEnv speeds up development, let’s start with a Team 2 example. Let’s say the team wants to test updates to Services B and C on their own schedule, without waiting for other teams. This self-sufficiency is crucial for fast and smooth development.

To do so, they need:

• A working version of service A (to generate service B’s traffic)
• PostgresDB with some valid data (a snapshot)
• AWS S3 bucket and AWS SQS Queue
• Service D as a dependency of C

To help them move forward, we’ll integrate PostgreSQL and Localstack to mimic AWS services. In addition to setting up a dynamic environment for Services B and C, we’ll also utilize the existing deployments of Services A and D within our cluster. This will allow us to reuse resources efficiently, optimizing our cluster utilization in the process.

This strategy is part of our focus on creating an environment that closely mirrors production, supporting isolated testing and development without impacting other operations.

Environment Structure

Our test environment is designed with a multi-layered architecture. Starting from the bottom, building blocks consist of databases and similar components that are essential for the services’ functionality and interaction. These are uniformly deployed and updated via Helm charts, providing a consistent testing backdrop.

The DynamicEnv layer sits above, pinpointing and configuring the specific services needed for the test, ensuring they’re correctly connected to these building blocks by modifying the necessary environmental variables.
Additionally, DynamicEnv enhances efficiency by deploying only the services that are undergoing modifications while continuing to utilize shared resources for the rest of the services that remain unchanged.

At the top, a testing framework oversees the entire operation, maintaining cohesion among all the parts.

Building Environment: Helm Chart Configuration

We manage each team’s or use case environment with a specific Helm chart.

For Team 2, we begin by setting up the chart.yaml. This includes PostgreSQL and Localstack as dependencies:

Note that these building block Helm charts are custom-built to suit Riskified’s specific needs, ensuring tailored and effective environment setups.

Customizing with values.yaml

The values.yaml in our setup is crucial for customizing the DynamicEnv. The DYNAMICENV_HEADER_VALUE plays a key role in traffic routing to the test versions, focusing on the request header value. For our test environment, we’re updating the images for Service B and Service C to their tested image versions

Additionally, PostgreSQL is set up with a pre-existing snapshot, pg-snapshot, for data persistence. Similarly, Localstack is configured with services like SQS and S3, and specific startup scripts are provided. Each building block, like PostgreSQL and LocalStack, is addressed as a service, named with the release name and a suffix indicative of its function.

Configuring the DynamicEnv Resource

DynamicEnv is designed to direct only relevant traffic to the test services. This is achieved using ‘istioMatches’, which filter traffic based on specific headers, and by defining ‘subsets’ for targeted services like Service B and Service C.

The DynamicEnv resource configures the environment by first taking the original service setup. It then overrides specific values through the environment variable section, as shown in the configuration example. This process ensures that services like Service B and Service C are correctly linked to the PostgreSQL and LocalStack services, tailoring the environment to precise testing requirements. This method exemplifies how DynamicEnv manipulates existing configurations to create a customized test environment.

In the setup, traffic targeting Services B and C with the specific header `x-dynamicenv=team-2-de-sdfsd4` is redirected to their new instances created for testing within the environment.

Piecing It Together: How DynamicEnv Facilitates Real-World Testing

In the context of our ongoing example with Services B and C, let’s explore how DynamicEnv enables effective real-world testing:

1. Defining the Environment: Developers create the environment using Helm charts, adding ‘PostgreSQL’ and ‘LocalStack’ as dependencies. The values.yaml and DynamicEnv resources are tailored for accurate service configurations.
2. Initiating Environment Creation: The CI/CD pipeline, upon detecting a code commit, triggers the creation of an isolated environment tailored to Team 2’s requirements.
3. Variable Customization: DynamicEnv adjusts environment variables for Services B and C to ensure proper connectivity with PostgreSQL and LocalStack.
4. Traffic Routing: Using Istio’s capabilities, the DynamicEnv operator configures traffic routing to ensure that only requests with specific headers (indicating they are part of the test) are directed to this new environment.
5. Automated Testing: Following environment creation, automated tests run, allowing immediate assessment of new features without affecting other operations.
6. Manual Testing and Debugging: If automated tests uncover issues, developers can manually interact with the environment for detailed debugging and problem-solving.

Efficient Cleanup Process and Cost Management

At Riskified, our efficient management of resources starts with creating several dozen new environments daily, a number that’s just the beginning and has the potential to grow rapidly. These environments are tailored to support the diverse requirements of each active GitHub branch, enabling our developers to work swiftly and effectively.

To maintain cost-effectiveness, these environments aren’t left running continuously. We have a scheduled cleanup process at each day’s end, eliminating any environments that are no longer needed. This efficient cleanup strategy keeps our monthly expenses to a minimum, showcasing our ability to equip our developers with essential tools while judiciously managing our budget, particularly as the adoption of DynamicEnv continues to expand.

DynamicEnv Delivers Proven Results

DynamicEnv has really changed the way we create software at our company. It’s more than just a simple tool — it’s a system that helps us set up and manage our services quickly and smoothly. With DynamicEnv, we’ve sped up our work, saved money, and still produced high-quality results. I encourage you to try out DynamicEnv for your own projects. It’s made a big difference for us, and might do the same for you.

Learn more about DynamicEnv and see how it can make your development easier.

For more information and to get started, have a look at our DynamicEnv Documentation.