Dynamic Environment: Streamlining Testing and Development in Microservices

Managing dependencies and testing environments has become an extreme challenge in the fast-changing world of microservices. At Riskified, we understand these challenges all too well, particularly at the scale we operate.

With more than 1,700 Kubernetes services across two expansive Kubernetes clusters, efficient, on-demand testing and development environments became a critical priority for us.

This blog post explains the journey and significance of a novel open-source tool called Dynamic Environment (DynamicEnv), our two-year venture aimed at addressing the chaos in microservices development and testing.

The Microservices Challenge

Working with microservices, we have faced a significant challenge when it comes to testing and developing software, that is, the complexity of ensuring that they all work seamlessly together.

As we work on new features or bug fixes, we are constantly challenged to ensure they seamlessly integrate with all other services and flows of our system. Specifically, deploying a new version to a subset of services while testing it against all others adds an extra layer of complexity to this puzzle.

Creating dedicated environments for each testing scenario seems like an obvious solution, but at our scale, it’s impractical. The large number of services and their complex interdependencies make any attempt to set up isolated environments for every feature or test iteration a resource-intensive nightmare.

This not only increases the development timeline but also strains resources, leading to increased operational costs and the need for significant support from our infrastructure engineers. The notable cost and resource impacts prompted us to seek a more efficient alternative.

This brings us to the heart of our innovation — the DynamicEnv project.

Essential Needs Influencing the Creation of DynamicEnv

Before we explore this innovative solution and delve into the challenges we face, let’s go through a list of important needs for our testing and development environments:

• Cost efficiency: Optimize resource utilization and reduce costs associated with maintaining and managing testing environments.
• Kubernetes configuration consistency: Ensure that the K8s configuration in testing environments matches our real production to avoid surprises.
• Isolation: Guarantee isolation in testing environments to avoid problems between services and ensure accurate testing results.
• Observability: Observe our services’ behavior while testing with suitable tools.
• Unified solution: Use one system for both tests and development, making it easier for everyone.

The DynamicEnv project emerged as the answer to each of these critical requirements, revolutionizing how we approached microservices development and testing.

DynamicEnv — an Istio-Centric, Resource-Efficient, Testing Environment

DynamicEnv is an innovative Kubernetes operator designed to empower developers by simplifying the process of launching on-demand environments. At its core, DynamicEnv offers a streamlined approach to deploying and testing different versions of specific services within a shared Kubernetes cluster.

It effectively deals with the issues mentioned by creating special versions just for the services being tested. At the same time, it makes smart use of the ‘production’ versions for the rest of the environment, ensuring efficient resource use. The traffic is cleverly directed, sending ‘real’ users to the ‘production’ version of services and directing the traffic of a test user to the specific test version.

With DynamicEnv, developers can efficiently create on-demand environments, deploy various service versions, and seamlessly handle dependencies, all while optimizing resource utilization and reducing operational costs.

DynamicEnv harnesses the capabilities of Istio, a powerful service mesh, to achieve this goal.

What is Istio?

Istio acts as a service mesh, improving communication among microservices within a Kubernetes cluster. It offers essential features like traffic management, security, and observability to ensure the reliable and efficient operation of services. In this context, we utilized Istio’s routing capabilities to enable the intelligent traffic routing mentioned earlier.

Testing a New Version with DynamicEnv

Let’s go through an example!

Our chosen application is a customized adaptation inspired by the BookInfo application, a standard example with Istio. For more comprehensive information about Istio’s BookInfo application, refer to the official application page.

The image above shows how traffic flows between the services, where ‘shared’ indicates the last stable release version.

Now, let’s consider a scenario where you’ve developed a feature requiring changes in the ‘details’ and ‘reviews’ services, and you wish to test it before making it accessible to all users. In this example, you’ve updated the relevant services and uploaded a Docker image to the registry. To test your application, all you need to do is create a simple manifest for the dynamic environment:

apiVersion: riskified.com/v1alpha1
kind: DynamicEnv
metadata:
  name: dynamicenv-sample
spec:
  istioMatches:
    - headers:
        user:
          exact: test-user
  subsets:
    - name: details
      namespace: my-namespace
      containers:
        - containerName: server
          image: docker.io/istio/examples-bookinfo-details:test-version
    - name: reviews
      namespace: my-namespace
      containers:
        - containerName: server
          image: docker.io/istio/examples-bookinfo-reviews:test-version

Let’s go over this together:

‘istioMatches’ holds the instructions on how Istio should route traffic in the DynamicEnv — and in this example, when the ‘user’ header is set to ‘test-user, it’ll be routed to the tested versions.

‘Subsets’ define the microservices included in the DynamicEnv, specifying the deployment’s name, namespace, container details, and image for the tested version.

We optimize resource usage by deploying only the services requiring changes, using shared cluster resources for the unchanged services.

Simple enough, isn’t it? But there’s more. This solution allows multiple developers to deploy multiple versions of their services (even the same ones) while routing the traffic through the correct testing path using those unique headers.

Why DynamicEnv Matters

On-Demand Environments

DynamicEnv significantly accelerates microservices development and testing by facilitating the swift creation of on-demand environments.

The on-demand capability enables faster cycle times and a more agile and efficient development workflow, allowing teams to seamlessly adapt testing environments to specific needs.

Cost Efficiency

DynamicEnv isn’t just about solving problems; it’s about doing so cost-effectively, optimizing resources, and ensuring a sustainable development and testing infrastructure. We use our resources well and keep costs down.

Saving Time

DynamicEnv cuts down the time spent on environment setup, allowing our teams to focus on what matters — building and testing high-quality software.

DynamicEnv only spins up the services under tests, while keeping the rest of the cluster’s workload shared among developers and use cases.

Ensuring Reliability

DynamicEnv improves reliability by organizing how services work together, reducing risks associated with testing in complex microservices environments. It does this by using configurations just like those in our production system, ensuring dependable results.

Growing With Us

As our services grow, DynamicEnv enables us to add more services to our tests without breaking the ever-complex testing environment.

Explore DynamicEnv Yourself, It’s Open Source!

For those eager to delve deeper into DynamicEnv, it’s an open-source initiative. Visit our GitHub repository for the latest updates, detailed documentation, and the opportunity to engage with our vibrant open-source community. We welcome your contributions and feedback.

The Journey Ahead

With DynamicEnv, we’re not just solving problems; we’re paving the way for a future in microservices development and testing that is more efficient, reliable, and scalable. In our upcoming blog posts, we’ll explore the features, implementation, and real-world impact of DynamicEnv in more detail.

Join us on this exciting journey as we embrace the challenges of microservices and champion innovation in development and testing.

Open Source Collaboration

Riskified proudly maintains DynamicEnv. We value the teamwork within our team that improves and expands this project — a special thank you to our dedicated maintainers for their valuable contributions. You can discover more about our contributors by checking their profiles on LinkedIn:
Haim Ashkenazi, Adi Gal, Yonathan Shtekel and Nissan Itzhakov.