Announcing DataStax Stargate v2

In this post, we will reveal our new architecture for Stargate, an open-source data gateway that sits between your app and your databases. Stargate brings together an API platform and data request coordination code into one OSS project.

After a year of adding new APIs and features, we realized that this is an ideal time to take a step back and look at the overall architecture of Stargate. We’re listening to feedback from the Stargate developer community and taking a look at our technical debt as well. Based on your inputs, we’ve begun some planning for a “Stargate v2” and have identified the following high-level goals:

  • Breaking up the monolith
  • Improving developer experience for contributors
  • Making Stargate more cloud-friendly

We’ll expand on these goals below, but first, let’s take a look at how we reached this point.

Stargate: a brief history

In September 2020, we announced the Stargate project as an open-source data gateway. Our goal was to help speed up application development and reduce the workload for teams that find themselves creating and maintaining layers of microservice APIs on top of databases like Apache Cassandra®.

Since that first release, which included support for CQL and REST APIs, the project has seen a number of milestones since then. Here are a few of the highlights:

This represents a lot of change for a project that’s just over a year old, and now it’s time for us to set the stage for this open-source community to grow even more.

Breaking up the monolith

The current Stargate design is monolithic. Each Stargate node runs as a single process containing all of the supported APIs. This has a few consequences for deploying and running Stargate:

  • All of the APIs are enabled by default. You could run your own version with individual APIs disabled, but not by using the official Stargate Docker images. Instead, you’d have to download and build the Stargate source and selectively omit specific OSGI bundles from the startup script. This isn’t intuitive and requires knowledge of the existing bundles (more on OSGI below).
  • It’s not possible to scale the individual APIs. If you’re primarily using the REST API and need to scale up to meet increased REST traffic, you can’t just scale up REST independently. Instead, you have to add additional Stargate nodes supporting all the APIs to meet the demand. The nodes are “heavier” than they need to be, leading to inefficient resource usage.
  • Today, extending Stargate with a new API requires creating a new module. (adding the module’s JAR files to the existing stargate-lib directory, then packaging all the JAR files, and configuration up for deployment.)

As you can see, the current design makes Stargate more difficult for both users and contributors, and it’s time to break up this monolith to improve the experience for everyone.

Improving the developer experience for users

As you’d expect, we absolutely want the Stargate APIs to be as easy to use and adopt as possible, but our goals for v2 are specifically focused on making Stargate easy to deploy and manage, rather than on making API revisions.

Stargate v2 will move the API implementations out of the Stargate node (aka “monolith”) into separate microservices. This will allow you to scale each API independently. You can even disable APIs that you don’t intend to use entirely, allowing you to focus on a smaller surface area of the project and giving you fewer endpoints to secure.

Improving the developer experience for contributors

We’re also committed to growing the community of open source developers who are actively contributing code to Stargate. For this reason, another major goal of v2 is to make the implementation itself easier to understand, debug, enhance, and extend.

Removing OSGI

The original Stargate (“v1”) has a pluggable design, using the OSGI framework to allow different modules to be composed at deployment time. This is useful for swapping in different Cassandra persistence modules, such as Cassandra 3.11, 4.0, and DataStax Enterprise 6.8. We chose OSGI because it’s a proven technology that provided us with the plugin framework we needed to iterate quickly.

However, OSGI isn’t widely used these days, and several developers have reported that debugging can be difficult due to the nuances of working with OSGI. The move to microservices will help reduce the need for OSGI, and we’ll work toward replacing the remaining usage with another solution.

Encouraging modernization and innovation

The current requirement to support a Cassandra 3.11 persistence layer means that the Stargate nodes must run Java 8 — the lowest common denominator. Unfortunately, this has prevented the usage of more modern Java frameworks, such as Quarkus or Micronaut that require later Java versions.

We believe that one of the great benefits of breaking up the monolith will be to encourage innovation in the Stargate community. The v2 architecture will be more conducive to external contributions since the developer of each microservice will be able to make their own choices about what language and frameworks to use. This has the dual benefit of both modernizing Stargate and potentially attracting contributors who are interested in working with new and exciting frameworks or languages.

Refactoring and cleanup

As with any project that has many contributors, the codebase has started to show some entropy. Although it’s nowhere near “spaghetti code” state, the modules and Java package structure could use some refactoring. We’re hoping to clean up some dependencies and improve the overall code organization to make it easier for you to find your way around.

Making Stargate more cloud-friendly

While the Stargate project currently provides Docker images that can be used for deployment in containerized environments, Stargate v2 is going to add additional images for the various microservices. This will add some complexity to deployments.

Since Kubernetes has emerged as the clear choice for container orchestration in both public and private clouds, v2 will provide assistance in deploying Stargate on Kubernetes via Helm. We envision providing a Helm chart to expose each enabled API as a Kubernetes Service, using Deployments to manage replicas of the API implementations, creating a StatefulSet to manage the persistence nodes, and so on.

On another note, we look forward to continuing our collaboration with the K8ssandra project, which provides deployments of Stargate and Cassandra on Kubernetes. The K8ssandra team is also building a Stargate Controller as part of the K8ssandra Operator for their own v2 release.

The journey to V2 is just beginning

We’re just getting started on working toward a Stargate v2 release and are committed to sharing our plans and progress early and often. We’d love to get as much input as possible from the Stargate open source community. In our next blog post, we’ll introduce a proposed design for Stargate v2, other design options we considered, and share how you can provide feedback on the proposal.

Follow DataStax on Medium to keep up with our latest announcements and developer resources on all things Apache, Cassandra, Stargate, and more. To join our growing developer community, follow DataStaxDevs on Twitter.

Note: This article first appeared on Stargate.io.

Resources

  1. Stargate.io
  2. Stargate Community
  3. Stargate and Cassandra 4.0 — Better Together
  4. Stargate APIs | GraphQL, REST, Document
  5. K8ssandra, Apache Cassandra® on Kubernetes

--

--

--

We’re huge believers in modern, cloud native technologies like Kubernetes; we are making Cassandra ready for millions of developers through simple APIs; and we are committed to delivering the industry’s first and only open, multi-cloud serverless database: DataStax Astra DB.

Recommended from Medium

A Beginner’s Guide To Regular Expression Operations In Python

Those 11 HTML project ideas for beginners will help you to start creating amazing front-end…

Season 2. Weather conditions.

Galaxy Shooter — Phase I: Framework

Content Management Systems — A Deep Dive

Make your own music Discord bot for FREE

How to wisely choose the right AWS Direct Connect for your hybrid cloud environment

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DataStax

DataStax

DataStax is the company behind the massively scalable, highly available, cloud-native NoSQL data platform built on Apache Cassandra®.

More from Medium

How to Set up a Kubernetes CronJob to Create Medusa Backups

Using the Apache APISIX proxy gRPC service

Spinning Apache Kafka® Microservices With Cadence Workflows

The Next Generation Database