Easily Manage Workflows at Scale with Temporal.io and Astra DB

This is the first in our three-part series on how to connect Astra DB, a managed cloud-native database with open-source Temporal.io to more easily manage even your heaviest workloads with the high performance and reliability of Apache Cassandra®.

Managing your workflows has never been easier using open source Temporal.io as your microservices platform. With Temporal’s quick start installation, getting started is simple plug-and-play for anyone wanting to use it with their own personal workflows.

Currently, Temporal provides different configurations using different databases and dependencies, including Apache Cassandra. DataStax Astra DB, which is built on Cassandra, allows you to add a managed, cloud-native database backing your Temporal platform that will support your heavy workloads while providing high performance and reliability.

In this first post, we introduce you to Temporal and Astra DB and illustrate the advantages of using these two powerful technologies together to manage your workflows reliably and with less effort.

What is Temporal?

Temporal is an open source, distributed and scalable workflow orchestration engine capable of running millions of workflows. Workflows can hold state and describe which activities (workflow tasks) should be carried out.

Illustration of a Temporal Cluster, which consists of four independently scalable services.
Figure 1. Illustration of a Temporal Cluster, which consists of four independently scalable services. (Source: Temporal.io)

The Temporal “system” consists of a Temporal Server (either the Temporal Cloud service or self-hosted) orchestrating work with a fleet of Temporal workers (operated by application developers), and Temporal Clients (embedded in applications) over gRPC.

These workflows are carried out by the Temporal (or self-hosted) server which consists of four independently scalable services:

  • Frontend gateway (rate limiting, routing, authorizing)
  • History subsystem to maintains data (mutable state, queues, and timers)
  • Matching subsystem to host task queues for dispatching
  • Worker service to handle the internal background workflows

Temporal Clients embedded within your app (route handlers or serverless functions) can start, cancel, signal, and query individual workflow executions. Activities are distributed using task queues and executed on worker nodes organized in a cluster.

A Temporal cluster is the Temporal Server paired with a persistence layer (i.e. the data access layer). The workflow data is stored in its respective backend depending on what you choose to configure Temporal with. Supported databases include PostgreSQL, MySQL, CockroachDB, and Cassandra.

One aspect of the Temporal system is that it abstracts the complexity of a distributed system. Distributed systems are known to scale computation across multiple machines and handle the potential load of system changes. In theory, a distributed system facilitates a reliable and highly performant application. Application developers don’t have to worry about handling failures because that’s handled by the engine.

However, any failure that leaves the downstream part of the application waiting for a response can make things very complicated, especially at a large scale.

Temporal Persistence Layer

Temporal supports Cassandra, MySQL, and PostgreSQL schemas and can be used as the server’s database. The database stores the following types of data:

  • Tasks to be dispatched
  • The state of Workflow Executions
  • The mutable state of Workflow Executions
  • Event History, which provides an append-only log of Workflow Execution History Events
  • Namespace metadata for each Namespace in the Cluster
  • Visibility data, which enables operations like “show all running Workflow Executions”
Parts of a Temporal Cluster stored in a database.
Figure 2. Parts of a Temporal Cluster stored in a database (Source: Temporal.io).

Why Cassandra and Temporal work so well together

In addition to the fact that Temporal supports Cassandra schema, Temporal is write-intensive and can leverage the lightweight transactions (LWT) feature of Cassandra to provide significantly more database capacity. With Temporal, 50% of the writes are LWT making it ideal for heavily distributed applications that experience high load. The architecture of Cassandra allows it to easily scale horizontally to handle very high loads.

Cassandra provides unparalleled performance and reliability and is a highly scalable, fault-tolerant database. When you combine Temporal with Astra DB, you get all the benefits of Cassandra with the ease of use that comes with a multi-cloud, serverless database-as-a-service (DBaaS).

Astra DB simplifies cloud-native Cassandra application development

Cassandra is the open-source NoSQL database behind some of the largest applications in the world, including Netflix and Instagram. Astra DB is built on Cassandra to simplify cloud-native Cassandra application development. Using Astra DB takes care of the heavy lifting often associated with using a powerhouse database like Cassandra, reducing deployment time from weeks to minutes.

Astra DB creates the Cassandra database for you. Pairing Astra with your Temporal workflow will make carrying out your application development much easier, increase performance, and optimize your time. Astra also provides you with a convenient dashboard where you can visualize your database usage, health and other metrics.

Ready to dig in?

Astra DB makes leveraging the benefits of Temporal easy. In our next post in this series, we’ll show you how you can connect Temporal with Astra DB in just five easy steps:

  1. Astra prerequisites
  2. Temporal pre-setup
  3. Temporal schema migration to Astra DB
  4. Persistence Layer configuration
  5. Test and validate

You don’t have to wait until our next post to get started though. Sign up for your free Astra DB account now and start exploring how it can benefit your application development. And, be sure to check back here to learn more and get the details for connecting your Astra DB with Temporal.

Follow the DataStax Tech Blog for more developer stories. Check out our YouTube channel for free tutorials and DataStax Developers on Twitter for the latest news in our developer community.

Resources:

  1. Temporal
  2. Apache Cassandra
  3. DataStax Astra DB
  4. Introduction to Temporal
  5. What is a Worker Process?
  6. What is a Temporal Cluster?
  7. Tour of Temporal: Welcome to the Workflow
  8. What is a Task?
  9. What is a Workflow Execution?
  10. What is an Event History?
  11. What is a Namespace?
  12. Lightweight transactions (LWT) feature of Cassandra

--

--

--

We’re huge believers in modern, cloud native technologies like Kubernetes; we are making Cassandra ready for millions of developers through simple APIs; and we are committed to delivering the industry’s first and only open, multi-cloud serverless database: DataStax Astra DB.

Recommended from Medium

Leave it for the next one.

Satori for Engineers

FindFurryFriend (part 3) — rails api (controllers and serializers)

The Differences Between Static and Dynamic Libraries.

How to write Clean, Beautiful and Effective C++ Code

A crash course on floating point numbers in a computer, Part I

July LeetCoding Challenge

Optimising tceetree for the Linux kernel

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DataStax

DataStax

DataStax is the company behind the massively scalable, highly available, cloud-native NoSQL data platform built on Apache Cassandra®.

More from Medium

Proven: Starlight for RabbitMQ and Apache Pulsar Offers 10x Better Throughput

3.0.0 Alpha Release !

Healthy Clusters with Kaffia, an Intuitive Apache Kafka Visualizer and Administrator

Using Timeplus and Redpanda for low latency, real-time streaming analytics