Building your first Cadence Workflow
Motivation for using a new system
As our CTO Nino has mentioned, in the middle of 2019, we have started to look for a new system to help us manage more complex workflows. We have already been using Airflow extensively, but as we start building more products and having more complex features, there are some situations where Airflow just can’t cut it. If it is just one or two exceptional cases, we can definitely hack our way through them. But as these exceptions get more numerous, it is more crucial to find the right tool for the right job. We happen to chance upon Uber Cadence and after some brainstorming sessions and further research, we decided to dive in and give it a try.
So what is Uber Cadence and how do you set it up and use it to orchestrate your various microservices? Read on and hopefully by the end of this article, you will be able to pick up the following topics!
- A brief look at Cadence Architecture
- Getting the Cadence Service and WebUI up and running.
- Introduction to the Cadence CLI
- Set up a basic Worker
- Adding your very first Workflow
- Managing the worker via REST calls
Brief look at Cadence Architecture
For a detailed breakdown of the cadence architecture, you can read Cadence Deployment Topology. But if you just want a quick overview, here is a simple diagram of what we will be building.
A short description of all the various service is as follows. But do not worry if you do not understand them fully, I will go through them more at their respective section. Furthermore, you will also gain more understanding by working on the project and interacting with the cadence service.
Firstly, Cadence Service is the tool provided by the Uber Cadence team. It serves as the brain that manages the history of workflows, assigning tasks to workers, directing signals to the correct worker etc.
The Worker houses all our business logic and is built and maintain by engineering teams like us using the cadence client library. Activities are codes that perform logic such as an API call. By chaining activities together, we get Workflows. If your business logic gets too complicated, we can even further segregate the logic with Child Workflows but that is a topic for another time.
Cadence Web UI is a tool made by the Uber team to help teams like us visualize any running or completed workflows better in a fully fleshed out GUI. Through this GUI, we will be able to see the event history of workflows and even terminate them.
Cadence CLI is usually run from a docker container. It is this admin tool that allows us to manage the cadence service such as registering of domains and to start workflows.
Finally, the Go Http Server is a simple rest server that we will be building that allows us to manage our worker using REST calls. This allows our worker to be managed by other microservices. In StashAway, we even included a Kafka consumer in this server so our worker can be managed by Kafka messages.
Project Structure
Before we begin coding, here is the overview of how our project will look at the end.
The Cadence Service and Web UI
Let us start by getting the Cadence Service up on running in your local environment. The best way to run the cadence service locally is by using docker-compose. You can get the latest and official docker-compose file from the Uber team at this link. But we are going to make a tiny bit of change. In our example, we will choose to launch a cadence service without integration with elastic search. So let us create a docker-compose.yml file and fill it with the following code.
As we are running the database docker, we will also mount a volume to a local data folder at ./data/cassandra to make sure that the database data are not lost between every docker shut down. Make sure the folder exists. If not you can run the following command in the folder where your docker file is located: mkdir -p ./data/cassandra. The network config is optional, but I always like to name my network so if I am hosting other external services using another docker-compose command, such as an external Kafka, I will be able to link them all together in the default cluster network.
Now go to the folder where the docker file is located and run docker-compose up. Give it a while to boot up and when ready you should be able to visit http://localhost:8088 for the cadence web interface. If there are no error messages, then congrats! You managed to get the cadence service up and running!
Cadence Command Line Interface(CLI)
The first thing to do right after getting your service running is to create a domain. A domain in cadence is something to segregate different workflows into groups. You can create one domain for each workflow or just keep all workflows in one domain. I imagine that it makes the most sense to create a domain to group your workflows by types so it makes debugging easier. Eg. in StashAway, we can have a domain to manage onboarding workflows and another domain to manage trading workflows. To create a new domain, it is time to introduce you to a new tool built by the Cadence team, the Cadence Command Line Interface (CLI).
The Cadence CLI not only allows you to manage domains but also manage your workflows. During development, it is crucial to be able to have an easy way to start/terminate workflows and signal them and all these can be done by the CLI tool. However do note that CLI tool is more beneficial for development and in a production environment, it is still better to manage your workflows by writing your own API/UI and using the cadence client library, eg: https://godoc.org/go.uber.org/cadence.
To use the CLI tool, we will make use of the CLI docker image provided by the cadence team.
docker run --network=host --rm ubercadence/cli:master --do simple-domain domain register --rd 10This first part of the command docker run --network=host --rm ubercadence/cli:master tells docker to download the ubercadence/cli image from the master branch and run it in the host network. The --rm command will remove the container once you have finished the command so you will not end up with too many stray containers. --do simple-domain option tells the command to work in the domain called simple-domain and finally, domain register tells it to register the domain to cadence memory. It is important to change the --rd option according to how many days you want the retain the workflow data. Eg. If you enter 1, the workflow that has been completed will no longer be searchable in the cadence UI after a day has passed.
After running the command, you can now enter simple-domain in the search box of cadence web ui to view your new simple-domain.
Some other useful commands to keep in mind are commands to manage your workflows. To start a new workflow, you can
docker run --network=host --rm ubercadence/cli:master --domain simple-domain workflow start --tl <tasklistName> --wt <workflow_type> --et <execution_timeout> --dt <decision_timeout> -i '<input_value>'docker run --network=host --rm ubercadence/cli:master --domain simple-domain workflow terminate -w <workflowId> -r <runId> --re "<reason>"docker run --network=host --rm ubercadence/cli:master --domain simple-domain workflow signal -w <wid> -r <rid> -n <signal-name> -i '"signal-value"'
The commands look pretty long doesn’t it? If you are going to be using this a lot, I would recommend adding the following alias to your shell’s profile. Bash user should add it to the file ~/.bash_profiles and zsh users ~/.zshenv
alias cadenceCli='docker run --network=host --rm ubercadence/cli:master'# or if you only work with a single domainalias cadenceCli='docker run --network=host --rm ubercadence/cli:master --domain samples-domain'
make sure to update the profile by using source ~/.zshenv or source ~/.bash_profiles. Now we can shorten the commands to
cadenceCli wf start --tl <tasklistName> --wt <workflow_type> --et <execution_timeout> --dt <decision_timeout> -i '<input_value>'cadenceCli wf terminate -w <workflowId> -r <runId> --re "<reason>"cadenceCli wf signal -w <wid> -r <rid> -n <signal-name> -i '"signal-value"'
The Worker: Where your business logic lives
Now that we have our very first domain, let us start writing workflows! The workflow codes live in a program that we call the worker. The worker is a go binary when compiled that collects tasks to work on from the cadence service based on your predetermined tasklist. The good thing about this design is that you can scale up your worker and service independently. Eg. When you start having a lot of long-running workflows, the workers will most likely be the first to take the hit at performance. In such an instance, you will be able to scale up your workers to handle the workload without needing to scale up the main cadence service.
To not rely on the project being in the GOPATH, let us begin by creating a go.mod file. In the project root directory, run the command go mod init github.com/nndd91/cadence-api-example. Create a new directory call app and we can begin coding.
Let us begin by setting up the config struct that will allow us to specify some variables that are required in our application. For the sake of simplicity, we will use one struct for both our API server and worker. To be able to read the config from both a local file as well as environment variables, we will make use of the library Viper. Firstly, create a file called app/config/appConfig.go and add the following code
To finish up, we will add our config file to app/resources/application.yml
Connecting to Client in Code
Let us begin by creating a connection from the worker to the cadence service. In order to do so, we will need to build the client using Uber cadence client library. For a more complete client builder, you can refer to Uber cadence samples repository. The following code that we will be building is a condensed version of that so that it will be easier to run through and understand.
We will create a new file in app/adapters/cadenceadapter folder and name it factory.go. This file will allow us to create 3 different kind of cadence client that each has various methods.
The ServiceClient is an RPC service client that connects to the cadence service. It also serves as the building block of the other clients and will also be required when you want to spawn a new worker.
The CadenceClient is the main client that you will be using to interact with cadence. It consists of methods that allow us to manage our workflows and will be using 2 such methods in our example, namely StartWorkflow and SignalWorkflow. You will also be able to use methods like ListOpenWorkflow or QueryWorkflow to build an API to explore your running workflows.
Finally, the DomainClient will allow us to perform operations on the domains available in our cadence service. In our example, we will be using it to verify that the domain indicated in the config file is registered in our service. When it fails to verify it, it will log down an error.
To make sure that we create just a single client throughout our app life cycle, we will create a struct called the CadenceAdapter. It comes with a singular Setup method that is run once during application startup to initialise the config as well as the various cadence clients.
Building the worker
Finally, we will build the entry point to the worker. Create a new file called app/worker/main.go and copy the following code in. The main function will set up both the config as well as the adapter. We will then make use of the adapter to build and start the worker.
At this point, you will be tempted to start the worker but we have one more step! The workflow itself! Without workflows, your worker is just an empty container consuming resources and doing nothing!
Your very first Workflow!
Our first workflow will be a simple hello world workflow. Here is a quick summary of a few terminologies used in cadence.
Workflows
- Fault-oblivious stateful code is called workflow.
- Used to chain together activities.
- For more complex business logic, we can segregate this further into child workflows.
Activities
- Workflow fault-oblivious code is immune to infrastructure failures. But we need to connect to other services where failures are common. Hence codes that do all these are called activities.
- All communication with external service should be using activities. Eg API call, Kafka Messages, Slack messages.
- Think of it as the building blocks of Cadence.
Signals
- Sometimes we want to have some influence while the workflow is running
- Used to communicate with the workflow from external services such as Admin
When we run the program, first it will go through the init() function to register itself with Cadence Service. Once we start the workflow with the CLI, the workflow will first run the helloworldactivity which will ask the user how old you are. Once the activity has ran successfully, it will wait for a signal so the user can specify his age. We can then signal the workflow with our age using the CLI and the workflow will complete reporting the age that you have sent to it.
Wrapping up the worker
To build the worker, run the command go build -i -o bins/worker app/worker/main.go. The -o option allows us to specify a folder to build the binary file to and the last variable is the path to the main.go file of the worker. Once the build is completed, you can run the worker with the command ./bins/worker and you should see the following response.
We can now try starting the workflow with docker run --network=host --rm ubercadence/cli:master --domain simple-domain workflow start --tl helloWorldGroup --wt github.com/nndd91/cadence-api-example/app/worker/workflows.Workflow --et 999 --dt 60 -i '"YourName"'. If you see the following log, then congratulations! You got your first workflow running!
You can also view your current running workflow in the cadence UI at http://localhost:8080. In order to signal the workflow, you can use the command. Making sure to replace <wid> with the workflow id that you obtained from the previous response.
docker run --network=host --rm ubercadence/cli:master --domain simple-domain workflow signal -w <wid> -n helloWorldSignal -i '25'In the Cadence Web UI, you should now be able to see that your workflow is now completed!
HttpServer: Managing the worker via REST calls
As I mentioned previously, the Cadence CLI is great for testing of workflow in development. In a production environment, I am sure you would not want to start workflows by manually going into the server and typing long commands. That is why on top of the worker, we will build a simple http server that exposes 2 API endpoints to start the workflow as well as to signal the age to it.
Create a new file at app/httpserver/main.go
Build the server with go build -i -o bins/httpserver app/httpserver/main.go and run it with the command ./bins/httpserver
With this simple http server, we can now easily start and signal the workflow using 2 endpoints! To start the workflow, use your favourite REST client and make a post request to http://localhost:3030/api/start-hello-world. The response should be something like this
{
"ID": "8e1e251a-6603-46f9-b0bf-05c0fcb8baf1",
"RunID": "c2d68c3b-4786-4148-a240-2e467f57c9d2"
}Now check the Cadence web UI to see if the workflow has successfully started! If so we can now signal the workflow by making a post request to http://localhost:3030/api/signal-hello-world?workflowId=<wid>&age=25
Bonus Tip
One last tip. If you find it troublesome to always remember the command go build -o <folder> -i ..., fret not! We can actually simplify this by creating a Makefile at the root folder.
With this simple change, you can now build your binaries by running make httpserver , make worker or make bins to build them both at the same time!
If you made it this far, Congratulations! You should now have a cadence server running and a worker ready and waiting for a trigger to run the simple hello world workflow! Not to mention you have made it easier for your other teammates to run the workflow by exposing it through API with the httpserver! If you need, be sure to check out the full source code available here!
I hope that through the example, you will be able to better understand how to set up and begin writing workflows for cadence. The team at StashAway is excited about this new tool and hopefully, you will also be able to introduce this exciting new tool to your existing architecture as well!
We are constantly on the lookout for great tech talent to join our engineering team — visit our website to learn more and feel free to reach out to us!
