Using Temporal to Build Scalable and Fault-Tolerant Applications in Golang

we will cover how to build ready-to-run services using Temporal in Golang. This Tutorial will teach you how to design and build fault-tolerant and scalable services that are suitable for use in distributed systems.

Step 1: Run Temporal Locally with Docker

If you want to run Temporal locally for testing and development purposes, you can use Docker. The Temporal team provides an official Docker image that you can use to run the entire Temporal stack locally.

To get started, make sure you have Docker installed on your machine. You can download Docker from the official website.

Once you have Docker installed, you can use the official docker-compose setup provided by Temporal to run the entire Temporal stack locally in a few simple commands.

To get started follow temporal Quickstart for localhost development, or download the docker-compose setup files by running the following command:

curl -o docker-compose.yml https://raw.githubusercontent.com/temporalio/docker-compose/master/docker-compose-cas.yml

This will download the docker-compose setup for Temporal Community Edition with a Cassandra data store.

Next, start the services by running the following command:

docker-compose up

This command will start the necessary services, including the Temporal server, the Cassandra database, and the Temporal Web UI for monitoring workflows. Once the services are started, you can use the Temporal CLI to interact with the Temporal server, and the Temporal Web UI on http://localhost:8080 to monitor your workflows.

docker-compose exec temporal-server tctl namespace list

To stop the services, run the following command:

docker-compose down

This will shut down and remove all of the containers defined in the docker-compose.yml file.

Running Temporal locally with Docker is a great way to develop and test your workflows before deploying them to production.

Step 2: Set up your workspace

Create a new project in Golang and import the Temporal SDK using the following commands:

Init Go Mod

go mod init goenv // you can change your project name goenv -> your project name

go get go.temporal.io/sdk

Step 3: Design your workflow

Define the long-running business process you want to orchestrate using your Golang code. Use Temporal’s API to model your workflow, making sure to handle exceptions and failures appropriately. Here’s an example workflow to retrieve and process weather data:

package workflow

import (
 "goenv/activity"
 "goenv/messages"
 "time"

 "go.temporal.io/sdk/workflow"
)

// define the workflow function
func WeatherWorkflow(ctx workflow.Context, cityName string) ([]messages.WeatherData, error) {
 options := workflow.ActivityOptions{
  StartToCloseTimeout: time.Second * 5,
 }
 ctx = workflow.WithActivityOptions(ctx, options)

 // start the activities
 currentWeatherFuture := workflow.ExecuteActivity(ctx, activity.GetWeather, cityName)

 // wait for activities to complete
 var current messages.WeatherData
 if err := currentWeatherFuture.Get(ctx, &current); err != nil {
  return nil, err
 }

 var response []messages.WeatherData
 // combine results
 response = append(response, current)

 return response, nil
}

Step 4: Define your activities

Define the individual tasks that make up your workflow using Temporal’s API. These activities should be encapsulated in self-contained functions, which should be idempotent and retryable. Here’s how to define an activity to get weather data:

// activity/main.go

package activity

import (
 "context"
 "goenv/messages"
 "goenv/store"
)

func GetWeather(ctx context.Context, cityName string) (result messages.WeatherData, err error) {
 result, err = store.GetCurrentWeather(ctx, cityName)
 if err != nil {
  return result, err
 }
 return result, nil
}

Step 5: Implement your workflow and activities

Write the Golang code to create your workflow and activities, using Temporal’s API to ensure proper execution and handling of exceptions. Here’s how to implement the example workflow:

// workflow/main.go

package workflow

import (
 "goenv/activity"
 "log"

 "go.temporal.io/sdk/client"
 "go.temporal.io/sdk/worker"
)

func main() {
 c, err := client.Dial(client.Options{})
 if err != nil {
  log.Fatalln("unable to create Temporal client", err)
 }
 defer c.Close()

 w := worker.New(c, "weather", worker.Options{})
 w.RegisterWorkflow(WeatherWorkflow)
 w.RegisterActivity(activity.GetWeather)
 // Start listening to the Task Queue
 err = w.Run(worker.InterruptCh())
 if err != nil {
  log.Fatalln("unable to start Worker", err)
 }
}

Step 6: Handle failed tasks

In a distributed system, errors are inevitable. One of the most important things to keep in mind when building a system with Temporal is to handle errors correctly. You must have a plan in place for when things go wrong, such as network connectivity or user input errors.

When a task fails and an error is returned, there are two approaches: retry the task or ignore the error. The approach you choose depends on the type of error and how critical the task is to the workflow.

To retry a failed task, you can use the NewRetriableApplicationError method of the Temporal SDK. NewRetriableApplicationError takes three parameters: a message string, a task token, and an error. The task token is a unique substring used to identify the specific task.

Here’s an example:

// activity/main.go

package activity

import (
 "context"
 "goenv/messages"
 "goenv/store"

 "go.temporal.io/sdk/temporal"
)

func GetWeather(ctx context.Context, cityName string) (result messages.WeatherData, err error) {
 result, err = store.GetCurrentWeather(ctx, cityName)
 if err != nil {
  return result, temporal.NewApplicationError("unable to get weather data", "GET_WEATHER", err)
 }
 return result, nil
}

In this example, if the call to GetWeather fails, the task is marked for retry with a task token of retry-getweather and the error is returned.

On the other hand, sometimes errors are not important to the workflow and can be ignored. In that case, you can return a non-nil error from the activity function, and that error will be logged, but the workflow will continue. For example, if GetWeather is not able to find weather data for the specified city, and that data is not critical for the workflow, you can simply return a non-nil error and let the workflow continue.

func GetWeather(ctx context.Context, cityName string, result chan<- messages.WeatherData) error {
    weather, err := store.GetWeather(cityName, weatherType)
    if err != nil {
        logger.Warnf(ctx, "failed to get weather data: %v", err)
        return nil
    }
    result <- weather
    return nil
}

In this example, if GetWeather returns an error, the error is logged, and the activity returns a nil error. This will cause the workflow to continue running without that specific weather data.

Make sure to handle errors appropriately and test your application regularly to ensure that it is resilient to failures.

Step 7: Test your system

Use Temporal’s SDK to test your system, simulating failures and testing fault tolerance. You can use the TemporalTestSuite utility provided by the Temporal SDK to test your system. Here’s an example test:

// workflow/test/workflow_test.go

package test

import (
 "goenv/activity"
 "goenv/messages"
 "goenv/workflow"
 "testing"

 "github.com/stretchr/testify/mock"
 "github.com/stretchr/testify/require"
 "go.temporal.io/sdk/testsuite"
)

func TestWeatherWorkflow(t *testing.T) {
 // Set up the test suite and testing execution environment
 testSuite := &testsuite.WorkflowTestSuite{}
 env := testSuite.NewTestWorkflowEnvironment()

 // Mock activity implementation
 env.OnActivity(activity.GetWeather, mock.Anything, mock.Anything).Return(messages.WeatherData{
  Temperature: 41,
  Humidity:    80,
  WindSpeed:   4,
 }, nil)

 env.ExecuteWorkflow(workflow.WeatherWorkflow, "Cairo")

 require.True(t, env.IsWorkflowCompleted())
 require.NoError(t, env.GetWorkflowError())

 var data []messages.WeatherData
 require.NoError(t, env.GetWorkflowResult(&data))
 require.Equal(t, []messages.WeatherData{
  {
   Temperature: 41,
   Humidity:    80,
   WindSpeed:   4,
  },
 }, data)
}

This test executes the weatherWorkflow workflow with the given cityName, and uses the TemporalOption to configure the TaskQueue for the GetWeather activity.

Step 8: Use Temporal Cloud and Set Up a REST Server

To take full advantage of Temporal, you can use Temporal Cloud, which is a fully managed service that provides scalable and reliable orchestration for your workflows. Temporal Cloud handles the deployment, scaling, and maintenance of Temporal, so you can focus on building your application.

To get started with Temporal Cloud, create an account and obtain the necessary credentials. You’ll need to set up the Temporal SDK to use these credentials.

Next, you can set up a Golang Mux server to expose your workflows as REST services. Mux is a popular HTTP routing library for Golang. Here’s an example of how to set up a simple server:

// main.go
package main

import (
 "goenv/activity"
 "goenv/handler"
 "goenv/workflow"
 "log"
 "net/http"

 "go.temporal.io/sdk/client"
 "go.temporal.io/sdk/worker"
)

func main() {
 // set up the worker
 c, err := client.Dial(client.Options{})
 if err != nil {
  log.Fatalln("unable to create Temporal client", err)
 }
 defer c.Close()

 w := worker.New(c, "weather", worker.Options{})
 w.RegisterWorkflow(workflow.WeatherWorkflow)
 w.RegisterActivity(activity.GetWeather)

 mux := http.NewServeMux()
 mux.HandleFunc("/weather", handler.WeatherHandler) // curl -X GET http://localhost:8080/weather?city=Cairo
 server := &http.Server{Addr: ":5000", Handler: mux}

 // start the worker and the web server
 go w.Run(worker.InterruptCh())
 log.Fatal(server.ListenAndServe())
}

// handler/weather.go

package handler

import (
 "encoding/json"
 "goenv/messages"
 "goenv/workflow"
 "log"
 "net/http"

 "go.temporal.io/sdk/client"
)

func WeatherHandler(w http.ResponseWriter, r *http.Request) {
 // execute weather workflow with the city name from request query
 cityName := r.URL.Query().Get("city")
 if cityName == "" {
  http.Error(w, "city name is required", http.StatusBadRequest)
  return
 }

 // create a new temporal client
 // set up the worker
 c, err := client.Dial(client.Options{})
 if err != nil {
  log.Fatalln("unable to create Temporal client", err)
 }
 defer c.Close()

 we, err := c.ExecuteWorkflow(r.Context(), client.StartWorkflowOptions{
  ID:        "weather_workflow",
  TaskQueue: "weather",
 }, workflow.WeatherWorkflow, cityName)
 if err != nil {
  http.Error(w, "unable to start workflow", http.StatusInternalServerError)
  return
 }

 // wait for workflow to complete
 var result []messages.WeatherData
 if err := we.Get(r.Context(), &result); err != nil {
  http.Error(w, "unable to get workflow result", http.StatusInternalServerError)
  return
 }

 // convert result to json in key-value pais
 response := make(map[string]interface{})
 for _, data := range result {
  response[cityName] = data
 }

 jsonResponse, err := json.Marshal(response)
 if err != nil {
  http.Error(w, "unable to marshal response", http.StatusInternalServerError)
  return
 }

 w.Header().Set("Content-Type", "application/json")
 w.Write(jsonResponse)
}

The weatherHandler function takes the city name from the incoming HTTP request and starts the weatherWorkflow workflow with that input using Temporal’s ExecuteWorkflow function. It then waits for the result and returns it to the client.

Now you can run your http server

go run main.go

and execute the example workflow

curl -X GET http://localhost:8080/weather?city=Cairo

and this will result with a success temporal workflow which will return a json response

// Example JSON Response

{
  "Cairo": {
    Temperature: 36,
    Humidity:    40,
    WindSpeed:   21,
  } 
}

Workflow in temporal Dashboard

Workflow Dashboard

Example Weather Workflow Events

Temporal Data Layer Request → Response

Example Code: https://github.com/unijad/temporal-tutorial-example-01