Building a microservice with Golang, Kafka and DynamoDB — Part II

The Journey of API response time from 1.2sec to under 50ms

This is the Part II of the series, Building a microservice with Golang, Kafka and DynamoDB. The contents of this post might make more sense if you read the previous post in which I covered the basic requirements to build a microservice using Golang and Kafka, how to wire them together to create a skeleton. In this post, I will cover DynamoDB integration and simple enhancements to make it scale.

Integration with DynamoDB

We are using guregu/dynamo which integrates with the official AWS SDK GO. Guregu/dynamo’s API’s are heavily inspired by mgo (MongoDB driver for Golang), as the DynamoDB client. Below is the template struct we are using for dynamo table creation

type LocationTrackingTemplate struct {
   UserId  string `dynamo:",hash"`
   Timestamp int64  `dynamo:",range"`
   TripId    string `localIndex:"tripid-local-index,range"`
}

Below is an example code to create the DynamoDB table using the above table template and the dynamo client library.

import (
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
)
awsConfig := aws.Config{
   Region: "eu-west-1",
}
sess, err := session.NewSessionWithOptions(session.Options{
   Config:  awsConfig,
})
if err != nil {
   log.Fatal("failed to create AWS session")
}
awsCreds, err = sess.Config.Credentials.Get()
if err != nil {
   log.Fatal("unable to find usable AWS credentials")
}
func SetupDB(tableName string, sess *session.Session) error {
   if tableName == "" {
      return errors.New("no table name provided")
   }
   db := dynamo.New(s)
   ct := db.CreateTable(tableName, LocationTrackingTemplate{})
   return ct.Provision(10, 10).Run()
}

Optimizations and Enhancements for better performance

During load testing, I found that it’s response time is around 1.2 seconds which is too high, considering this API needs to be working with high throughput. Now below are the few tweaks that I did to reduce the API response time.

• The initial code version used Sarama’s Kafka Sync Producer config set to `RequiredAcks. I have covered a bit more on how to wire sarama, a Go libarary for Kafka and sarama-cluster, a Go library for Sarama Cluster Extenstions in the Part I of my post.

In a typical messaging system, acknowledgements generally take more time so I tried changing the acknowledgement type to `NoResponse`, which surprisingly did not yield any improvement. This prompted me to change the Kafka Sync Producer to Kafka Async Producer with `RequiredAcks` acknowledgement type. Also write what kind of improvement did you get out of this.

• Secondly, The DynamoDB writes were being throttled even with a provisioned capacity of 500 WCUs. Digging down further, I found that the DB write function was inside a Goroutine and there was no control over the number of Goroutines getting spawned and trying to write to DynamoDB.

The second enhancement I made was to update the above code using go-channels to control the number of Goroutines running and thus controlling the number of parallel writes on DynamoDB, and getting rid of throttled writes.

• Moreover, because of my inexperience with DynamoDB, the initial table template was created with a Global Secondary Index (GSI) for better querying. GSIs have to be provisioned with separate RCUs and WCUs which add to the cost. Later I converted the GSI to a Local Secondary Index(LSI) and since LSIs share the provisioning of the parent DynamoDB Table, we saved on the cost and performance. An additional benefit was that we were consuming a mere 50 WCUs to track a fleet of 5k users.

After all these minor tweaks and fixes, we were able to run the tracking service at an amazing fast throughput. Just to give you an idea, a single node can now easily handle 100k requests per minute without sweat.

That’s a wrap.

Hope you guys like it and learn from it. Thanks.

Happy Learning.