Processing Kinesis Streams w/ AWS Lambda and Golang

Using Kinesis Streams as a trigger for AWS Lambda has made it easier than ever to process real-time event data. The days of managing servers, persisting checkpoints, and other complexities associated with using the Kinesis Client Library are over.

You can subscribe Lambda functions to automatically read batches of records off your Kinesis stream and process them if records are detected on the stream.

The example below illustrates real-time user event data flowing into a Kinesis Stream. Processing is then handled by a Golang Lambda function, and the aggregated results persisted to a DynamoDB table.

Streaming event data with Kinesis, Lambda, and DDB

This is a remix of Segment’s post on Managed Analytics Pipeline (if you’re not familiar with Segment, check them out, they’re awesome).

While Lambda supports a number of languages out of the box (Node.js, Java, and Python to name a few), Go is currently not one of them.

To get around this limitation the excellent Apex CLI can build, deploy, and manage our Lambda functions written in Go.

The Golang app performs the following steps:

• Loop over Kinesis Streams records
• Aggregate events into hourly buckets
• Update counters in the DynamoDB table

The event stream will expect JSON payloads in the following format:

{"event": "Viewed Homepage", "timestamp": "2017-06-24T22:08:41+00:00"}

The Apex CLI makes deploying the function a breeze:

ksis-stream(master) $ apex deploy stats
   • creating function         function=stats
   • created alias current     function=stats version=1
   • function created          function=stats name=user_events version=1

Last step, set up a trigger in AWS Console:

Viola! As new events flow into the stream, Lambda will automatically pick them up, aggregate counts, and persist to DynamoDB:

While this is a trivial example (and probably better performed by StatsD), hopefully it illustrates some of the tools we have at our disposal for processing large amounts of real-time streaming data (without needing to manage servers) in a language we ❤️.