Build a Go based REST API to query time series tweet data in Cassandra

One of the previous blogs demonstrated a time series app using which a continuous stream of tweets (high velocity data) is consumed and persisted to a Cassandra instance (using Oracle Data Hub Cloud) and is later queried using another service

Getting started with Cassandra using Oracle Data Hub Cloud

Both the sample applications (i.e. the Tweets Producer service as well as the Tweet Query service) accompanying the blog post were Java based. In this blog post will, we will re-work the Twitter Query app to showcase a Golang based service (instead of the Java based Spring Boot app) and deploy it to Oracle Application Container Cloud (yes, it supports Golang!)

To be specific

• Continue to use Oracle Data Hub cloud as the persistent store for tweets
• use the gocql library to interact with Cassandra in order to write the core logic of getting tweets based on different crietria
• use the Gorilla toolkit (the muxpackage to be specific) to set up our HTTP routes

Before moving on, I would recommend reading the following sections from the previous blog to get some context of the use case/solution (which is pretty simple!)

• Architecture overview
• Few words on the Cassandra data model

Solution

the app is available on Github

abhirockzz/accs-golang-cassandra-tweet-query-service

We will only explore the relevant details of our Go based Tweet Query service

Start by connecting to the Cassandra instance on Data Hub Cloud and creating a gocql.Session object

Notice how the connection properties (Cassandra node/host and port as well as the credentials are picked up from the environment variables DHCS_* — thanks to Service Bindings!

Create Session

The core logic resides in the handlers — there are four of them (with self explanatory names)

• getTweetsByTweeter
• getTweetsByDate
• getTweetsOnDateByTweeter
• getAllTweets

Notice the getTweets function — it iterates over the bunch of tweets received from Cassandra, creates tweetinfo objects and sends back a slice

Cassandra objects to tweetinfo slice

tweetinfo is a simple struct with required attributes of the tweet

struct to represent tweets

All these handlers follow the same pattern

• executes a gocql.Query
• invokes getTweets to get a slice of tweetinfo objects
• and encode them as json and return to the caller — using the native encoding/json package

HTTP handler in action

the main method is the entry point to the app which brings everything together to start off the HTTP server

func main()

Service Binding to Data Hub Cloud

Application Container Cloud provides out-of-the-box Service Binding for Data Hub Cloud. This gives your app a secure communication channel without you having to do anything explicitly

no port related configuration is required at the database infrastructure level

Here is the documentation for this feature

Build & deployment

Before moving on, please go through the Infrastructure setup section of the previous blog to complete the following mandatory steps

• Setup a Cassandra cluster using Oracle Datahub Cloud console and bootstrap Cassandra (keyspace and table)
• Create a Twitter app which provides us with the required authentication tokens

Build the Tweet producer app

Start by fetching the project from Github — git clone https://github.com/abhirockzz/accs-cassandra-twitter-timeseries-app

• cd accs-dhcs-cassandra-tweets-producer
• mvn clean install

The build process will create accs-cassandra-tweets-producer-dist.zip in the target directory

Build the Tweets query service

Start by fetching the project from Github — git clone https://github.com/abhirockzz/accs-golang-cassandra-tweet-query-service

Zip up the following files — main.go, start.sh e.g. accs-golang-tweet-query-service.zip

zip file contents

Deployment a.k.a push to cloud

With Oracle Application Container Cloud, you have multiple options in terms of deploying your applications. This blog will leverage PSM CLI which is a powerful command line interface for managing Oracle Cloud services

other deployment options include REST API, Oracle Developer Cloud and of course the console/UI

You can download and setup PSM CLI on your machine (using psm setup) — details here

Deploy both the applications

• Tweets producer

Update the deployment.json with your Twitter access tokens and Oracle Data Hub Cloud instance details

{
 “instances”: 1,
 “memory”: “2G”,
 “environment”: {
 “TWITTER_CONSUMER_KEY”: “<as per your app>”,
 “TWITTER_CONSUMER_SECRET”: “<as per your app>”,
 “TWITTER_ACCESS_TOKEN”: “<as per your app>”,
 “TWITTER_ACCESS_TOKEN_SECRET”: “<as per your app>”,
 “TWITTER_TRACKED_TERMS”: “cloud,nosql”
 },
 “services”: [
 {
 “type”: “DHCS”,
 “name”: “<as per your instance>”,
 “username”: “<as per your instance>”,
 “password”: “<as per your instance>”
 }
 ]
}

Launch that zip into the cloud !

psm accs push -n TweetsProducer -r java -s hourly -m manifest.json -d deployment.json -p target/accs-cassandra-tweets-producer-dist.zip

Tweets Producer service deployed

• Tweet query service

Update deployment.json for this service as well

{
  "instances": 1,
  "memory": "1G",
  "services": [
    {
      "type": "DHCS",
      "name": "<name of your Cassandra instance>",
      "username": "<Cassandra username>",
      "password": "<Cassandra password>"
    }
  ]
}

.. and deploy this app as well

psm accs push -n TweetsQueryService -r golang -s hourly -m manifest.json -d deployment.json -e dockerhub -p accs-golang-tweet-query-service.zip

Once executed, an asynchronous process is kicked off and the CLI returns its Job ID for you to track the application creation

Go based Tweet query app on Application Container Cloud

After the apps are deployed, navigate to the Oracle Application Container cloud applications page to confirm — note down the application URLs

Test it out….

The way you would test this is already outlined in Test Drive section of the previous blog — no point repeating it. Here is the gist

• Start the tweets producer app and wait for the producer to start consuming tweets and persisting them to Cassandra
• Then use the REST API exposed by the Tweets Query service

Quick recap

You just,

• created a service to bombard a Cassandra instance on the cloud with Tweets!
• and then deployed a Go based REST API to interact with Cassandra to help you access those tweets in different ways

That’s all there is to it!

Don’t forget to…

• go through the Oracle Data Hub Cloud documentation for a deep dive

Oracle Data Hub Cloud Service — Get Started

• check out the tutorials for Oracle Application Container Cloud — there is something for every runtime!

Oracle Application Container Cloud Service — Create Your First Applications

• other blogs on Application Container Cloud

Latest stories and news about App Container Cloud — Medium

Cheers!

The views expressed in this post are my own and do not necessarily reflect the views of Oracle.