‘Go’ with Kafka & MySQL on Oracle Cloud

Oracle Application Container Cloud now has support for Go (golang) in addition to its existing set of runtimes which include Node.js, Java SE, Java EE, Python, PHP and Ruby

Welcome Gopher to Oracle Cloud!

This blog demonstrates a Go based asynchronous worker application

• Oracle Event Hub Cloud (managed Apache Kafka) acts as the message hub
• Oracle Application Container Cloud (polyglot, cloud native development platform) hosts the (Go) consumer application which makes use of the sarama client for Kafka
• Oracle MySQL Cloud is used as the sink for storing the processed messages from the consumer application— the Go MySQL Driver is used for this purpose

Here is a blog post which talks about the basics of Worker applications with an example based on consuming data from Redis queues

Application overview

Code is available on Github

The application is dead simple — a diagram is enough to make sense of what’s going on

• producers push data to Kafka topic
• the Go based Kafka consumer app (horizontally scalable) processes the data and,
• pushes them to MySQL with the partition, offset, key-value and processing node/instance info

High level architecture

Service Bindings to Event Hub and MySQL cloud services

Service bindings are utilized for private and secure communication b/w our application Oracle Event Hub and Oracle MySQL Cloud — details here

Here are some code snippets which highlight usage of the environment variables which we get as a result of the binding

Building the connection string (DSN) for MySQL

mysqlUser := os.Getenv(“MYSQLCS_USER_NAME”)
mysqlPwd := os.Getenv(“MYSQLCS_USER_PASSWORD”)
mysqlConnString := os.Getenv(“MYSQLCS_CONNECT_STRING”)
 
mysqlConnStringAndDB := strings.Split(mysqlConnString, “/”)
hostport, dbname := mysqlConnStringAndDB[0], mysqlConnStringAndDB[1]
mysqlDSNForDriver := mysqlUser + “:” + mysqlPwd + “@tcp(“ + hostport + “)/” + dbname

Creating a consumer instance for Kafka broker on Oracle Event Hub

ehcsBroker := os.Getenv("OEHCS_EXTERNAL_CONNECT_STRING")
 if ehcsBroker == "" {
  ehcsBroker = "192.168.99.100:9092"
 }
ehcsTopic := os.Getenv(“OEHCS_TOPIC”)
 if ehcsTopic == “” {
 ehcsTopic = “test”
 }
config := cluster.NewConfig()
config.Consumer.Return.Errors = true
config.Group.Return.Notifications = true
brokers := []string{ehcsBroker}
topics := []string{ehcsTopic}
consumer, err := cluster.NewConsumer(brokers, “test-consumer-group”, topics, config)

Infrastructure setup

Oracle Event Hub Cloud (Kafka broker)

The Kafka cluster topology used for the sample application in the blog was relatively simple i.e. a single broker with co-located with Zookeeper).

Please refer to the documentation for further details on topology and the detailed installation process (hint: its straightforward!)

Create topic — create the topic (detailed documentation)

Creating custom access rule

You would need to create a custom Access Rule to open port 6667 on the Kafka Server VM on Oracle Event Hub Cloud — details here. This is just a temporary change to help with testing using the Kafka CLI

Oracle Application Container Cloud does not need port 6667 (Kafka broker) to be opened since the secure connectivity is taken care of by the service binding

Oracle MySQL Cloud

Provision a MySQL database instance — you can refer to the detailed documentation here

Oracle MySQL Cloud service instance

Create custom access rule

You would need to create a custom Access Rule to open port 3306 on the VM on Oracle MySQL Cloud — details here. This is just a temporary change to help with testing

Oracle Application Container Cloud does not need port 3306 (for MySQL service) to be opened since the secure connectivity is taken care of by the service binding

Create database table

Connect to the MySQL instance on Oracle Cloud and bootstrap the database table

CREATE TABLE `datadump` (
 `uid` INT(10) NOT NULL AUTO_INCREMENT,
 `topic` VARCHAR(64) NULL DEFAULT NULL,
 `partition` INT(64) NULL DEFAULT NULL,
 `offset` INT(64) NULL DEFAULT NULL,
 `key` VARCHAR(64) NULL DEFAULT NULL,
 `value` VARCHAR(64) NULL DEFAULT NULL,
 `processedby` VARCHAR(64) NULL DEFAULT NULL,
 `createdat` VARCHAR(64) NULL DEFAULT NULL,
 PRIMARY KEY (`uid`)
 );

We have the foundation — its time to deploy our application to the cloud and test it out

Build & deployment

Build

• git clone https://github.com/abhirockzz/accs-go-kafka-mysql.git
• cd accs-go-kafka-mysql
• zip accs-go-kafka-mysql.zip gokafkamysql.go start.sh

Deployment

Pre-requisite

Before deployment, you would need to update the deployment.json with the details of

• Oracle MySQL Cloud service (which you created earlier)
• Oracle Event Hub Cloud topic (which you created earlier)

deployment.json

With Oracle Application Container Cloud, you have multiple options in terms of deploying your applications. This blog will leverage PSM CLI which is a powerful command line interface for managing Oracle Cloud services

other deployment options include REST API, Oracle Developer Cloud and of course the console/UI

• Download and setup PSM CLI on your machine (using psm setup) — details here
• deploy the application — psm accs push -n accsgokafkamysql -r golang -s hourly -m manifest.json -d deployment.json -e dockerhub -p accs-go-kafka-mysql.zip

Check your application

Once you’re done, you should be able to see the deployed application and its details — notice that this has 2 instances and the type is worker

bazzingGO is ready!

Test drive

Produce data

Push data to Kafka topic using the Kafka CLI based producer

kafka-console-producer.bat --broker-list <event_hub_host>:6667 --topic <identity_domain>-<your_topic> --property "parse.key=true" --property "key.separator=:"

Using Kafka CLI producer

Check MySQL

Use any SQL client to inspect the datadump table to check the corresponding results — select * from datadump

Processed data in Oracle MySQL cloud

Notice that the Kafka topic partitions are distributed among the two consumer instances — the information about instance/node which has processed the event from Kafka is also pushed to the table (highlighted in red)

That’s all there is to it ! You just created a simple yet (hopefully) useful Golang application using Kafka and MySQL on Oracle Cloud

Don’t forget to…

• check out the tutorials for Oracle Application Container Cloud — there is something for every runtime !
• other blogs on Application Container Cloud

Cheers!

The views expressed in this post are my own and do not necessarily reflect the views of Oracle.