Helidon messaging with Oracle Streaming Service

Daniel Kec
Helidon
Published in
4 min readNov 10, 2020

Oracle Cloud Infrastructure Streaming service is a great alternative to Apache Kafka in the OCI environment. It is a fully managed service so maintenance, security patching, replication, monitoring and metrics are literally one click away from your OCI console. Connecting to OCI Streaming with Helidon is as easy as connecting to Kafka, or actually it is quite the same because OCI Streaming understands Kafka clients.

Streams are organized in the logical groups called StreamPools. When connecting to OCI Streaming with Kafka client, Streams are handled as Kafka topics and StreamPools as virtual Kafka clusters. If you configure Auto Create Topics to true in your StreamPool, it is equivalent of using auto.create.topics.enable=true in Apache Kafka.

With auto create topics option the streams in your stream pool are going to be created when Kafka client uses KafkaAdminClient.createTopics.

Let’s set up OCI Stream manually, you will need paid OCI tier as Streaming is not free eligible, more info about pricing can be found here.

Setting up OCI Streaming

To create new stream in the OCI Console, navigate to Analytics>Streaming>Streams and select Create Stream.

In the following dialog input stream name, compartment, and create, auto create or select existing stream pool. Stream pool is important because we will need its OCID to connect to our new stream.

Save the stream name and messages endpoint for later and navigate to its stream pool.

Save OCID from the stream pool detail for later.

Notice Kafka Connection Settings in the left bottom corner, where a handy example of Kafka connection properties is revealed.

We can see that bootstrap server value is actually an OCI Streams messages endpoint with port delimited by colon. OCI also prepared an SASL connection string example, only authentication token is needed.

Connect OCI Streaming from Helidon MP

Dependencies

Only dependencies needed in your Helidon MP app are the mp messaging engine and Kafka connector.

Configure Helidon

First of all we need to have available connection properties in the config, let’s save them to the custom attributes to use them later:

  • Tenant — name of the tenancy
  • User — OCI account user name
  • TokenOCI authentication token
  • Name — Stream name
  • Endpoint — Stream message endpoint
  • Port — usual Kafka port 9092
  • Streampool-ocid — ocid if the stream pool containing our desired stream

Configure messaging channels from-stream, to-stream to use Helidon’s Kafka connector:

Whole configuration file is available in the example project.

Expose messages with SSE endpoint

To easily test our connection to Oracle Streaming Service, let’s prepare simple REST endpoints connected to messaging channels to-stream and from-stream. For receiving messages from TestStream we will leverage the outgoing messaging method submitting payload to SubmissionPublisher broadcasting over SSE endpoint. For sending messages to TestStream, we will use Helidon’s buffered emitter registered as publisher in the to-stream channel.

After adding a simple UI from the example project we can start Helidon and test sending and receiving messages. Visit http://localhost:8080 and send a test message.

Now you are ready to connect OCI Streaming service with Helidon MP. How about Helidon SE?

Connect OCI Streaming from Helidon SE

Dependencies

Only dependencies needed in your Helidon SE app are the messaging engine and Kafka connector.

Helidon SE API makes the whole thing even simpler, no extra config is needed for following example. But if you want to, you can supply standard Helidon config to the publisherConfig method.

Keep the example running with correct properties and send a message manually from the OCI console with the Produce Test Message button.

And observe how Helidon loves OCI in the output!

OCI Stream says: Helidon ❤️ OCI!

Conclusion

Producing and receiving from OCI Streaming service with Helidon is easy and straightforward. Messaging with OCI streaming brings possibilities for transitive integration with other great OCI services such as GoldenGate, Integration Cloud, Database and many others.

More resources:

https://github.com/danielkec/helidon-oci-streams-example

https://medium.com/helidon/helidon-2-0-with-kafka-b97ba4f422bd

https://medium.com/helidon/reactive-messaging-with-helidon-2-0-f5de1ca5dc63

https://github.com/oracle/helidon/tree/master/examples/messaging

--

--