Integrating Amazon Web Services (AWS) & Ultra ESB with no codes involved!

UltraESB is a lightweight enterprise service bus (ESB) capable of supporting many transports and message formats natively. It is powered by a framework called Project X which is the base for a new generation of redesigned integration products by Adroitlogic.

Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help businesses scale and grow.

In this article let’s see how we can integrate these two powerful technologies to develop an end to end solution without writing a single line of code using Adroitlogic’s innovative Integrated Development Environment UltraStudio.

The Problem Scenario…

Reporters use a mobile app to publish breaking news.

24x7 NEWS is a news agency which delivers breaking news to their readers via email and sms. They have a huge crew of regional reporters who submit news items via email. All reporters write their news items in English and they have a separate crew to translate news items to other languages.

After identifying the possibility of automating above procedure by integrating some translation APIs and a mobile notification service, 24x7 NEWS decides to give it a go by utilizing AWS, Adroitlogic Ultra ESB and Yandex as the technology providers.

24x7 NEWS wanted to perform this by putting the minimum effort & minimum time during the development phase.

So they came up with the following high level architecture to fulfill above requirement.

High Level Architecture Overview

This solution has an Ultra ESB cluster to accept news items from reporters and each of these nodes will publish received news items to an AWS SQS queue.

Then there will be another cluster of Ultra ESBs nodes which will perform following tasks in order.

  • Pull news items from AWS SQS queue
  • Translate news items
  • Publish to AWS SNS topics

Getting Started with UltraStudio

I am assuming that you have already downloaded and installed UltraStudio IDE. If not, go ahead and download it for free!

Check this video out if you are not familiar with UltraStudio and its features.

Let’s start by creating a new Empty UltraStudio project and adding following set of connectors and processors from component registry.

Connectors

  • AWS Connector
  • HTTP NIO Connector

Processors

  • Flow Control
  • Json Operations
  • Logging
  • Payload Manipulation
  • Transport Header Manipulation
  • XML Operations

Now let’s build integration flows one by one, to build up our integration solution from scratch.

UltraStudio Integration Flow to Receive News Articles From Reporters.

24x7 NEWS develops a dedicated mobile app for its regional reporters where it communicates real time with a HTTP endpoint to publish breaking news.

Mobile app is configured to send a special HTTP header, x-author with each request, so server side application can identify the respective reporter.

UltraStudio Integration Flow 1 : receiver.xcml

STEP 1 — Receiving News Item over HTTP

Ultra ESB HTTP NIO Ingress Connector
Configuration of Ultra ESB HTTP Ingress Connector

HTTP NIO Ingress connector has been configured to listen on port 8080 and this will accept all requests of any method (GET, POST, DELETE etc.) under default configuration. Anyway in this case we are expecting a POST request from reporters mobile app. So we need to change Allowed HTTP method property under Transport Configurations of connector to POST.

Configuring Ultra ESB NIO HTTP Ingress Connector to accept only POST

STEP 2 — Determining the Author

Whenever Ultra ESB receives a news item, NEWS 24x7 wants to determine its author for management purposes. So as the next step of this flow, we extract out x-author transport header from the message and log it into the console.

For the sake of simplicity, I am denoting a complex sub flow of writing to a database and doing many other things by simply logging it to the console.

Ultra ESB Logger Processing Element
Configuration of Ultra ESB Logger Processing element
All HTTP headers(Transport protocol headers) get automatically copied to the platform specific Message container as transport headers.
We are reading the header sent with HTTP request using an Ultra ESB specific expression syntax @{message.headers.x-author}.

STEP 3 — Sending NEWS item to AWS Simple Queue Service (SQS)

As the final step of reception flow, we push our news article to an AWS Queue via the SQS Egress Connector, so processing nodes in the second cluster can translate it to different languages and publish.

Ultra ESB SQS Egress Connector
Configuration of Ultra ESB SQS Egress Connector

Configuration Explanation

SQS Region : AWS region where our target queue is located and in this case it is us-east-1

Use Profile credentials : We have used profile credentials for authentication. Each processing node should have a file with name ‘credentials’ at ~/.aws in order to function this element properly. The format of this file is,

If you don’t won’t to use profile credentials, you can enter the AWS Access Key & AWS Secret Key here in the connector itself by flipping Use Profile Credentials switch.

Authenticating without using profile credentials

UltraStudio Integration Flow to Do translations & Publishing

Integration Flow 2 : main.xcml

As shown in high level architecture diagram, this flow is going to be running in multiple Ultra ESB nodes of the processing cluster.

STEP 1 — Pulling messages from AWS Simple Queue Service (SQS)?

As the first step of the flow, we have a SQS Ingress Connector which will be continuously polling the AWS queue for any available news items.

Ultra ESB SQS Ingress Connector
Configuration of Ultra ESB SQS Ingress Connector

Configuration Explanation

Temporary buffer location : Once an item is pulled from the queue, instead of keeping it in the memory(heap) throughout the flow, connector is going to write that to a temporary location of hard disk. Location specified will be used as the buffer location and make sure user who invokes Ultra ESB has write access to this location.

Visibility timeout : Once we pull an item from the queue, that item will not be visible for other nodes for 20 seconds. Anyway, if flow get completed without any error, news item will get automatically removed from the queue.

Wait time : Even though we are not going to make use of this parameter in this case, if we set this to something like 10, during each polling cycle if connector couldn’t find any message in the queue, it will wait 10 more seconds looking for a message before returning back. But here, connector will return immediately if it couldn’t find any message.

At the end of this step, we have a message inside the Ultra ESB engine with news item as the payload. Although I referred that as the payload, it is just keeping a reference to the temporary buffered file.

STEP 2 — Cloning message to handle supported languages

Initially, NEWS 24x7 is planning to release news in 3 different languages (Sinhala, French & Tamil). Since we have to handle these languages separately, as the second step of the flow, we clone the message into three different paths using the Multi Clone Processing Element.

Ultra ESB Multi Clone Processor

At the end of this step, Ultra ESB engine will be holding three messages having the news item as the payload. (This time, payload refers to three different references to the same temporary buffered file).

STEP 3,4 & 5 — Attaching language identifiers

These steps will be the starting points for each language and since we are going to push all these messages through the same flow from this point onward, we are adding a Message Context(Not a familiar term? Check below for the definition) variables using the Add Variable Processing Element for each of these messages to denote/mark the respective language. This would be useful in the later parts of the flow when we want to specifically process messages for each language.

Ultra ESB Add Variable Processing Element
Configuration of Ultra ESB Add Variable Processing Element

Using Add Variable Processing Element, I have created a scope variable with name lan with a hard coded value (Extraction Type : NO_EXTRACTION) en-si in the message context.


What is a Message Context in Ultra ESB X world?

Structure of an Ultra ESB Message Context

A Message Context wraps a particular instance of Message and holds the related peripheral information such as scopes, variables, transactional properties, etc.

Rule of thumb to remember is, you will have to deal with the same Message Context throughout the flow(Unless you clone the message at some point of the flow), even though you can change the Message at different stages of the integration flow.


STEP 6 — Translating News Using an UltraStudio Sub Flow

Translation of news item is completely handled by an UltraStudio sub flow.

UltraStudio Sub Flow is exactly similar to an UltraStudio Integration Flow, but Sub Flow can be considered as a reusable template which can be embedded inside multiple Integration Flows.

In this case, I have created 2 Integration Flows and 2 Sub Flows and I have embedded sub flow translator, inside main integration flow using a Sub Flow Referrer processing element.

UltraStudio Sub Flow Referrer

Calling Yandex APIs to translate NEWS items

In this case, I am going to use Yandex translate APIs to do the translations. Yandex API endpoint is expecting a request of following format and Content-Type should be application/x-www-form-urlencoded as well.

https://translate.yandex.net/api/v1.5/tr.json/translate ? 
key=<API key>
& text=<text to translate>
& lang=<translation direction>

The response from Yandex is returned wrapped in a XML as follows.

After receiving the response, in order to continue with publishing translated news item to AWS Simple Notification Service (SNS), we have to extract out the content between <text> tag from the response XML document.

So let’s see how we can compose an integration flow to cater above requirements without writing a single line of code!

UltraStudio Sub Flow 1 : Integrating with Yandex API to translate news items

The Message Context handed over to this flow now takes the following format.

Since we have already cloned our message into three different messages, this Sub Flow is going to execute trice for each news item that will be fetched from SQS queue.

Let’s take English — Sinhala translation as an example.

Yandex API request has 3 query parameters key, text & lang. So our first step would be creating the query string of the translation API call using Add HTTP Query Parameter Processing elements.

Ultra ESB Add HTTP Query Parameter Processing Element

STEP 6.1 — Adding key parameter

Adding API key to the query string using Ultra ESB Add HTTP Query Paramater Processing Element

STEP 6.2 — Adding text parameter

Adding text parameter using Ultra ESB Add HTTP Query Paramater Processing Element

Here I am using platform specific expression syntax to extract message payload and attach it as a query parameter.

Still we are carrying the original message received from SQS, so payload will be the original English news item.

STEP 6.3 — Adding lang parameter

Adding lang parameter using Ultra ESB Add HTTP Query Paramater Processing Element

Yandex is expecting this parameter in the format,

original_language_code-target_language_code

which is exactly the scope variable that we have attached to the cloned message initially to distinguish between language flows.

STEP 6.4 — Adding Content-Type Header

In this step, I am going to attach a transport header to the message using Add New Transport Header processing element.

Ultra ESB Add New Transport Header Processing Element
Configuration of Ultra ESB Add New Transport Header processing element

STEP 6.5 — Executing the HTTP API call

At this stage, we have attached all parameters to the message context which are necessary for the Yandex api call. So we just have to call out of the engine using an HTTPS NIO Egress Connector.

Ultra ESB HTTPS NIO Egress Connector
Configuration of Ultra ESB HTTPS Egress Connector

STEP 6.6 — Extracting payload from response

As described earlier, payload of the HTTP response from Yandex will be of type text/xml. So we have to use a XPath String Extractor Processing element to extract out the text content of <text> tag.

Ultra ESB XPath String Extractor Processing Element
Configuration of Ultra ESB XPath String Extractor processing element

By above configuration, we extract out the text content and persist it as a Message Context Scope Variable with variable name text.

STEP 6.7 — Logging the status

Our flow is continuing just after a network call. So there can be many reasons to fail and not continuing the flow from that point onward. So it is wise to log the current state, so we can find if something has gone wrong.

Configuration of Ultra ESB Logger Processing Element

Here we are again making use of Ultra ESB expression language to create the actual log line as follows.

Translated[@{variable.lan}] news item received : @{variable.text}

STEP 6.8 — Setting the payload

Up to step 5 of this flow, we were carrying the payload from SQS and immediately after completing Step 5, initial payload got replaced by the response of the HTTP call. But for the rest of our process, we don’t need that XML payload and what we need is the extracted content from <text> tag of XML. So we use a String Payload Setter processing element to update the message/payload of the current context.

Ultra ESB String Payload Setter Processing Element
Configuration of Ultra ESB String Payload Setter Processing Element

That concludes Step 6 of main flow & at the end of this flow, our message context will be as follows.

If you are worrying about the memory consumption, you can remove the redundant MessageContext.ScopeVariables.text using a Remove Variable processing element.


UltraStudio Integration Flow 2 : main.xcml

STEP 7 — Publishing News to AWS Simple Notification Service (SNS)

At the Step 7 of main integration flow, message context is entering again into an UltraStudio Sub Flow.

UltraStudio Sub Flow 2 : publish.xcml

When it comes to this point, ideally we should receive 3 messages per English News item, and each of these messages will hold the translated news content as the payload. So the next step is all about separately handling these messages to publish them into the respective AWS SNS topics.

Logic behind this flow is exactly reflecting the following pseudo code.

STEP 7.1 — Switching

In UltraStudio, switch-case can be implemented by combining Switch Processing element and couple of Case processing elements.

Ultra ESB Switch Case Processing Element
Configuration of Ultra ESB Switch Processing Element

With these two processing element combo, we can even do more things than using the traditional switch-case code block, since Predicate Function of Switch Processor supports two additional functions, MATCH & CONTAINS other than just EQUALS.

STEP [7.2,7.3,7.4] — Case

Case processing element expects the value of the expected case and it will pass the MessageContext to the next processing element only if extracted value from the scope variable and case value matches with each other.

Configuration of Case Processing Element

STEP [7.5,7.6,7.7] — Publishing to Amazon Simple Notification Service (SNS)

Final step of the flow is to publish translated news items to the SNS topics using Ultra ESB SNS Egress Connector, so the subscribed users will receive the breaking news instantly as an email or a sms.

Ultra ESB SNS Egress Connector
Configuration of Ultra ESB SNS Egress connector

Configuration Explanation

Topic ARN : Every resource we create in AWS gets an unique identifier called ARN, which stands for Amazon resource name. So you should provide the ARN of target topic, when configuring SNS Egress Connector.

Use json format : By setting this option to true, we get the ability to send different messages for different protocols(destinations). For example, using one publish action, you can send a short message to your SMS subscribers and a longer message to your email subscribers. Check AWS docs for more information regarding this feature.

Override Subject : The subject of the message can be pre configured by setting this option to true or can be use a dynamically added subject by setting aws.sns.subject scope variable at run time.

STEP 7.8 — Finalizing

As the final step of whole integration flow, we have to aggregate the multi cloned messages using the Multi Aggregate Processor.

Testing

UltraStudio provides you the ability to test Integration Flows instantly without having to deploy them manually in an Ultra ESB node. You can start the built in Ultra ESB X Server by simply clicking start icon in UltraStudio or by pressing Shift+F10 key combo.

While Ultra ESB is running, you should be able to start testing by sending your first request to the http://localhost:8080/publish endpoint.

Using a HTTP client to send the POST request

If nothing is wrong with the 1st Integration Flow, you will see a JSON response similar to the one shown in above figure.

To verify whether rest of the flows are working, you can try to subscribe your self into the AWS SNS topics. You can either subscribe for email alerts or for sms alerts from the AWS console iteself.

So that is it & Congratulations!!! You have developed an end to end solution integrating, AWS services & Yandex translate APIs without even seeing or writing a single line of code. Yep, not even a semicolon ;-)


AWS is the registered trademark of Amazon Inc.

Call To Action

  • Clap. Appreciate and let others find this article.
  • Comment. Share your views on this article.
  • Follow me. Chathura Widanage to receive updates on articles like this.
  • Keep in touch. LinkedIn