Data Transformations with Ballerina

Ballerina language as an integration language boasts on the type system as one of its many limelights. Ballerina natively supports XML and JSON. When it comes to integrating various systems, ability to natively support various forms of data is of utmost importance.

Let’s take a simple cloud integration [1] as an example — connecting Salesforce with Gmail. Say, when a new opportunity is created in Salesforce, an email with account details is generated and sent via Gmail. The salesperson creates an opportunity in Salesforce. The black box indicates an integration solution which will get the opportunity details and send an email with the details via Gmail.

Now let’s explore the role of the integration solution that is represented by the black box here. First, there is an inbound which listens to Salesforce opportunities. When an opportunity is created, it triggers an outbound to send an email via Gmail. In between the inbound and the outbound, occurs a data transformation that maps the Salesforce opportunity to a Gmail email. This is the data transformation step involved in solving this integration problem. If you are using Ballerina as your integration solution, you can use Ballerina Transformer to visually compose your data transformation to easily and quickly complete your integration.

This is just one example where you could use data transformations. When an integration solution is composed of multiple interactions with external systems — connectors and endpoints, data transformations is a must.


Ballerina transformer performs the special task of transforming data. It has lived through many iterative development phases to be the ideal solution for data transformations.

When it comes to Ballerina type system and integration solutions; JSON, XML and Structs are the most commonly transformed data types. If we take the above example, it would be converting the Salesforce JSON response to Gmail JSON request. Similarly, XML and Structs are also transformable.

For the purpose of simplicity to discuss the syntax and usage, we will explore an example of transforming Employee struct to a Person Struct.

Transformer Syntax

There are three variations of syntax for defining a transformer. It is a top-level construct (similar to functions) and will have the following.

  1. public keyword — optional
  2. Source type and source name
  3. Target type and target name
  4. Identifier — optional
  5. Input arguments — optional. Identifier is mandatory if arguments are present.

As we discussed earlier, transformer is used for data transformations. Hence, it is achieved with the conversion syntax in Ballerina. We will look at the different forms of usages below as well. The examples will also include the visual view of the transformations in Ballerina Composer which we will discuss in more detail later.

Default Transformer

The source type and the target types are present and there is no identifier or additional arguments. This is known as the default transformer and there can be only one default transformer for Person to Employee conversion.

transformer <Person p, Employee e> {
e.age = p.age;
e.name = p.firstName + p.lastName;
e.address = p.street + p.city.toUpperCase();
}

Using the default transformer :

Employee e = <Employee> p

Transformer with Identifier

Suppose that there can be more than one way of transforming Person to Employee. At this point, we can define multiple transformers that can be distinguished by the identifier.

transformer <Person p, Employee e> setCityToNewYork() {
e.age = p.age;
e.name = p.firstName + p.lastName;
e.address = p.street + "New York";
}

Usage of the above transformer is as follows.

Employee e = <Employee, setCityToNewYork()> p;

Transformer with Input Arguments

Suppose you need to pass in some input arguments that are needed for the transformation. You can declare the arguments similar to function arguments as below.

transformer <Person p, Employee e> setCountry(string country) {
e.age = p.age;
e.name = p.firstName + p.lastName;
e.address = p.street + p.city + country;
}

During the invocation, we need to pass the arguments as below.

Employee e = <Employee, setCountry("United Kingdom")> p;

Transformer Visual Representation

When it comes to data transformations, ability to visualize them is of formidable value. Ballerina language along with Ballerina Composer, provides means of visualizing a data transformation.

Consider the following source.

transformer <Person p, Employee e> {
e.age = p.age;
e.name = p.firstName + p.lastName;
e.address = p.street + p.city.toUpperCase();
}

Below is the visual representation of the above transformation in Ballerina Composer.

The source Person p is shown on the left and the target Employee e is shown on the right in the above. If there are any arguments those will be shown along with the source on the left. The arrows indicate the data flow for the transformation.

Let’s explore each of the statements. The explanations will also include diagrams, each depicting only the statement under discussion for clarity in understanding.

Statement 1:

e.age = p.age;

This is a simple assignment of person’s age to employee’s age. This is simply represented by an arrow from the source to target.

Statement 2:

This is an addition of two strings by using the + operator.

e.name = p.firstName + p.lastName;

The + operator is indicated by a box with two inward arrows (left) and one arrow outward (right) from the box representing the output. The two inputs are p.firstName and p.lastName and it is mapped to e.name.

Statement 3:

e.address = p.street + p.city.toUpperCase();

Let’s first take a look at a portion of the above statement which includes only the function invocation.

e.address = p.city.toUpperCase();

The diagram shows a box representing the function ‘toUpperCase’. The input to the function is p.city which is mapped with an inward arrow (left). The output is shown as an outward arrow to e.address (right).

Now let’s check the next statement with the + operator and the function invocation. We call these as nested operator/function expressions.

e.address = p.street + p.city.toUpperCase();

Notice the arrow flow above. One input to the + operator is p.street and the other input is the output of the toUpperCase function. Finally, the result of the operator is mapped to the e.address.

With reference to above diagrams and explanations, it is clear that visually grasping a data transformation is way easier than looking through the source. It is extremely useful when there is a lot of logic that needs to be shown at a glance.

Compare the following two views of the same transformation.

What you see on the right is the collapsed view of the same transformation simply depicting the input and outputs from each of the operators/functions. This is an especially useful view if you need to see the bird’s eye view of your transformation. You can selectively click on the collapse and expand icons to get this view for each of the functions/operators. Similarly, you can also collapse the struct/JSON to hide away the details if you see fit, to save space on the canvas.

Composing Transformer Visually

We discussed how the visual representation of the transformer is laid out — functions, operators, the arrow notation, source and targets that make up the transformer.

Now let’s look at how you can compose the transformer in Ballerina Composer visually.

Since the transformer is a top-level construct, it appears along-side other constructs in the tool palette. You just need to drag and drop a transformer to the canvas.

By default the source and target are given as Source struct and Target struct which are undefined, and hence the semantic error. But you can start filling them up. Optionally you can also give a name to the transformer.

Refer the blog by Tharik Kanaka explaining more detailed steps of writing your own transformation logic at http://blog.tharik.org/2017/11/data-transformations-in-ballerina-095.html. With reference to this blog, let’s directly move into the section of creating mappings visually.

By expanding the transformer view you can configure the mapping arrows functions and operations. Suppose a simple transformer transforming Person to Employee. Following is the transformer view along with the tool palette.

There are a bunch of operators — unary, binary and ternary and the functions that are builtin and imported to the Ballerina file is listed in the tool palette. You can simply drag and drop each of these operators and functions and start drawing mappings. Refer the following screencast for creating the example data transformation above visually.

Note that the screencast is done with the split view of the Composer. This way you can see the source been created when you visually drag components and draw arrows to create the mapping. Similarly, changes done in the source view are also reflected in the diagram.

Ballerina Transformer Screencast

Understanding the Transformer Scope

Although the transformer syntax is similar to a function with a bunch of statements, it is not actually the case. In favor of the more advantageous visual representation, there are few limitations in the types of statements you can write in the transformer construct. However, the important thing to know is that this in no way limit the features or trades-off any capabilities of Ballerina, as there are alternative better-represented ways of performing your integrations.

1. One source + arguments and one target.

There can be only a single source type and target type. You can pass in additional arguments that act as sources to construct the target. But there can be only one target type.

This is the usual case of any transformation, you use multiple data to construct a single output. Hence this approach.

2. Source and parameters are input and input only. Target is output only.

Source, arguments will always act as input and target will always act as the output. They cannot swap their position, meaning inputs cannot be modified in the transformer scope.

Example: In above example, since Person is the source and Employee is the target, e.name = p.firstName is a valid statement. However p.firstName = p.firstName.toUpperCase() is not a valid statement because we are modifying inputs. Similarly, Employee cannot be input in any statement since the transformer’s role is to construct Employee, not to use it as an input.

The main reason for this approach is to emphasize on the data flow of a transformation. Going back and forth and modifying the input will complicate the diagram and lose its meaning. Also, it makes sense in terms of single responsibility of a transformation. You are intending to transform A to B. Why change A along the way?

3. No if-else block.

The usual if-else block if (condition) {} else {} is not supported by the transformer. But do not panic, we do support conditional transformation, but in a more meaningful data-oriented way.

if-else is supported via the ternary operator :

e.category = (p.age > 50) ? "senior" : "other";

The main reason for choosing this approach over the traditional if-else block is to limit the logic to strict data transformation as well as be able to visually show them. The ternary operator guarantees that an assignment is performed in both if and else blocks of execution and the value of assignment is of consistent e.category (string) type. We cannot guarantee such behavior in the traditional if-else block statement.

The example above is an extremely simple format, but you can include any complicated transformation by performing a function invocation in the ternary operator.

4. No while loop.

Again, do not panic :) We provide a bunch of iterable functions that can perform better than the while block. This is a component that is still under development and you will hear more from us in future.

The idea is to support iterable functions such as apply, map, filter where you can attach lambdas or anonymous functions to achieve anything and more than what can be done with while loops. Again similar to the ternary operator, by allowing iterable functions, we are restricting the logic to be data collection based, and hence stop any deviations from the transformation. More on this later. Follow our dev group if you are interested in knowing our approach on this.

In conclusion, the idea of these restrictions on the syntax is to stop developers from making the transformer construct contain anything unrelated to transformation. Also, with apply, map and filter, transformer enforces manipulation on data collections over data-independent while of if-else blocks. By having this cleaner, data-oriented syntax, the visual diagram of the transformation logic is cleaner and meaningful.

Future Work in Ballerina Transformer

Ballerina Transformer is in constant development and it is working towards improving the visual and conversion capabilities. Following are some of the major features we are working on at the moment.

  • Operations on iterable collections — map, apply, filter and anonymous functions
  • XML transformations
  • Declaring unsafe conversions in transformer

Try out the Ballerina Transformer and let us know your feedback and suggestions on features, improvements and bugs that will help us improve the transformation experience with Ballerina[3] [4].

Reference

[1] https://medium.com/@maheeka/ballerina-connectors-for-seamless-integration-a5bc8386326

[2] Transformer Examples in Ballerina By Example — https://ballerinalang.org/docs/by-example/transformers

[3] https://github.com/ballerinalang/composer/issues/ and https://github.com/ballerinalang/ballerina/issues/

[4] https://groups.google.com/d/forum/ballerina-dev

Like what you read? Give Maheeka Jayasuriya a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.