DeepDive into DataWeave Language (DWL)
In Integration World, Developer’s Day to Day life is around the DATA(Information).
(a) What we are doing with data, how we are doing with it and why we are doing with it, what ever the use-case it is , may be Transformation, formatting , query/reformat , or just reading some values/fields.
wait a minute, All this is okay,
But what is DataWeave Language( aka DWL )?? why it is required in MuleSoft.
We use the DataWeave language is to define the business logic over the data so that Mule application perform typical Integration jobs.
In other words it’s a data transformation language.
In MuleSoft we can able to do all the above mentioned data processes(a) using DataWeave Language.
DataWeave is a language designed by MuleSoft to perform typical data manipulation over the data or massaging the data according to our use cases. Not only for the data massaging, it will be used in Connector configuration or Property reading or calling java class.
Let’s debug the structure of the DWL script format, How it looks, what each block will do and what else..
Above picture depicts the basic structure of DWL
- Declaring the DWL file and DWL version (Optional)
- Declaring the format of the output of this DWL script
- This section, where the most of the logic goes here and where the execution starts.
The three dashes separates the header and body parts of DWL scripts.
Apart from these section , I have added the declaration for Variable and Function.. there are lot of topics in dwl like how to use conditional logics, how to use the existing libraries or custom libraries and using data manipulative operations…..
Dataweave scripts are saved with a ‘.dwl’ extension and stored under “src/main/resources”…
Below mentioned things also can be done using DWL !!!
- Define our own set of scripts or functions which can be externalized from the implementation as Libraries and used in other implementation or can be used as a plugin and use it where ever it can be or in similar use-case.
- We can be able to call Static Java Methods from DataWeave.
How DWL reads Data???
There are 3 ways or strategies that the DWL uses
- In-memory , DWL can read all formats of data using this strategy. This stretegy loads the entire data/document and stores in memory. This helps for DWL to get access to any part of the data at anytime.
- Indexed, This strategy can help in reading the large volume of data/documents(works only with xml, csv and json), and will store data in disk-space so that it can avoid out of memory issues and helps to get access to any part of data.
- Streaming, This strategy will splits the data in chunks and stores the each chunk of data in memory and process it. once it done with one part then it moves to another part in sequential manner. In this strategy we won’t able to access the other parts of data.
We can also able to define how much memory that can be allocated to process the larger files.
We will cover few important topics of DWL scripting
Variables
Variables are Immutable in DWL, you can assign a value as constant or as a lambda function. Each variable behaves like a Function as DW is a functional programming language.
Each Variable define with a key word “var”
To give a name to any variable, we must follow some naming conventions which are defined in DWL.
“variable name can’t start with a number or a underscore/special characters or using ‘type’ keyword ”
examples of valid identifiers for a variable … “name, street1, street_” and these are not valid identifiers “_name, 1street, %street, type”
Each variable has it’s own scope, each variable created in header sections are treated as GLOBAL variables and its scope for entire script.
As a best practice, use the ‘do scope’ to declare and use the local variables as described in the below screenshot. and if you try to access the localvariables outside it’s scope results to failure of execution stating ‘Unable to resolve reference of ’
Below picture depicts the scope and how to define the local , global and assigning the lambda functions to variables.
Functions
There are many pre-define functions and libraries are present which comes along with dwl, this particular feature helps to define your own set of functions.
Each function will declared using “fun” keyword.
Look at below snippet, each function starts with keyword “fun” and followed with a distinguished function name and then logic/code assigned to it. Each function name should follow the naming conventions(refer to variable section) same as Variable.
fun functionName() = <logic goes here>You can not only declare a function with parameters and also you can set the data-types(constraints) for each variable and return type of a function returned values(It checks for the returned value of type with the defined returned type).
Can set optional parameters by assigning a static value to it.
You can also use the Function overloading feature to define your own set of funcitons which will be executed based on the input that you pass to those functions.
Below snapshot will help you to understand how the above explained things can be explored in DataWeave.
MimeTypes
MimeTypes might you have heard of this word, ever gave a thought what it is and what it will do ???
MIME is short form for “Multipurpose Internet Mail Extension”. It indicates the type or format of the payload/data/file.
It usually defined like “type/subtype”, If you look at below snipped, observe the italicized word, in DWL it defines format of script ‘s output.
%dw 2.0
output application/java
---you can set the mime type for the input and output payload in your mule application.
From runtime version 4.3.0, you can directly define your output just with the subtype instead of defining the type and subtype
If you are reading/writing a csv file you can set the seperator for each values using reading or writing properties i.e..output application/csv separator=”|”.
You can also change the mimetype of a script’s output. below examples states that can set a mimetype’s subtype and produce a output in the form of json using the keyword “with”.
Selectors
Selectors helps to choose/navigate/point to the fields/desired part of a payload/file/variable/function/anyObject.
Using selectors you can choose/select any variable but different is if variable is defined in line code/outside script it will be called using “vars.variableName” keyword, when you want to call a global variable you can directly refer the name of the variable.
In a typical dataweave script, if there any selector is not present in the input/missing in the payload it shows as ‘null’. But we can also able to make the script to throw an error if any field is missing from the input file/payload using a selector “!” after the key-name.
Below snippet will results an exception as “ There is no key named ‘id’ ”
output json
var data = {name: "Nikhil Animella"}
---
data.id!You can also use the dynamic selectors or filtering a set of data using some conditional expression or get the key & value pair from a object or get the metadata of the payload/file.
Will try to add few more topics in the other blogs….
Wondering where you can try your hands at DataWeave…
- Go to DataWeave Playground and try to get your hands dirty in DWL, It’s a playground accessible via any browser and you can play with vast libraries in DWL.
- You can also use the DWL CLI, basically a command line input that you can play with command prompt.
- You can use VS code Library — an extension that we can add to VS code for DWL and start play with it.