How to Remove Empty Values and Key-Value Pair Which Having Empty Values from Complex Data with DataWeave
If you work with JSON data, you may encounter situations where you need to remove empty values from arrays or objects. For example, you may want to filter out null, empty strings, or empty arrays from your input data. This can help you reduce the size of your output data, avoid errors, and improve performance.
In this blog post, I will show you how to use DataWeave, the powerful data transformation language in MuleSoft’s Anypoint Platform, to remove empty values from JSON data. I will use a custom DataWeave function called FilterTree that recursively filters out empty values from any data structure. I will also use two helper functions, filterArrayItems and filterObjectDetails, that apply the FilterTree function to arrays and objects respectively.
Let’s start with the codes:
CODE-1:
%dw 2.0
output application/json skipNullOn="everywhere"
//remove key value pairs from payload if value is empty for a key
fun ExtractRequiredFields(inBound) = (
inBound match {
case is Array -> removeEmptyDataFromArray(inBound)
case is Object -> removeEmptyDataFromObejct(inBound)
else -> if(!isEmpty(inBound)) $ else null
}
)
//removing the key value pairs, if value is empty from array of items
fun removeEmptyDataFromArray(arr :Array) = do{
var data = (
arr filter(!isEmpty($)) flatMap (
$ match {
case is Array -> removeEmptyDataFromArray($)
case is Object -> removeEmptyDataFromObejct($)
else -> if(!isEmpty($)) $ else null
}
)
)
---
if(!isEmpty(data)) data else null
}
//removing key value pairs, if value is empty from an Object
fun removeEmptyDataFromObejct(inBound :Object) = do{
var data = (
inBound filterObject(!isEmpty($)) mapObject ((value, key, index) ->
(value match {
case is Array -> (key): removeEmptyDataFromArray(value)
case is Object -> (key): removeEmptyDataFromObejct(value)
else -> if(!isEmpty(value)) (key): value else null
}
)
)
)
---
if(!isEmpty(data)) data filterObject(!isEmpty($)) else null
}
---
ExtractRequiredFields(payload)CODE-2:
%dw 2.0
import * from dw::util::Tree
output application/json skipNullOn="everywhere"
//remove the empty values under array
fun filterArrayItems(arr:Array) = (
(arr filterTree ((value, path) -> !isEmpty(FilterTree(value)))) filterArrayLeafs ((value, path) ->
value match {
case a is Array -> !isEmpty(filterArrayItems(a))
case a is Object -> !isEmpty(filterObjectDetails(a))
else -> !isEmpty($)
}
)
)
//remove the empty values under object
fun filterObjectDetails(obj :Object) = (
(obj filterTree ((value, path) -> !isEmpty(FilterTree(value)))) filterObjectLeafs ((value, path) ->
value match {
case o is Array -> !isEmpty(filterArrayItems(o))
case o is Object -> !isEmpty(filterObjectDetails(o))
else -> !isEmpty($)
}
)
)
//remove the values under the payload
fun FilterTree(InBound) = (
InBound filterTree ((value, path) ->
value match {
case s is Array -> !isEmpty(filterArrayItems(s))
case s is Object -> !isEmpty(filterObjectDetails(s))
else -> !isEmpty($)
}
)
)
---
FilterTree(payload) NOTE: CODE-1 and CODE-2 functionality same
Based on my analysis, I think code-2 is better than code-1 for the following Reasons:
Code-2 is more concise and readable than code-1. Code-2 uses the built-in functions from the dw::util::Tree module, such as filterTree, filterArrayLeafs, and filterObjectLeafs, which are designed to handle nested data structures. Code-1 uses custom functions that use match expressions, recursion, and filtering, which are more verbose and complex.
Code-2 is more efficient and performant than code-1. Code-2 avoids unnecessary variable assignments and null checks, which can reduce memory usage and improve execution speed. Code-1 uses do blocks and var declarations, which can create intermediate data structures that consume more memory and time. Code-1 also uses multiple null checks, which can slow down the processing of large data sets.
Code-2 is more maintainable and reusable than code-1. Code-2 follows the principle of separation of concerns, which means that each function has a single responsibility and a clear input and output. Code-1 mixes different levels of abstraction and logic in the same function, which makes it harder to understand and modify. Code-2 also follows the principle of don’t repeat yourself (DRY), which means that it avoids duplicating code by using existing functions. Code-1 repeats similar code in different functions, which increases the risk of errors and inconsistencies.
The FilterTree function takes an input parameter called InBound and applies the filterTree function from the dw::util::Tree module. The filterTree function takes a lambda expression as an argument and returns a new data structure that only contains the values that satisfy the lambda expression. The lambda expression checks if the value is an array, an object, or anything else, and calls the appropriate helper function to filter out empty values.
The helper functions, filterArrayItems and filterObjectDetails, use the same logic as FilterTree, but they also use another function from the dw::util::Tree module called filterArrayLeafs or filterObjectLeafs. These functions take a lambda expression as an argument and return a new array or object that only contains the leaf nodes (the nodes that do not have any children) that satisfy the lambda expression. The lambda expression checks if the leaf node is an array, an object, or anything else, and calls the appropriate helper function to filter out empty values.
Let’s see how this code works with an example input:
{
"name": "John",
"age": 25,
"address": {
"street": "123 Main St",
"city": "New York",
"state": "",
"zip": null
},
"hobbies": [
"reading",
"",
null,
{
"name": "gaming",
"platforms": [
"PC",
"",
null
]
}
]
}The output of the code is:
{
"name": "John",
"age": 25,
"address": {
"street": "123 Main St",
"city": "New York"
},
"hobbies": [
"reading",
{
"name": "gaming",
"platforms": [
"PC"
]
}
]
}example input-2:
{
"breakfast": {
"fruits": {
"apples": {
}
}
}
}The output of the code is:
{
}As you can see, all the empty values (empty strings, nulls, and empty arrays) have been removed from the output data.
I hope this blog post has helped you learn how to use DataWeave to remove empty values from JSON data. If you want to learn more about DataWeave, you can check out the DataWeave documentation, the DataWeave examples, and the DataWeave interactive learning environment.
