SOLVED: DataWeave 2.0: Merging Two Objects Without Losing Data

DataWeave serves as a powerful tool for transforming data between different formats and structures. Among its many capabilities, the mergeWith operator stands out as a versatile and indispensable tool for merging and combining data structures in complex integration scenarios.

Understanding the “mergeWith” Operator

The mergeWith operator in DataWeave allows developers to merge multiple objects or arrays into a single object or array based on specified criteria. It offers flexibility and control over how data is combined, making it a valuable asset in various integration use cases.

Key Features and Benefits or “mergeWith”

1. Flexible Data Merging: The mergeWith operator supports merging objects and arrays of varying structures, allowing developers to combine data from disparate sources seamlessly.
2. Customizable Merge Logic: With the mergeWith operator, developers can define custom merge logic using DataWeave expressions, giving them full control over how data is merged and combined.
3. Support for Nested Structures: The mergeWith operator supports merging nested objects and arrays, enabling developers to handle complex data structures with ease.
4. Handling Conflicts and Overrides: Developers can specify how conflicts and overrides are handled during the merge process, ensuring data integrity and consistency in merged datasets.

Practical Use Cases

1. Data Enrichment: The mergeWith operator can be used to enrich data by combining information from multiple sources, such as customer profiles, product catalogs, and transaction histories.
2. Configuration Management: In application configuration scenarios, the mergeWith operator allows developers to merge default configurations with user-defined settings, providing a flexible and customizable configuration management solution.
3. Aggregate Data Streams: When dealing with streaming data or batch processing, the mergeWith operator can aggregate and merge data streams from multiple sources, consolidating information for further analysis or processing.

Example: Merging Customer Data

Consider a scenario where you need to update the CustomersList record of an Organization object in Salesforce without overriding the existing information:

%dw 2.0
output application/json
var latestPayload={
    "Id": "a0H8d00001914LGEAY",
    "purpose": "To merge the objects",
    "details":{
        "state": "Karnataka"
    },
    "members": [
        {
            "age": 31
        }
    ]
}
var previousPayload={
    "Id": "a0H8d00001914LGEAY",
    "details":{
        "countryOfBirth": "INDIA",
        "state": "Andhra Pradesh"
    },
    "members": [        
        {
            "name": "Jagadishwar",
            "age": 30
        },
        {
            "name": "Swetha",
            "age": 29
        }
    ]
}
fun mergePayloads(previousPayload: Object, currentPayload: Object) =
    {
        //Fields present in latest object but not present in previous object at root level
        (currentPayload filterObject ((value, key, index) -> !previousPayload[key]?)), 
        (previousPayload mapObject ((value, key, index) ->  {
                (key): value match {
                    case val if(!currentPayload[key]?) -> val
                    case val is Object -> val mergePayloads currentPayload[key]
                    case val is Array -> val map ((item, index) -> 
                        if(item is Object) 
                            item mergePayloads ((currentPayload[key][index] default {}) ++ item) 
                        else item)
                    else -> currentPayload[key]
                }               
            })
        )
    }
---
previousPayload mergePayloads latestPayload

Update Request:

{
    "Id": "a0H8d00001914LGEAY",
    "purpose": "To merge the objects",
    "details":{
        "state": "Karnataka"
    },
    "members": [
        {
            "age": 31
        }
    ]
}

Existing Record:

{
    "Id": "a0H8d00001914LGEAY",
    "details":{
        "countryOfBirth": "INDIA",
        "state": "Andhra Pradesh"
    },
    "members": [        
        {
            "name": "Jagadishwar",
            "age": 30
        },
        {
            "name": "Swetha",
            "age": 29
        }
    ]
}

Expected Output:

{
  "Id": "a0H8d00001914LGEAY",
  "purpose": "To merge the objects",
  "details": {
    "countryOfBirth": "INDIA",
    "state": "Karnataka"
  },
  "members": [
    {
      "name": "Jagadishwar",
      "age": 31
    },
    {
      "name": "Swetha",
      "age": 29
    }
  ]
}

Algorithm:

fun mergePayloads(previousPayload: Object, currentPayload: Object) =
    {
        //Fields present in latest object but not present in previous object at root level
        (currentPayload filterObject ((value, key, index) -> !previousPayload[key]?)), 
        (previousPayload mapObject ((value, key, index) ->  {
                (key): value match {
                    case val if(!currentPayload[key]?) -> val
                    case val is Object -> val mergePayloads currentPayload[key]
                    case val is Array -> val map ((item, index) -> 
                        if(item is Object) 
                            //Merge the Object present at 'index' with fields matching in the currentPayload
                            item mergePayloads ((currentPayload[key][index] default {}) ++ item) 
                        else item)
                    else -> currentPayload[key]
                }               
            })
        )
    }

Call the above function by passing latestPayload and previousPayload as below:

previousPayload mergePayloads latestPayload

Test Results

Conclusion

The mergeWith operator in DataWeave is a powerful tool for merging and combining data structures in MuleSoft integration projects. Its flexibility, customizability, and support for nested structures make it a valuable asset for handling complex data transformation scenarios. By leveraging the mergeWith operator effectively, developers can streamline data integration processes, improve data quality, and unlock new insights from disparate data sources.

As organizations continue to embrace digital transformation, mastering DataWeave and its powerful operators like mergeWith will be key to driving innovation, agility, and success in the evolving landscape of data integration and analytics.