Unraveling Data Challenges with DataWeave Part 2: Advanced Filtering and Aggregation

Introduction

Welcome back to our series on DataWeave! In the previous article, we introduced basic DataWeave concepts. Now, we’ll explore data filtering and transformation — common tasks when dealing with data.

Scenario

Let’s say we have a list of students with their names, ages, and grades. We want to filter out the students who scored more than 80 and transform the data into a more readable format.

Here’s a sample input:

[
  {"name": "John", "age": 18, "grade": 75},
  {"name": "Alice", "age": 19, "grade": 85},
  {"name": "Peter", "age": 20, "grade": 90},
  {"name": "Emma", "age": 21, "grade": 70}
]

We want our output to be a list of student names and grades, but only for students who scored more than 80.

DataWeave Solution

We can solve this with the following DataWeave code:

%dw 2.0
output application/json
---
payload filter ((student) -> student.grade > 80) map (student) -> {
    name: student.name,
    grade: student.grade
}

This script first filters the input (payload) to include only students with grades greater than 80. Then, it transforms the data to create a new structure that includes only the name and grade of each student.

DataWeave Output

[
  {
    "name": "Alice",
    "grade": 85
  },
  {
    "name": "Peter",
    "grade": 90
  }
]

DataWeave Playground

Conclusion

In this article, we’ve seen how to use DataWeave to filter and transform data. These tasks are commonplace when dealing with data, and DataWeave provides us with easy-to-use functions to perform these operations.

In our next article, we’ll dive into more complex data transformation tasks with DataWeave. As always, remember to practice with different types of data and operations to gain a better understanding of DataWeave’s capabilities. Until next time, happy data weaving!