Intelligent Document Processing With AWS AI Services and MuleSoft RPA

Leonardo Padua Cassimiro
Another Integration Blog
5 min readNov 2, 2022

Introduction

Insurance claims, invoices, various forms and applications — every organization deals with a wide range of unstructured documents. Although manual extraction of critical information is still common, many departments have automated these processes.

Amazon Textract, a machine learning (ML) based service, goes beyond simple optical character recognition (OCR) to identify and extract data from scanned files.

Today, I will demonstrate how to use MuleSoft RPA, the AWS Intelligent Document Processing (AWS IDP) component in MuleSoft RPA Builder, and Amazon Textract together. We will fully automate the process of reading an email, processing its attachment, and using its data to create a new record in a CRM system.

The Process

I started by creating a Process Automation (bot) to read emails and save the attached Invoice.pdf in a specific folder. My bot will extract data stored in the invoice, including the vendor’s name, contact information, invoice number, and purchased items (description, quantity, and price). To be precise, the bot will call the AWS IDP component in MuleSoft RPA Builder to do the job. Once the data is extracted, the bot will use the information to create and populate a new invoice record in Zoho CRM.

Note: For the sake of simplicity, I am not explaining how to create Zoho CRM and AWS accounts. You can use both accounts with a free trial period.

Bot tasks

I have added the following RPA Bot Tasks to the process automation:

  1. Read emails : read emails and save the attached PDFs in a specific directory (to Process)
  2. Log into Zoho CRM
  3. New invoice: navigate to the invoices page
  4. Extract PDF data using AWS: connect to AWS Service to extract data from each PDF file saved to the “To Process” directory
  5. Enter invoice details: create and enter the invoice details
  6. Move the file to the “Completed” folder: if there are more files to be read, go to step 3, “New invoice”
  7. Sign out of Zoho
Bot tasks

Here is the format of the document I am using for testing, Invoice.pdf.

Extract Data From PDF Using the AWS IDP Component

ERP Docs With AWS

ERP Docs with AWS in MuleSoft RPA Builder

The ERP Docs With AWS component uses the Amazon Textract service to analyze invoices and receipts and extract relevant data in JSON format.

Note: You must have an AWS Account.

The configuration is simple and requires the AWS Client ID, Client Secret, Region Endpoint, and the file to analyze. The supported file types are PDF, JPEG, and PNG.

ERP Docs With AWS

The AWS Service has three types of responses:

  • Raw Text: the extracted raw text of the document
  • Normalized JSON: the parsed result based on the normalized result and defined JSON path
  • Raw JSON: the complete AWS response

JSON Query

JSON Query

The “JSON Query” component extracts data from JSON with JSONPath Expressions. I used this online tool to create and test the expressions.

By setting JSON Object as Normalized JSON, the AWS Service response will look like this:

[
{
"ExpenseIndex": 1,
"PageNumber": 1,
"SummaryFields": [
{
"Type": "VENDOR_NAME",
"Value": "RPA Mule Company",
"Label": "",
"Confidence": 99.923393249511719
},
{
"Type": "OTHER",
"Value": "Josephine Darakjy (Sample)",
"Label": "Payable to",
"Confidence": 92.0
},
{
"Type": "OTHER",
"Value": "Chanay (Sample)\n4B Blue Ridge Blvd\nBrighton, MI, 48116, United States",
"Label": "Invoice for",
"Confidence": 77.0
},
{
"Type": "OTHER",
"Value": "Project name 1",
"Label": "Project",
"Confidence": 64.0
},
{
"Type": "OTHER",
"Value": "29/09/2022",
"Label": "Submitted on",
"Confidence": 48.5
},
{
"Type": "TOTAL",
"Value": "£15,540.00",
"Label": "Total",
"Confidence": 99.946990966796875
},
{
"Type": "INVOICE_RECEIPT_ID",
"Value": "123456",
"Label": "Invoice #",
"Confidence": 99.971229553222656
},
{
"Type": "SUBTOTAL",
"Value": "£15,540.00",
"Label": "Subtotal",
"Confidence": 99.975227355957031
},
{
"Type": "DUE_DATE",
"Value": "14/10/2022",
"Label": "Due date",
"Confidence": 98.978599548339844
},
{
"Type": "OTHER",
"Value": "£0.00",
"Label": "Adjustments",
"Confidence": 77.353431701660156
}
],
"LineItemGroups": [
{
"ColumnName": "Description",
"Type": "ITEM",
"TypeConfidence": 70.0,
"CellContents": [
"Google Pixel 6a",
"OnePlus Nord CE 2 5G",
"OnePlus 10 Pro",
"Samsung Galaxy S22 Ultra"
]
},
{
"ColumnName": "Qty",
"Type": "QUANTITY",
"TypeConfidence": 70.0,
"CellContents": [
"5",
"2",
"3",
"7"
]
},
{
"ColumnName": "Unit price",
"Type": "OTHER",
"TypeConfidence": 70.0,
"CellContents": [
"£720.00",
"£320.00",
"£850.00",
"£1,250.00"
]
},
{
"ColumnName": "Total price",
"Type": "PRICE",
"TypeConfidence": 70.0,
"CellContents": [
"£3,600.00",
"£640.00",
"£2,550.00",
"£8,750.00"
]
},
{
"ColumnName": "",
"Type": "EXPENSE_ROW",
"TypeConfidence": 99.990859985351562,
"CellContents": [
"Google Pixel 6a 5 £720.00 £3,600.00",
"OnePlus Nord CE 2 5G 2 £320.00 £640.00",
"OnePlus 10 Pro 3 £850.00 £2,550.00",
"Samsung Galaxy S22 Ultra 7 £1,250.00 £8,750.00"
]
}
]
}
]

Below are some examples of evaluating and getting the values for Vendor Name, Project and Invoice Number.

  • Vendor Name: $..SummaryFields[?(@.Type==’VENDOR_NAME’)].Value
  • Project: $..SummaryFields[?(@.Label==’Project’)].Value
  • Invoice Number: $..SummaryFields[?(@.Type==’INVOICE_RECEIPT_ID’)].Value
JSONPath Expressions

Once I evaluated the values, I saved them in their respective variables.

JSON Query

Next, the bot logs into Zoho CRM to create a new invoice using Click Web Element and Keystrokes components.

I have recorded every step of this exercise — executing the bot, sending the Invoice.pdf to AWS Service and, extracting the data from the returned JSON, and opening and creating a new Invoice in Zoho CRM.

MuleSoft RPA: Extract PDF data using AWS IDP in MuleSoft RPA Builder

What do you think? What was your experience using AWS components in MuleSoft RPA Builder? Let me know in the comments.

I hope you liked this post.

Originally published at https://blogs.infomentum.com.

--

--

Leonardo Padua Cassimiro
Another Integration Blog

I’m Leonardo Padua Cassimiro, working as a MuleSoft Senior Consultant at Infomentum and MuleSoft Certified Developer Level 2