Photo by SIMON LEE on Unsplash

Image Recognition Using Mulesoft and Salesforce

My 2020 MuleSoft Hackathon Project

MuleSoft and Salesforce are the right combination of technologies to deliver robust and complex projects in a short amount of time. I will demonstrate how I used both to recognize images and associated information produced from a mobile device.

How this is done? Here are the required components for this project:

1. Salesforce Developer Org
2. Anypoint Platform (Sandbox) account
3. MuleSoft mule-aws-recognition-system-api
4. MuleSoft mule-aws-recognition-process-api

Salesforce Developer Org

You should have a developer account for Salesforce from their developer site. This will allows us to:

• Create custom objects (tables)
• Create custom fields on each of those objects
• Expose a mobile application (previously known as Salesforce 1) to be installed on IOS or Android devices
• Access Visualforce pages to customize what we show on a mobile app or browser
• Use Apex Classes, custom apex code to handle data from a page and allows us to expose REST services from a custom apex (java style) definition

Here we have the design:

• Standard Object (content version and content document link): allows us to store the actual binary file in Salesforce
• Custom Object (hackathon image): allows to have a record to link the photo taken
• Custom Object (image label): stores the image information labels and how accurate the image is with a label from AWS

Here comes the fun part:

• Visualforce page: allows us to show a UI to take the picture:

• Apex Controller: allows us to get all information from the picture and create the content version and content link record related to the hackathon image
• Apex Controller REST: exposes the mentions endpoint, triggering a push notification in the mobile device

Here is where you can get the code.

Now, a basic flow:

MuleSoft Mule-aws-recognition-system-api

Initially, this system API was for AWS only. I have included one of the pieces I need to complete this exercise.

As I mentioned, this system API allows us to process a Base64 image and send it to Amazon Recognition API. From this call, we are able to retrieve the generated labels.

This same application contains the logic to pull a few tweets using a parameter based on hashtags.

#%RAML 1.0
title: mule-aws-recognition-system-api
/image:
  post:
    body:
      application/json:
    responses:
      200:
        body:
          application/json:
/twitter:
  /tweets:
    get:
      queryParameters:
        q:
          description: "Parameters to filter by hashtag"
      responses:
        200:
          body:
            application/json:

To process the image, I used the AWS Java SDK to use the API. <y flow looks like this:

For the Tweets, we have a different endpoint which receives only the GET request and we return all tweets based on the hashtags provided.

Here is what the flow looks like:

As you can see, this is a simple HTTP request to the Twitter API. It is not included in the process API, as we are not using a connector to extract the logic of this request.

MuleSoft Mule-aws-recognition-process-api

At this point in the process API, we are doing more things and connecting the dots. I will explain the next part step-by-step.

The process API has this RAML:

#%RAML 1.0
title: mule-aws-recognition-process-api

/image:
  post:
    body:
      application/json:
    responses:
      200:
        body:
          application/json:
            example:
/sfdc:
  /images:
    get:
      responses:
        200:
          body:
            application/json:
  /contentVersion:
    get:
      queryParameters:
        id:
          description: imageId
          type: string
/tweets:
  get:
    responses:
      200:
        body:
          application/json:

1. After the mobile application saves the picture we took with our device, Salesforce calls the /images endpoint we exposed in MuleSoft. Basically, it passes three params: imageRecordID (hackathon image), contentVersionID (ID of the actual file in Saleforce), and contentDocumentLinkID (link document to the picture).
2. After MuleSoft gets the parameters, we use the Salesforce connector to make a query to content version and we download the file (actual image in Base64). Then, we call the system API to pass the image and wait for the labels that AWS recognizes.

3. Once AWS responds, we also create the labels in Salesforce (image labels) for the uploaded image as records. Lastly, we call the REST service we exposed in Salesforce to notify the person that the image has been processed and has labels.

Now, we have to call the REST service from the connector. In older versions of the connector, we were able to connect by getting the session ID and using the REST endpoint directly. In Mule 4, we are not able to do so and we must use the connector capabilities instead.

In the last part as a user, you can use your device to see the labels created per record. I also created a feature on this process API and have created a page on MuleSoft to show the information we saved.

How I did do it? In the same process API, I placed a new configuration file named “portal”, a flow that contains a “Load Static Resource” that serves a page stored in a folder named “web” on src/main/resources.

The main page contains a script that uses JQuery to show the information from the images and tweets.

This is what the render page looks like:

Basically, the page tells you the labels we got from AWS, shows the image we took, and the tweets related to the generated labels.

You can get this code for this here, and watch a video on this process here.