Salesforce Einstein Vision

Published in

Analytics Vidhya

7 min readNov 3, 2020

In a commercial setting, the tasks to be delivered always comes to be completed in a short deadline. One of our clients has a very good reputation in the insurance sector. They always come with very creative ideas to help their client base and local repair store. In this new engagement, they had come with a new and unique requirement where they wanted to automatically tags uploaded images into a car and bike. Moreover, they also wanted to automatically locate dent in images to process with insurance. Although this looks like a very useful and interesting application, implementing and completing the integration in a short period of time is rather challenging. Moreover, the worrying part is that in my team no one had an extensive set of working experience with image data.

It is always challenging to work with Image data. Data science problems involving image data are often met with challenges in handling both volume and variety. In general working on problems related to computer vision, requires having a solid background in image processing fundamentals. These presumptions changed while I got to work with Salesforce Einstein's vision.

Einstein Vision provides different API that explores the power of deep learning models to classify and detect objects in the images at a larger scale. It also provides a pre-trained classifier or trains your custom classifier.

Einstein Vision provides the following APIs:

Einstein Image Classification — API for developers to train models and classify images into different categories at scale

Einstein Object Detection — API for developers to train model and detect multiple objects within the Image

Let’s understand more about Einstein Vision Terminology

Please follow the following steps to get started with Salesforce Einstein.

Image Classification

Einstein's vision provides a solution to automatically classify the images into appropriate categories. It uses the advanced state of the art Deep Learning algorithm to get good accuracy.

Prerequisites:

Signup for the account: Create Salesforce developer login — https://developer.salesforce.com/signup?d=70130000000td6N
Find the key: If your account is activated then store the einstein_platform.pem file in the appropriate folder to generate the access token
Install Curl (optional): This tool is available in Linux. If you are working on a different system download it from here, Curl
Generate a token from this page

Please provide your registered email id (don’t enter the user id)
Click Browse and navigate to the einstein_platform.pem file that you downloaded in step 2.
Set the maximum life span of the token.
Click “Get Token”.
JWT token is available and can be used for different tasks.
For more details, you can visit

Image Classification Steps:

Note: In all the below step your need to replace TOKEN with a generated token from the Prerequisite step.

1) Create a Dataset

Upload a dataset into Einstein platform.

First, create a different folder for each class and store the images in appropriate folders.
Folder name would be treated as a class label
Create a Zip file of all this folder
Data can be uploaded from the URL as well.
You can consider this data. It contains images of cats and dogs. I have stored it in “D:/DogVsCat.zip”

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "type=image" -F "data=@D:/DogVsCat.zip[CK1] " https://api.einstein.ai/v2/vision/datasets/upload/sync

In the above query, we are creating/uploading data from a local drive. Users can upload the files from URL as well with the path variable.
In case of data upload is from local drive then provide a path to the .zip file on the local drive (FilePart) with “data” argument (instead of a path). The maximum .zip file size you can upload from a local drive is 50 MB. For Windows use “data=@D:/DogVsCat.zip”
Remember the datasetID from a response
Advance parameters are provided

- learning rate: Specify how much gradient change model’s optimization at each timestamp. Smaller values would prefer i.e. 0.001. Valid range 0.0001–0.01
- epochs: Number of training iteration for a model. Valid range 1–1000
- trainParams {“trainSplitRatio”: 0.n}: Training and testing split ratio during
model creation. If “trainSplitRatio” set to 0.9 then 90% of data considered in training and the remaining considered in testing.

2) Train the Dataset

Once the dataset is uploaded successfully it is ready for further training.
Copy the DATASET_ID and TOKEN with your appropriate values.

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Beach and Mountain Model" -F "datasetId=<DATASET_ID>" https://api.einstein.ai/v2/vision/train

Training would take time depending upon the data dataset size. Please note the modelId for further reference.
To check the training status run the following command with the appropriate modelId observed from the previous step.

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Beach and Mountain Model" -F "datasetId=<DATASET_ID>" https://api.einstein.ai/v2/vision/train

The response returns the status of the training process. If it’s running, you see the status of RUNNING. When the training is complete, it returns a status of SUCCEEDED and a progress value of 1.

3) Accuracy Measures

Once your model is trained, you can see different metrics like f1 score, precision, recall, and confusion matrix.

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/models/<MODEL_ID>

4) Test on unseen data

Once your model is ready you can classify any image.
Model would return the following two values.

Label: class label
Probability: Probability for given a class

Used can provide local image(sampleContent) or image from URL (sampleLocation) For example, local image from windows use “sampleContent =@D:/new.png”

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "sampleContent=@D:/image.png " -F "modelId=<YOUR_MODEL_ID>" https://api.einstein.ai/v2/vision/predict

Object Detections

Object Detection workflow for custom model works as follows

1) Create Dataset

Create a proper dataset for Object Detection with the following Guidelines.

Objects in the images are visible and recognizable.
Images are forward-facing and not at an angle.
Images are neither too dark nor too bright.
Images contain 100–200 or more occurrences (across all images) for each object you want the model to identify. The more occurrences of an object you have, the better the model performs.

Labeling and Zip File Format

The labeling content is stored in a comma-separated (csv) file named annotations.csv.
The annotations file contains the image file name and the labels and coordinates (in JSON format) for each object in the image.
See the Annotations sample to create a Dataset from a Zip File Asynchronously. Here are the first four lines from the annotation sample.
CrowdFlower and SharinPix are available to annotate data.
Put all the images in a single folder with label CSV file and create a zip file for upload.

Please find the sample annotation.

image_url,box0,box1,box2,box3,box4,box5,box6,box720171030_133845.jpg,"{""height"": 1612, ""y"": 497, ""label"": ""Alpine - Oat Cereal"", ""width"": 1041, ""x"": 548}","{""height"": 1370, ""y"": 571, ""label"": ""Alpine - Oat Cereal"", ""width"": 904, ""x"": 1635}","{""height"": 1553, ""y"": 383, ""label"": ""Alpine - Corn Flakes"", ""width"": 1059, ""x"": 2580}",,,,,20171030_133911.jpg,"{""height"": 1147, ""y"": 2299, ""label"": ""Alpine - Oat Cereal"", ""width"": 861, ""x"": 374}","{""height"": 1038, ""y"": 2226, ""label"": ""Alpine - Oat Cereal"", ""width"": 752, ""x"": 1263}","{""height"": 1464, ""y"": 709, ""label"": ""Alpine - Bran Cereal"", ""width"": 1056, ""x"": 179}","{""height"": 1470, ""y"": 746, ""label"": ""Alpine - Bran Cereal"", ""width"": 697, ""x"": 2327}","{""height"": 1434, ""y"": 752, ""label"": ""Alpine - Corn Flakes"", ""width"": 831, ""x"": 1312}","{          ""height"": 1080, ""y"": 2378, ""label"": ""Alpine - Corn Flakes"", ""width"": 965, ""x"": 2059}",,20171030_133915.jpg,"{""height"": 496, ""y"": 1811, ""label"": ""Alpine - Oat Cereal"", ""width"": 344, ""x"": 922}","{""height"": 100, ""y"": 112.18126888217523, ""label"": ""Alpine - Oat Cereal"", ""width"": 100, ""x"": 100}","{""height"": 517, ""y"": 1753, ""label"": ""Alpine - Oat Cereal"", ""width"": 337, ""x"": 1282}","{""height"": 590, ""y"": 1157, ""label"": ""Alpine - Bran Cereal"", ""width"": 383, ""x"": 889}","{""height"": 587, ""y"": 1157, ""label"": ""Alpine - Bran Cereal"", ""width"": 368, ""x"": 1674}","{""height"": 575, ""y"": 1169, ""label"": ""Alpine - Corn Flakes"", ""width"": 350, ""x"": 1303}","{""height"": 532, ""y"": 1757, ""label"": ""Alpine - Corn Flakes"", ""width"": 365, ""x"": 1629}"

2) Upload Dataset

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@D:/alpine.zip" -F "type=image-detection" https://api.einstein.ai/v2/vision/datasets/upload

While uploading dataset set for object detection use type = “image-detection”.
While uploading dataset set for image classification use type = “image”.
Please note down the id from the response field. It corresponds to DATASET_ID.
When available is true and statusMsg is SUCCEEDED, the upload is complete, and the dataset is ready to be trained.

Check the Status of Upload

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/datasets/<DATASET_ID>

3) Train the Model

Run the following command to train the model. This is the same as Image Classification.

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Alpine Boxes on Shelves" -F "datasetId=<DATASET_ID>" https://api.einstein.ai/v2/vision/train

Note down the modelId from the response header.

To check the status of training execution.

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/train/<YOUR_MODEL_ID>

To check the different accuracy/recall/precision/f1 score

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/models/<MODEL_ID>

4) Test the Model

Run the following command to classify the new Images.

Specify the input file path with sampleLocation (URL path) or sampleContent (local path with @ as suffix).
The deference of prefix “detect” here in API call. (Image classification have the predict as a suffix)

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "sampleContent=@D:/alpine.jpg" -F "modelId=BN2PTZQ6U2F7ORW57ZIZWZWRDQ" https://api.einstein.ai/v2/vision/detect

Conclusion:

I personally found Salesforce Einstein very interesting, easy to go in a quick time. Please find the advantage and disadvantage as follow

Advantages:

Easy to use for building custom models.
No specific data science expertise required. Provided pre-trained models for use.
Vision model can easily integrate with the Salesforce APP for real-time prediction with apex class.
A larger set of training datasets can also be uploaded for training.

Disadvantages:

Vision provides very limited options for advanced parameters. Many important parameters are missing like Batch Size, Dropout, etc.
Creation or integration of custom algorithm is not possible with the current version
Batch processing of prediction is not available. Current Einstein Vision API provides only one prediction at a time. However, to combat this issue use can write custom logic in apex for batch prediction.
It is very difficult to interpret the built model since no algorithmic details are available as output.

Reference

1. https://metamind.readme.io/reference#predictive-vision-service-api
2. https://metamind.readme.io/docs/introduction-to-the-einstein-predictive-vision-service
3. https://metamind.readme.io/docs/scenario — Tutorial example
4. https://www.kaggle.com/chetankv/dogs-cats-images

Special Mention

Arun Aniyan: https://www.linkedin.com/in/arunaniyan/

LinkedIn connect

Chetan Khadke: LinkedIn

Salesforce Einstein Vision

Image Classification

Object Detections

Special Mention

LinkedIn connect

Written by Chetankumar Khadke