Add Deep Learning-based Image Recognition to your Box App with Amazon Rekognition

Over the past few years, images have become an increasingly important file type in the enterprise. From sharing the latest progress of a construction project to capturing the scene of a car accident for insurance purposes, capturing and sharing images has become an integral part of any digital business process. Meanwhile, advancements in image analysis technology have unlocked new ways for developers to build more intelligent applications with images, leveraging cloud services to automatically analyze and label the objects in an image.

Today, we’re excited to share a new way for you to add deep learning-based image recognition to your apps built with Box Platform using an integration with Amazon Rekognition. Through our partnership with Amazon Web Services, we’ve developed a tutorial and created some sample code that enables you to upload an image to Box via an app, automatically generate labels using Amazon Rekognition’s image analysis technology, and apply them to your image files as metadata. To do this, we’ll leverage Box Platform, Amazon Lambda functions, Amazon API Gateway, Amazon Dynamo DB, and Amazon Rekognition.

The Use Case

What we’re building today is an image tagging system for a retail store app. In the app, users can capture and upload photos of clothing to a folder in Box and have the images automatically labeled with metadata. From there, the file metadata is indexed for keyword search and sorting.

On the backend, a webhook is triggered from Box when a user uploads an photo to that folder. Using API Gateway and a Lambda function, the Rekognition service is then called to analyze the image and generate labels for any relevant information about objects that are found in the image. These labels are then stored as metadata associated with the file in Box. The entire image tagging process is completely transparent to the end user and all they’ll see is an image automatically tagged with this useful information.

Getting Set Up:

Before writing any code, you’ll need to set up a few things.

To build the integration, you’ll need:

1. An AWS account
2. A Box Developer account

Step 1: Create a Box application

First, you’ll need to create a Box application. You can also modify the configuration of your existing application. Your application will need to be authorized into your Box account and have the appropriate scopes and advanced features enabled.

You can see the exact steps here.

Step 2: Create an API in Amazon API Gateway

Next, you’ll need to setup an API Gateway that will act as the endpoint for the events from the webhook you’ll create later.

To set up an API Gateway:

1. Log into the AWS Management Console and go to the API Gateway console.
2. Click “Create API”
3. Click “New API”
4. Give API name as “box-node-rekognition-lambdas-sample” and fill out the description
5. Press “Create API”

Once you’ve set up your API Gateway, you’ll need to grab the URL on the “Triggers” tab to use as the target address for your webhook.

Step 3: Create a Dynamo DB instance

Next, you’ll need to create a small Dynamo DB instance to store events from Box.

To set up a Dynamo DB instance:

1. Log into the AWS Management Console and go to the DynamoDB console
2. Click “Create table”
3. Name your table “box_file_uploaded”
4. Set the Primary key as “file_id”. The sample code expects the exact name
5. Leave the Table settings as default
6. Click “Create”

Step 4: Create an AWS Identity and Access Management (IAM) role

1. Go to IAM
2. Press “Create New Role”
3. Give Role Name “box-node-rekognition-metadata-sample-role”. Click “Next Step”
4. Select “AWS Lambda”.
5. Select Policies
6. AmazonDynamoDBFullAccess
7. CloudWatchLogsFullAccess
8. AmazonRekognitionFullAccess
9. Review and press “Create Role”.

Build the Integration

Next, you’ll create two Lambda functions and a webhook on your Box folder.

Step 5: Create a Lambda function to handle a Box webhook

Next, you’ll need to create a Lambda function that triggers when a FILE.UPLOADED event is sent using a Box webhook. In this tutorial, we’ll use the Box Node SDK and AWS Node SDK for the integration.

1. Log into the AWS Management Console and go to the Lambda Management Console
2. Press “Create a Lambda function”

• Choose the “Blank Function” blueprint

3. Configure a trigger for the Lambda function by clicking in the gray outlined area

• Choose API Gateway
• Leave the API name and Deployment stage with default values
• Choose “Open” for Security. This enables the Box webhook to call the API externally
• Press Next

4. Create the deployment package for the Lambda function

• Run npm install to install the Box Node SDK and AWS Node SDK
• Run npm run zip to create box-node-rekognition-webhook-lambda-sample.zip
• The zip file includes the sample code in index.js, Box Node SDK and AWS Node SDK.

5. Configure the lambda function

• Name = “box-node-rekognition-webhook-lambda-sample”
• Description = “Receives the FILE.UPLOADED webhook event and stores the event in DynamoDB”
• Runtime = “Node.js”
• Code entry type = “Upload a .ZIP file”
• Function package = Browse and select box-node-rekognition-webhook-lambda-sample.zip
• Environment variables:

BOX_AWS_ACCESS_KEY_ID = <AWS_ACCESS_KEY_ID>
BOX_AWS_SECRET_ACCESS_KEY = <AWS_SECRET_ACCESS_KEY>
BOX_AWS_REGION = <AWS_REGION>
BOX_EVENT_TABLE_NAME = box_file_uploaded
BOX_WEBHOOK_PRIMARY_SIGNATURE_KEY = <YOUR_WEBHOOK_PRIMARY_KEY>
BOX_WEBHOOK_SECONDARY_SIGNATURE_KEY = <YOUR_WEBHOOK_SECONDARY_KEY>

• Handler = “index.handler”. This sets the entry point to be the handler() function of the index.js file
• Role = “Create new role from template”
• Role Name = “box-node-rekognition-webhook-lambda-sample-role”
• Policy Templates = Leave blank
• Leave all of the advanced settings with default values.
• Press Next

6. Press “Create function”

Step 6: Create a Box webhook to call the Lambda function

Next, you’ll need to create a webhook on your Box folder to call the Lambda function.

Note: See Getting Started with Webhooks V2 and Overview of Webhooks V2 for more info.

1. Create a folder in your account via the Box web application and record the “Folder ID.” This can be found in the URL when you’re looking at the folder in the Box web application.
2. Create a webhook using curl to call the Box webhook API using the Folder ID from step 1:

curl https://api.box.com/2.0/webhooks \ 
-H “Authorization: Bearer <DEVELOPER_TOKEN>” \ 
-d ‘{“target”: {“id”: “<FOLDER_ID>”, “type”: “folder”}, “address”: “<YOUR_GATEWAY_API_URL>”, “triggers”: [“FILE.UPLOADED”]}’; echo

• Note: You must use the API to create V2 webhooks — there is no Web UI

3. You should get a response confirming that the webhook has been created:

{“id”:”<WEBHOOK_ID>”,”type”:”webhook”,”target”:{“id”:”<FOLDER_ID>”,”type”:”folder”},”created_by”:<YOUR_USER_INFO>,”created_at”:”2016–11–10T15:00:10–08:00",”address”:”<YOUR_GATEWAY_API_URL>”,”triggers”:[“FILE.UPLOADED”]}

• Note down the <WEBHOOK_ID> in case you need to modify or delete the webhook

4. The webhook will call the Lambda function each time a file is uploaded to the folder

• See here for details on how Box webhooks handle timeouts, retries, and exponential backoff

Note: Before calling the actual Lambda functions, make sure to check that the webhook was valid by calling our handy validateWebhookMessage() function in the Box Node SDK.

Step 7: Create the Rekognition Lambda function

Next, you’ll need to create a Lambda function that receives the Dynamo DB stream event that represents the FILE.UPLOADED webhook event from Box. Each time an event occurs, using the file_id in the event the content of the image file is read from Box. That content is given as input to the AWS Rekognition service. The Rekognition service will generate labels with confidence percentage. The labels and values are stored as metadata of the image file in Box.

1. Log into the AWS Management Console and go to the Lambda Management Console
2. Press “Create a Lambda function”

• Choose the “Blank Function” blueprint

3. Configure a trigger for the Lambda function by clicking in the gray outlined area

• Choose DynamoDB as trigger
• Select DynamoDB table ‘box_file_uploaded’ from dropdown.
• Batch Size as 1.
• Starting position as Latest.
• Check the Enable trigger.
• Click Next

4. Create the deployment package for the Lambda function

• Run npm install to install the Box Node SDK and AWS Node SDK
• Run npm run zip to create box-node-rekognition-metadata-sample.zip
• The zip file includes the sample code in index.js, the Box Node SDK and the AWS Node SDK.

5. Configure the Lambda function

• Name = “box-node-rekognition-metadata-sample”
• Description = “Demonstrates connecting a Box webhook to an AWS Lambda function”
• Runtime = “Node.js”
• Code entry type = “Upload a .ZIP file”
• Function package = Browse and select box-node-rekognition-metadata-sample.zip
• Environment variables:

BOX_AWS_ACCESS_KEY_ID = <AWS_ACCESS_KEY_ID>
BOX_AWS_SECRET_ACCESS_KEY = <AWS_SECRET_ACCESS_KEY>
BOX_AWS_REGION = <AWS_REGION>
BOX_EVENT_TABLE_NAME = box_file_uploaded
BOX_ENTERPRISE_ID = <YOUR_BOX_ENTERPRISE_ID>
BOX_CLIENT_ID = <YOUR_APP_CLIENT_ID>
BOX_CLIENT_SECRET = <YOUR_APP_CLIENT_SECRET>
BOX_PUBLIC_KEY_ID = <YOUR_APP_PUBLIC_KEY_ID>
BOX_PRIVATE_KEY_PASSPHRASE = <YOUR_APP_PRIVATE_KEY_PASSPHRASE>

• Handler = “index.handler”. This sets the entry point to be the handler() function of the index.js file
• Role = “Create new role from template”
• Role Name = “box-node-rekognition-metadata-sample-role”
• Policy Templates = Leave blank
• Leave all of the advanced settings with default values
• Click “Next”

6. Click “Create function”

Step 8: Test!

1. Login to box.com using the credentials you used to create your Box application.
2. Upload an image file in to the folder in which the webhook is enabled.
3. The webhook Lambda function should receive the event and store the event in DynamoDB.

• A new Item with the file name is created in ‘box_file_uploaded’ table.
• If any errors, logs will provides more details.

1. The Rekognition Lambda function is triggered if the Item is added to the ‘box_file_uploaded’ table.
2. Once the Lambda function is successful, the image file in Box is updated with the labels generated as output from Rekognition and stored as metadata.

• Click the image file to preview the file.
• Click “Info.” The ‘CUSTOM METADATA’ section is updated with the labels identified and their corresponding confidence level percentage.

Done! Now, every image uploaded to your Box folder will receive automatic image tagging applied as file metadata. You can then use that metadata in search results or display it in your app’s user interface.

You can see the complete Github repo with working code available here.

If you have any questions or suggestions, feel free to reach out to us on Twitter or post in our Developer Forum.