Serverless Image Analysis Stack in AWS

Hello learners, Today we’ll be creating a Serverless Image Analysis Stack on AWS as shown in the above picture.

The flow of the image analysis stack will be as follows:

• User/Service uploads an image to S3
• Using S3 event notifications, the metadata of that object goes to an SQS queue as an event.
• With the help of the event source mapping configured, a lambda function will get triggered for that event/message.
• Lambda will start processing the payload message.
• If the image size is above the Rekognition service quota limit, lambda resizes the image, deletes the existing object from S3, and uploads this resized image back to S3, which will again trigger this workflow.
• If the image size is under the Rekognition service quota limit, lambda detects faces on the image with the help of the Rekognition service and sends the results as an email notification to the end users with the help of the SNS service.
• If due to some runtime errors, lambda is not able to process the image. After 5 retries, the message will go to the dead letter queue which can then be used for some custom downstream processing/analysis/troubleshooting.

Amazon Rekogntion is a serverless image and video analysis service offered by AWS which uses highly scalable deep learning technologies behind the scenes. With Amazon Rekognition, you can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content.

Let's start by creating the necessary infrastructure for our serverless stack.

Create a file auto_lambda.py in the root of your working directory and the below code to it:

In line #14, update the topic_arn with your region, account id, and a topic name of your choice. Remember these values as they will be used in a later step as well.

What this lambda does is basically checks the image size from the event payload. If the size is above a certain threshold, it resizes the image, deletes the existing object from s3, and uploads the resized image back to s3.

If the image size is well under the set threshold, it runs a detect_faces API call offered from the Rekognition service and passes the object metadata to it.

Lambda then sends an SNS notification to the topic we configured in starting with details returned by the detect_faces API call.

As our lambda function makes use of certain modules which are not part of the default lambda python runtime, let’s containerize our code and package all the dependencies together.

Create a Dockerfile at the root of your working directory. Notice that we have given the same name in the COPY and CMD statements in the below Dockerfile. This is because, when lambda will run your code it will look for a method lambda_handler in the auto_lambda.py file.

Build the image with the below command. Take note of the image-name and image-tag you provided as these will be used in a later step.

docker build -t <image-name>:<image-tag> .

Create an infra.py file in the root of your working directory and start adding the below snippets as we understand them one by one.

The below snippet declares the necessary imports and variables for our script to function. Update all the variables as per your preferences. Remember to provide the same values to the variables topic_name, aws_region and owner_account as we had given earlier while creating auto_lambda.py. Provide the docker image name created above to the variable repository_name and its tag to the variable image_tag.

The below snippet does a bunch of things, we’ll understand them one by one.

• We first create an ECR repository to push the image we had created above.
• Next, we create an SNS topic and subscribe to it with the email_endpoint provided above in the variables.
• We then get an authorization token from ECR, remember your IAM user/role should have a ecr:GetAuthorizationToken in their IAM policy to be able to get this token.
• We then do a docker login and push this image to the ECR repository using the docker python client.
• We then create an S3 bucket and enable versioning on it.
• We then create 2 SQS queues. One for the source queue and the second for the dead letter queue.
• We set an Access Policy on the source queue to allow sqs:SendMessage from the s3 bucket created above and a Redrive Policy which sends messages to the dead letter queue after 5 retries. We also set an Access Policy on the dequeue which allows sqs:SendMessage from the source queue.
• We then set an event notification configuration on the S3 bucket created above which sends a message to the source queue for event types s3:ObjectCreated:Putand s3:ObjectCreated:Post . This means whenever we upload a new object or update an existing object, s3 will send an event notification to the configured queue.
• We create a lambda execution role which is assumable by the lambda service, create an IAM policy to receive messages from the source queue, get and delete objects from the S3 bucket, create log streams, and put log events to CloudWatch, publish messages to the SNS topic, etc and then attach this policy to the lambda execution role. We then wait for about 11 seconds for this change to reflect (probably some latency issue from the AWS side).
• We create a lambda function and provide the ECR image name and other attributes. The lambda function will get its code and dependencies from the image we created earlier.
• We then purge the source queue because there could be a test event notification sent by the S3 bucket the first time set we setup this configuration.
• And finally, we create an event source mapping on the lambda function which reads messages from the source queue and invokes the lambda function.

Deploy the infrastructure by running the script.

python infra.py

Remember to subscribe to the SNS topic by clicking on the Confirm subscription link received in email that goes by the subject: AWS Notification - Subscription Confirmation.

Voila, you should now have your own running Serverless Image Analysis Stack on AWS!

Test the stack by uploading the below image to the S3 bucket:

You should now get a notification on the configured email address which will look something like this:

This marks the end of our “Serverless Image Analysis Stack on AWS” blog.
If any questions/suggestions please add them in the comments. If you learned anything new today, please consider giving it a clap 👏, it keeps me to keep motivated to write more AWS content. 😀