In my last post I went through the steps of editing and uploading files to S3 using Boto3 with Cloud9. This was a small part of the work I did for automating some processes for a previous project. For the purpose of that specific task I used Cloud9 as my IDE and ran the script in there, but what if we run the scripts in a serverless Lambda function? Lambdas are a great way to run code without provisioning or managing servers and are really easy to use.
In this post I’m going to upload a sample file to S3, create a Lambda function that downloads that file, reads it and also create a Simple Queue Service (SQS) to send that file to.
I started with a simple script that creates a SQS in Cloud9, before I decided to go full serverless and it looks like this:
And now our SQS is created! It’s important that you remember to specify the region when creating a queue, because if you don’t it will go to the Virginia region by default.
As it goes for the sample file that I’m going to send to the SQS, in this case, I manually uploaded it to S3. I used a simple JSON file, with a kitty in it:
I already had a bucket created:
And from there you can simply open the bucket and drag the file you want directly in to the browser. It ends up looking something like this:
Ok, so we have our SQS ready and a sample file inside a bucket in S3. The next step is to create our Lambda function. We start by searching the service in the AWS console:
We click on the “Create function” button:
We’ll pick the “Use a blueprint” option:
For the next step, I chose the hello-world-python3 blueprint. Then click the “Configure” button at the bottom of the screen.
Next, we define a name for the function and for the execution role we select the “Create a new role with basic Lambda permissions”. We leave the rest as it is and click the “Create function” button below.
We now have our Lambda function created! But we are not quite ready yet. Since our Lambda is going to be interacting with another AWS services such as S3 and SQS we need to give it the proper permissions to do that.
In our Lambda function we’re going to scroll down until we see the Execution Role section and click the “View the (…) role” link:
In the IAM Manager, we are going to click the “Attach policies” button:
Then we are going to search the S3 and SQS policies and select the ones that say “AmazonS3FullAccess” and “AmazonSQSFullAccess” and click the “Attach policy” button below:
And then you should see something like this:
Great! Now we are ready to start adding code to our lambda function. Going back to our lambda, I’m sure you noticed that it has an IDE for you to put your code in.
You’ll notice that my script is inside a function named lambda_handler that takes an event and a context. The Lambda handler is the convention that Lambda has to run a function. Everything inside the handler is going to be seen by the Lambda function as runnable code. You can get into more detail about this in the official AWS documentation.
The rest of the code is pretty much straight forward: It creates the resources that it’s going to use, it tries to download the file to a /tmp directory (this is because the file system where the function is executed is readonly). Then we have a simple function to read the file, return its data and sent it to our queue.
Now, a previous step for us to be able to run and test this function is to configure a test event and we do it like this:
Click the dropdown menu where it says “Select a test event” and then click on “Configure test events”. This is going to open the next dialog:
You only have to define an event name and leave the rest as it is, then hit the “Create” button. You’ll be back at the Lambda function screen and you’ll see your test event selected. Now click on the “Test” button.
It’s possible that you get an error message like this:
Don’t worry. This happens whenever you left the Lambda screen inactive for a while. You can hit the “Test” button again and this time it should run properly and display you a more friendly message like this one with the pretty “it-all-went-well” green background:
Now, if we go to our queue, we’ll see that it has a new message.
To verify that the message was properly sent we are going to right click on the queue and select the “View/Delete messages” option:
This opens the next dialog. Click on the “Start Polling for Messages” button in order to see the new messages in case you don’t see anything yet
And now we can see our kitty!:
And the cat that inspired everything:
