AWS serverless platform branding as “lambda” provides an awesome platform to build and solve unique development problems — I highly recommend adding this to your toolset to make you well rounded if you are a developer.
At 1M free requests per month and 400,000 GB-seconds of compute time per month you can’t beat the price. After that, because it is serverless you only pay for the time your function is invoked.
Anything involving scheduled data transfers at specific times is probably the strongest use case for AWS Lambda. Beyond this, the use cases here are really limitless for processing data on a regular cadence.
Outside Trigger Events
Anything that can live outside of a continuous dedicated server and help save costs — For example processing a trigger from a customer submitting their email and sending them a confirmation to their email.
How to Start
In your AWS Console navigate to Lambda>Functions>Create Function
Select your language of choice and hit “Create Function.” This will then bring you to your Configuration tab where all the magic happens.
If you are following along, my example below is using Python.
Project Working Locally
Before you really start diving into deploying your project on Lambda, you should make sure it runs successfully locally. When building the project use a virtual environment b/c this will be important later for creating your deployment package onto AWS.
To create it:
$ pip install virtualenv
$ cd my_project_folder
$ virtualenv venv
To activate it:
$ source venv/bin/activate
To run your project:
$ python handler.py (or whatever you call your handler file)
Ultimately, you need to structure your project and code to be able to execute fully from invoking a single function within one of your python files. This function is called the Lambda Function Handler.
For example, I have a python file in my project bundle called “meanreversion_script.py” and then my lambda function handler is called “lambda_handler(event, context)” — The below Gist illustrates this:
You can call this file and the lambda function handler whatever you want, that isn’t the important part. The important part is telling AWS where to point to when your function is invoked — see the screenshot below from the lambda configuration console where the ‘Handler’ box is filled out exactly the same as my snippet of project code above. If this isn’t in sync, your function will never get invoked.
Compiling Packages before
If you have any packages that are pre-compiled C code like Pandas and NumPy you will have to precompile them for it to work with AWS Lambda.
If you don’t want to fire up a linux EC2 instance and do this yourself, you can always fine somebody that has already done this for you — The beauty of open source : )
Once you are confident that your project works fine locally you will have to rearrange the project structure in preparation for deployment — This AWS documentation does a good job explaining. To summarize from a high level, you have to “flatten” your project structure and move all your python dependencies to the root of the project (taking your 3rd party packages out of your virtual environment).
Creating Source Bundle and Deploying
Once you are confident that your project works fine locally and your file structure is correct, then you are ready to create a deployment bundle — This AWS documentation does a good job explaining how to do that.
If you source file is too large, AWS will let you know and you will have to upload first to S3 and then point Lambda to this S3 link — You then can delete the S3 storage afterwards if you desire to just keep you project zipped locally.
As always, pay attention to your logs (CloudWatch Logs) if something is failing or not acting as expected when you are testing invoking your function after uploading it to AWS Lambda.
Python Auto-tweet Twitter Example
Like most things in tech, the best way to understand is by example — So I built a Twitter Bot that tweets regularly at a certain times every day w/ variable tweets after running module code on historical stock data.
Things I had to consider (for this specific example)
One of the most disappointing things about AWS Lambda is functions are limited to 5 minutes of being invoked, but there are ways to “hack” this.
Because your lambda function can accepted an “event,” I created a CloudWatch Event that targeted my Lambda function and passed in JSON.
You may have noticed earlier, that my lambda handler function accepted an “event” variable and parsed out the values of the JSON. Here it is again for reference:
By doing so, I am able to iterate through all my data and successfully tweet everything I wanted by limiting the amount of data that gets processed each time invoked, therefore satisfying the 5 minute rule of AWS Lambda.