Serverless Architecture Prototype

Serverless architecture is a new buzzword in software development. Let’s try it out with a simple example. Set up an AWS Lambda function which accesses an Amazon ElastiCache instance and also the Internet.

It looks easy at first sight, because you write a short code to access your cache service and a public API, upload it to Lambda, and you’re done. Life is not that easy though, so let’s see.

The test application waits for an URL and an expiration time, then checks this URL in our cache service, and returns it if there is a result. Otherwise, it calls the URL and stores the response to ElastiCache.

Setup and configure things

I followed this documentation, but I tried to stay on the AWS Console site: http://docs.aws.amazon.com/lambda/latest/dg/vpc-ec.html

Amazon ElastiCache

First of all, you have to define your cacher. Follow the steps on the site, create a Memcached cacher, and you’re almost done. Fill out the Cluster Name and the Number of Nodes fields. The only questionable part is the Cache Subnet Group. You don’t have predefined groups, so you can’t choose. Ok, leave it empty, and we’ll see what happens.

AWS Lambda

Use this code to define a Lambda function:

from __future__ import print_function
import base64
import elasticache_auto_discovery as ed
import requests
from pymemcache.client.hash import HashClient
elasticache_endpoint = "cacherservice.___.cache.amazonaws.com:11211"
nodes = ed.discover(elasticache_endpoint)
nodes = map(lambda x: (x[1], int(x[2])), nodes)
memcache_client = HashClient(nodes)

def handler(event, context):

    if not event.get("url"):
        raise Exception("Define an url parameter")

    url = base64.b64decode(event.get("url"))

    exp_second = 3600
    if event.get("exp_second") \
        and isinstance(event.get("exp_second"), int):
        exp_second = event.get("exp_second")

    memcached_response = memcache_client.get(url)
    if memcached_response:
        return memcached_response

    response = requests.get(url, timeout=5)
    if response.status_code != 200:
        raise Exception("Invalid URL")

    memcache_client.set(url, response.text, exp_second)
    return response.text

Create a python deployment package from the code. The only thing that you need to know is how to install all your dependencies with the following command into your project and zip your directory’s content:

pip install dependency_module_name -t your_project_dir

Don’t forget to create this role for your service: http://docs.aws.amazon.com/lambda/latest/dg/vpc-ec-create-iam-role.html

Connect Lambda and ElastiCache

Here is your first problem: trying to run your function. Lambda can’t access ElastiCache. After a thorough review of the documentation, you realise that these services should run in the same VPC because this is the only way they can communicate. But what the gosh is VPC?

VPC is a Virtual Private Cloud, which is a logically isolated part of your AWS where you can launch your resources. Let’s create a VPC and a subnet manually.

Now you have to reconfigure your cacher, and Lambda function to use this VPC. In your Lambda Configuration under Advanced Settings, choose the newly created VPC and Subnet and the default Security Group. ElastiCache follows a different approach, because you can’t modify your VPC settings on your service. Sadly, you have to delete your previously created cacher, and create a new one with the VPC setup. Define a Cache Subnet Group to access your VPC:

Choose this group in the second step when you are creating the new cacher:

Provide internet access to Lambda

After these, the Lambda function can access the cacher, but something is still not working. You defined a VPC for the Lambda function so it lost the Internet connection.

You could spot this when you modified the function

Ok, then forward internet traffic directly to your subnet. It could work, but it doesn’t because your resources don’t have public IP addresses. Related documentation section:

“If your Lambda function needs Internet access, do not attach it to a public subnet or to a private subnet without Internet access. Instead, attach it only to private subnets with Internet access through a NAT instance or an Amazon VPC NAT gateway.”

If you don’t want to struggle with AWS Console, just start a VPC wizard and choose VPC with Public and Private Subnets, and skip this part. In this case you have to reconfigure your resource again with the new VPC.

For a better understanding, you can create these manually. Rename your existing subnet to Private subnet, and create a new one called Public subnet. Create an Internet Gateway and attach to the VPC, which provides internet connection for your services. A NAT Gateway can translate private IP addresses to public, and this is what you need. Set one up for your Public Subnet.

Two Route Tables to your VPC are going to help you to forward traffic between subnets. One is associated with the Public Subnet (configure it under Subnet Associations), and routes all of the traffic to the Internet Gateway. 0.0.0.0/0 refers to the internet that goes through a subnet. The other one forwards everything to the NAT Gateway (it doesn’t need to associate with any subnet).

After the bulk of configurations, your tests for the Lambda are going to work. Then you can setup an API Gateway to access you Lambda with a public API. Don’t forget to deploy the API after you configured it.

Conclusion

I hope this documentation will help people who want to give Lambda a try. There are many pitfalls in the configuration, but after the first setup you can wrap the world in serverless APIs. AWS is fully documented, but it contains lot of hidden difficulties which are not entirely clear how to solve. More and more use-cases are needed to make it fully understandable.

I am a Data & Applied Scientist at Microsoft in Berlin, Germany. Working on Wunderlist’s data infrastructure. You can find me on LinkedIn, or Twitter if you‘d like to connect.