Serverless != Effortless

My attempt at simplifying serverless RESTful API Design, Development & Deployment on AWS.

“Serverless” is the new paradigm in cloud-computing. All major cloud-service providers support this idea of deploying code (functions) to fully managed containers. The users do not have to worry about planning and managing any infrastructure. AWS supports this idea with a couple of its product offerings, namely Lambda and API Gateway.

While building on these services seems straightforward , there is still a significant amount of configuration and wiring that needs to happen. This process is manual and error-prone. There have been several attempts at simplifying the design, configuration and deployment process. I will provide an overview of these evolving integration services/products and then talk about my own attempt at simplifying the process. While serverless computing can be used in several use-cases, I will focus on designing, building & deploying RESTful APIs.

AWS offerings for Serverless RESTful APIs

  • AWS Lambda, — With AWS Lambda, you can run code for virtually any type of application or backend service — all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. As of this writing, AWS Lambda supports Java, Node.js, C#, and Python. I am sure more languages will be supported in future.
  • API Gateway — Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. The API’s can act as a front-end for the backed-implementation on AWS Lambda. (API Gateway can be configured as a front-end for implementations on other AWS offerings like EC2 as well)

Whats involved in designing,developing & deploying serverless RESTful API on AWS ?

The first-step would be to plan out or define the APIs. There are several tools for modelling the APIs. I have found the following to be more widely adopted — Swagger and RAML — and my personal favourite is RAML. We will see some examples in later sections.

Assuming the design is complete , we will then have to provide the back-end implementation using our preferred language/platform. In providing the implementation , we will need to know a little about the requirements of the AWS Lambda — specifically what is expected in order to deploy the code as serverless functions. Most probably we would pick some sample code for a Lambda function and modify the same.

Once the implementation is in place, we would like to deploy and test the API. We would typically like to test this in a DEV/Staging mode and later be able to push it to a PROD environment.

Here are the typical steps to deploy the implementation to AWS Platform :

  1. Create account
  2. Install CLI (optional)
  3. Create IAM Role for Lambda execution (provide required permissions etc.)
  4. Create AWS Lambda Function (via AWS Management Console)
  5. Associate the implementation code with the Lambda Function (direct upload or via S3 bucket)
  6. (Optional) Test the function manually, via the console.
  7. Create API on API Gateway (via AWS Management Console)
  8. Create a “resource” under the API
  9. Create one/more “method” under the resource.
  10. Associate the “method” with the Lambda function
  11. Deploy the API
  12. Modify the permissions on the Lambda function — let it be accessed via API gateway.
  13. Manually test — using tool like curl etc.

Repeat step #4 through step #13 for each “method” on each API. You might even have to repeat all the steps to promote the code from Staging to Production environment.

Is there a way to simplify this ?

As its obvious, the workflow discussed in the earlier section is in no way simple. From personal experience I can tell you that the manual steps are tedious and error-prone. And its in no way a criticism of the AWS Platform — the platform tries to make it as easy as it can — there are just that many steps.

The good news is that a lot of steps in the entire life-cycle can be automated. There have been several attempts at this, each providing a different level of automation. Let us look at pros and cons of some of the prominent initiatives related to automation for RESTful APIs..

Serverless Framework — https://serverless.com/

This is a CLI to simplify the entire deployment process. Here are some of the key features :

  • Tool written in node.js and needs to be run from command-line.
  • Provides integration across multiple cloud service providers.
  • Uses its own YAML based configuration file — serverless.yaml — to generate and manage deployments.
  • Implementation can be Node.js, Python, Java, and more
  • Supports deployment and maintenance of code running on cloud infrastructure.

Things I don’t like :

  • Very high level of abstraction — no visibility into resources created. I think developers need some insight into what is being created at the cloud service provider.
  • No support for RESTful API modelling/design or development — the tool is more geared towards deployment of existing implementation.
  • Need to learn new set of commands and options for the CLI.

SwaggerHub — https://swaggerhub.com/

This is a tool for collaborative API modelling in the popular Swagger format. As an extension, this tool also offers deployment to popular API management platforms like AWS. Here are some of the key features :

  • Good collaboration platform if you work with API definition/design in the Swagger format.
  • Helps with iterative API definition.
  • Can push API definitions to AWS API Gateway using the integrator.
  • Can also create Lambda functions using the Amazon API Gateway Lambda Sync
  • Supports Node.js and Python for functions.

Things I don’t like :

  • Need to create custom AWS API Gateway specific elements or properties in the Swagger definition file.
  • Need to provide AWS credentials — for SwaggerHub to be able to deploy directly.
  • Will have to edit the function in the AWS Console or will have to download the generated-template and edit and upload.

AWS Serverless Application Model (SAM) — http://amzn.to/2grWzUo

CloudFormation is AWS’s offering for automation in regards to creating AWS Resources. Serverless Application Model (SAM) is a mechanism to simplify the CloudFormation template creation. Basically AWS has come up with some high-level resources that we work with that get transformed to multiple low-level resources. Here are some of the key features :

  • A template created using the high-level resources is much more readable. The “CloudFormation Transform” process , changes the template to a lot of resources behind the scenes.
  • The “package” option provides a good option to upload the code to a S3 bucket and use the bucket references in an updated cloud-formation template.
  • The “deploy” option can be used to create and/or update all necessary resources on AWS.

Things I don’t like :

  • Cloudformation is an automation tool for creating resources in AWS — so naturally it does not have any built in support for API Design , Definition or Development.

From what we have seen so far, I could not find a tool or platform that provides the full life-cycle support for RESTful APIs on AWS. This prompted me to build something to simplify the entire process. In the next section I will talk about this.

My approach to RESTful API Design,Development & Deployment.

Design Phase

This is phase where we are planning the API, mapping the business entities & defining the high-level workflows. I like the tools that support the API Design process to be very simple and non-intrusive. The design tool/format should not hamper the creative process by enforcing unnecessary overheads. The design is where we are modelling the API so something that is easy to edit , play around. share & collaborate will be the most productive tool. In this regard I am a big fan of RAML(https://raml.org/) . It is a combination of a very light schema for RESTful APIs and the simplistic YAML format. Swagger is a close contender — but I believe it is good for generating documentation — it is not the best choice for API Planning & Design.

You can learn more about RAML here : https://raml.org/

Just for illustration, here is a very simple example of a RAML specification that I created to model a simple API with two resources — one to find users and other to find their group memberships.

#%RAML 1.0
— -
title: User and Group Maintenance
version: v1
baseUri: /manageusers
types:
User:
properties:
id: string
name: string
example:
id: “abc123”
name: “John Doe”
Group:
properties:
id: string
name: string
description : string
example:
id: “550e8400-e29b-41d4-a716”
name: “Group 1”
/appuser:
/{userId}:
get:
description: Check if user exists on the system
responses:
200:
body:
application/json:
type: User
/appusergroup:
/{userId}:
get:
description: Get a list of user’s group membership
responses:
200:
body:
application/json:
type: Group[]

Development Phase

Although in the context of serverless we talk about functions (sounds simple), the development involves going through the typical rigor for computer programming. To simplify this process, I would like to generate the scaffolding , so the developers/programmers can then focus on the actual implementation of their business logic.

You could use sample code templates to get started. AWS has also recently announced “AWS Serverless Application Repository” — this provides a lot of code-samples. However I am looking for something that will generate the scaffolding or template code for the APIs that I have defined.

I could not find any tool that could help with this, so I ended up writing a tool raml2serverless that generates the template code based on the API definitions in the RAML format.

Deployment Phase

As discussed earlier, this is the most cumbersome and error-prone phase. AWS has made significant contribution in this space by rolling out the Serverless Application Model (SAM). This is an extension to CloudFormation, which is AWS’s tool for automated resource deployments. This tool can be used to create and update the resource definitions to AWS Lambda (along with the links to API Gateway).

We still need to code the CloudFormation template to facilitate this. While most developers do not like to be burdened with AWS resource creation, they are still interested in having visibility into what gets defined on AWS.

I extended my tool (raml2serverless) to generate the CloudFormation templates using the SAM Transform. We can then just execute a couple commands to upload the custom-code and create the necessary AWS resources. We can maintain the source code in a repository of our choice and keep using the commands to re-deploy to AWS as and when needed. The next section provides more information on this tool.

“raml2serverless” tool

I created this tool to support the Development and Deployment phases of RESTful APIs to be deployed as server-less implementations on the AWS Platform. This tool takes the RESTful API design in RAML format and generates the following:

  • Source Code Templates — (currently for node.js). These are empty function definitions as expected by AWS Lambda. These map to the resource-methods on AWS API Gateway
  • CloudFormation Template — This has the definition for the Lambda Functions and the mapping to corresponding resource-definition on AWS API Gateway. The template is simple and readable and uses the AWS Serverless Application Model (SAM).

When we run the tool — all the artifacts (source-code and CloudFormation templates) are created in the “output” folder.

We will have to create an S3 bucket with full permissions (or use an existing one) and then run the following commands to deploy the resources. (We will have to initialize the aws cli with appropriate credentials).

This first command will upload the source code to S3 bucket and generate an updated CloudFormation template with the references to the code in the S3 bucket.

aws cloudformation package --template <cloudformation-template> --s3-bucket <S3 bucket> --output-json <updated-cloudformation-template>

The second command is used to create and/or update the AWS Resources (Roles, Policies, Lambda Functions, API Gateway Definitions and deployments)

aws cloudformation deploy --template <updated-cloudformation-template> --stack-name <stack-name> --capabilities CAPABILITY_IAM

Hopefully I have been able to provide a good overview of what is involved in building serverless RESTful APIs. Also, I hope the tool will be able to provide a jump-start in building serverless APIs on AWS. Please do let me know if you have any thoughts or suggestions.

References

AWS Lambda — https://aws.amazon.com/lambda/
API Gateway — https://aws.amazon.com/api-gateway/
RAML — https://raml.org/
Serverless Framework — https://serverless.com/-
SwaggerHub — https://swaggerhub.com/
AWS Serverless Application Model (SAM) — http://amzn.to/2grWzUo
Raml2serverless — https://www.npmjs.com/package/raml2serverless

Like what you read? Give Prasad Joshi a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.