Building a Product Catalog web API using Serverless at AWS

Overview

In this blog, we develop a Scalable, Resilient, and Fault-Tolerant Product Catalog Web API for a Microservices Ecosystem.

Description

A company wants to build a robust web API that serves as the central data source for a microservices ecosystem. The API should be scalable, resilient, and fault-tolerant while delivering product information from the company’s catalog. The API must only accept requests from authorized sources and provide functionality for updating the product catalog by other services within the ecosystem.

Preconditions

• An existing microservices ecosystem that requires a centralized product catalog.
• Access control requirements for the API to ensure requests are authorized.
• Serverless Application

Let’s discuss concepts of serverless, REST API, and its services on AWS.

✨What is Serverless computing?

Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers. Pricing is based on the actual amount of resources consumed by an application.

✨What is an API?

API stands for Application Programming Interface. APIs are mechanisms that enable two software components to communicate with each other using a set of definitions and protocols.

✨How do APIs work?

API architecture is usually explained in terms of client and server. The application sending the request is called the client, and the application sending the response is called the server.

✨What are REST APIs?

REST stands for Representational State Transfer.REST is a software architectural style that was created to guide the design and development of web architectures.REST defines a set of functions like GET, PUT, DELETE, etc. that clients can use to access server data. Clients and servers exchange data using HTTP. REST API is a special type of Web API between a web server and web browser.

Let’s see serverless services for all three layers: compute, integration, and data stores.

✨Compute

AWS Lambda

AWS Lambda is event driven, pay-as-you-go compute service that lets us run code without provisioning or managing servers. We can use AWS Lambda to extend other AWS services with custom logic, or create back-end services that operate at AWS scale, performance, and security.

AWS Lambda automatically runs code in response to multiple events, such as HTTP requests via Amazon API Gateway, modifications to objects in Amazon Simple Storage Service (Amazon S3) buckets, table updates in Amazon DynamoDB, and state transitions in AWS Step Functions.

✨Application Integration

API Gateway

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. It handles all the tasks involved in accepting and processing thousands of concurrent API calls, including traffic management, CORS support, authorization, and access control, throttling, monitoring, and API version management.

you can create an API that acts as a “front door” for applications to access data, business logic, or functionality from your back-end services, such as applications running on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS) or AWS Elastic Beanstalk, code running on AWS Lambda, or any web application.

✨Data store

Amazon DynamoDB

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It’s a fully managed, multi-region, multi-active, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications. DynamoDB can handle more than 10 trillion requests per day and support peaks of more than 20 million requests per second.

Now that we have reviewed all concepts and checked the requirements, Let’s design a solution architecture based on Amazon API Gateway, AWS Lambda, and Amazon DynamoDB whereas the products catalog will reside on DynamoDB.

Solution Architecture

Architecture Diagram

The reference architecture is a general-purpose, event-driven, web application back-end that uses AWS Lambda, API Gateway for its business logic. It also uses Amazon DynamoDB as its database to store product catalog.

Application components

Back End Application (Business Logic)

The backend application is where the actual business logic is implemented. The code is implemented by Lambda functions fronted by an API Gateway REST API. In our case, we have different Lambda functions, each handling a different aspect of the application: list the product items, get details about a specific product, update a product, create a new product, and delete an existing item. The application saves all products in a DynamoDB table.

API Requests Authorization

🔑The API Gateway only accepts requests from authorized sources. To accomplish this, we are using Lambda Authorizer to control access to the API.

🔑A Lambda authorizer is an API Gateway feature that uses a Lambda function to control access to our API. It is a way to add additional security to our API.

🔑Lambda authorizers are AWS Lambda functions. With custom request authorizers, we can authorize access to APIs using a bearer token auth strategy such as OAuth.

🔑When an API is called, API Gateway checks if a Lambda authorizer is configured, API Gateway then calls the Lambda function with the incoming authorization token. We can use Lambda to implement various authorization strategies (e.g. JWT verification, OAuth provider callout) that return IAM policies used to authorize the request.

🔑If the policy return by the authorizer is valid, API Gateway will accept the request and cache the policy associated with the incoming token for up to 1 hour.

Let’s develop and implement a functional model.

The below GitHub URL contains the Python files and JSON policy. Download and save it on your local machine.

https://github.com/Sudarkodi-Muthiah-repo/12weeksAWSworkshopchallenge/tree/main/Week8/Code

📄 Step 1: Create an Amazon DynamoDB table to use as a product catalog database

• Let’s create a DynamoDB table named acme_products with Partition key uuid and keep it as a String.
• In Customize settings, Go to Secondary indexes and create a New Global secondary index named visible-index with Partition key visible and keep it as a String and choose all for Attribute projections.
• Now create the table keeping everything else unchanged.

DynamoDB table

📄 Step 2: Create the Lambda Functions to handle our product catalog data(CRUD)

🎫Step 2.1: Create a Lambda function to create/write products to a DynamoDB table

• Navigate to lambda dashboard. In the left menu, choose Functions and choose Create Function.
• Select Author from scratch
• Function name: acme_create_product
• Runtime: Python 3.9
• Expand Change default execution role.
• Execution role: Use an existing role
• Existing role: ACMEAPILambdaExecutionRole
• Choose Create Function

Note: This role grants this Lambda function and others permissions needed to interact with DynamoDB.

The content ACMEAPILambdaExecutionRole_policy.json from GitHub should be attached to the ACMEAPILambdaExecutionRole

The policy looks like below:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Action": [
    "dynamodb:List*",
    "dynamodb:Describe*"
   ],
   "Resource": "arn:aws:dynamodb:*:*:table/acme_products",
   "Effect": "Allow",
   "Sid": "ListAndDescribe"
  },
  {
   "Action": [
    "dynamodb:GetItem",
    "dynamodb:Query",
    "dynamodb:DescribeTable",
    "dynamodb:Get*",
..........

• Move to code section and replace the the existing code in the lambda_function.py file with the code present in acme_create_product.py.
• Choose Deploy
• Choose Test and Configure Test event with the following options.
• Test event action: Create new event
• Event name: ACMECreateProductTest
• Event sharing settings: Private
• Template — optional: hello-world
• Event JSON: Copy and paste the code below. And then save.

{
    "name": "Some Cool Product Name",
    "visible": "1"
}

• Choose Test

It will open a second tab named Execution results and, if the test runs without errors, you will see a Response with a status Code equal to 200.

We just created a Lambda function that wrote data to the DynamoDB table. Check the data is added to the DynamoDB table in its management console. Go to DynamoDB -> Table-> acme_products-> explore table items.

DynamoDB Table contents

We repeat the above steps to create Lambda functions to perform remaining operations like Retrieve(Get), Update, and Delete.

🎫 Step 2.2 Create a lambda function to retrieve products from the DynamoDB table

• Create a function named acme_get_product with the code present in acme_get_product.py file.
• Configure Test event with Event JSON as below:

{
    "uuid": "replace_with_a_valid_uuid"
}

• Choose Test

It will open a second tab called Execution results and, you will see a Response with a HTTPStatusCode equals to 200 and an Item key with the data related to the product uuid you passed.

Test Execution results

🎫 Step 2.3 Create a Lambda function that will be in charge of updating product data in the database

• Create and deploy a function named acme_update_product with the code present in acme_update_product.py
• Configure Test event with Event JSON as below:

{
    "uuid": "replace_with_a_valid_uuid",
    "visible": "1",
    "name": "Some UPDATED Product Name"
}

• Choose Test

It will open a second tab called Execution results and, we will see a Response with a HTTPStatusCode equals to 200 and the Attributes key with the updated data related to the product uuid we passed.

🎫 Step 2.4 Create a Lambda function that will delete data from the database

• Create and deploy a function named acme_delete_product with the code present in acme_delete_product.py
• Configure Test event with Event JSON as below:

{
    "uuid": "replace_with_a_valid_uuid"
}

• Choose Test

It will open a second tab called Execution results and, if the test runs without errors, we see a Response with a HTTPStatusCode equal to 200. The data related to the uuid passed was deleted from DynamoDB.

🎫 Step 2.5 Create a Lambda function that is in charge of retrieving a list of products from the catalog

• Create and deploy a function named acme_list_products with the code in acme_list_products.py
• Configure Test event with Event JSON as below:

{}

• Choose Test

It will open a second tab called Execution results and, if the test runs flawlessly, we will see a Response with a HTTPStatusCode equals to 200. We will also see in this response an array of Items that corresponds to all items on your DynamoDB table.

📄 Step 3 Creating API Gateway to connect a web endpoint to lambda functions

🎫 Step 3.1 Create an API gateway

• Choose APIs in your left navigation menu.
• In Choose an API type step, choose Build inside the REST API (NOT Private) box.
• Create new API: New API.
• API name: ACME Products API.
• Description: Products API that connects a web endpoint to several Lambda functions.
• Regional
• Choose Create API

The Gateway is ready to be configured.

🎫 Step 3.2 Create API Gateway resource

Create a resource called Products. This resource will be part of the API URL

• Resource Name: products
• CORS (Cross Origin Resource Sharing): Leave checkbox de-selected.

The next step is to create methods and link them with our lambda functions.

API Gateway

🎫 Step 3.3 Create API gateway GET method

• Keep your /products resource selected.
• Click on Create method. Method type as GET
• Integration type: Lambda Function
• Lambda proxy integration: Leave toggle button de-selected.
• Lambda Function: acme_list_products (as soon as you type, it will open a selector dialog for you to select the function that already exists)
• Default Timeout: Leave toggle button selected.
• Choose Create Method

API Gateway GET method

Go to Test tab and click on Test at the bottom.

If your test succeeded, you will see the response from the Lambda Function which includes a list of products from your catalog.

Your expected Response Body output should look similar to the following:

{"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": {"Items": [], "Count": 0, "ScannedCount": 0, "ResponseMetadata": {"RequestId": "11E92NP9LN10AAA3F06U66I6T3VV4KQNSO5AEMVJF66Q9ASUAAJG", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Mon, 24 Oct 2022 17:17:55 GMT", "content-type": "application/x-amz-json-1.0", "content-length": "39", "connection": "keep-alive", "x-amzn-requestid": "11E92NP9LN10AAA3F06U66I6T3VV4KQNSO5AEMVJF66Q9ASUAAJG", "x-amz-crc32": "3413411624"}, "RetryAttempts": 0}}}

Repeat the above steps 3.2 and 3.3 for other verbs like POST, PUT, and DELETE.

API Gateway methods

• Test POST method with the below request body.

{
"visible": "1",
"name": "Some Product Name"
}

If your test succeeds, you will see the response from the Lambda Function which includes a statusCode 200.

The API Gateway passed the Request Body to your Lambda Function which got the payload and wrote to the DynamoDB table. You can go back to your table and check the new item there!

Your expected Response Body output should look similar to the following:

{"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": {"ResponseMetadata": {"RequestId": "DTCSO906O5GNKFSCHPUFED2LTRVV4KQNSO5AEMVJF66Q9ASUAAJG", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Mon, 24 Oct 2022 17:28:57 GMT", "content-type": "application/x-amz-json-1.0", "content-length": "2", "connection": "keep-alive", "x-amzn-requestid": "DTCSO906O5GNKFSCHPUFED2LTRVV4KQNSO5AEMVJF66Q9ASUAAJG", "x-amz-crc32": "2745614147"}, "RetryAttempts": 0}}}

Test PUT method with the below request body.

{
    "uuid": "replace_with_a_valid_uuid",
    "visible": "1",
    "name": "Some UPDATED Product Name"
}

If your test succeeded, you will see the response from the Lambda Function which includes a statusCode 200 and the Attributes updated.

The API Gateway passed the Request Body to your Lambda Function which got the payload and updated the DynamoDB table item related to the uuid you passed. You can go back to your table and check the updated item.

Your expected Response Body output should look similar to the following:

{"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": {"Attributes": {"visible": "1", "name": "Some UPDATED Product Name", "updated_date": "2022-10-24 17:49:52"}, "ResponseMetadata": {"RequestId": "1KUDMV5RE9M7467O509PF1HIKRVV4KQNSO5AEMVJF66Q9ASUAAJG", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Mon, 24 Oct 2022 17:49:52 GMT", "content-type": "application/x-amz-json-1.0", "content-length": "120", "connection": "keep-alive", "x-amzn-requestid": "1KUDMV5RE9M7467O509PF1HIKRVV4KQNSO5AEMVJF66Q9ASUAAJG", "x-amz-crc32": "2058753312"}, "RetryAttempts": 0}}}

Test DELETE method with the below request body.

{
    "uuid": "replace_with_a_valid_uuid"
}

If your test succeeded, you will see the response from the Lambda Function which includes a statusCode 200.

You can go back to your DynamoDB table to check if the product was deleted indeed.

Your expected Response Body output should look similar to the following:

{"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": {"ResponseMetadata": {"RequestId": "NGJ9VAMKPNRFKI461KEOMVQ87VVV4KQNSO5AEMVJF66Q9ASUAAJG", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Mon, 24 Oct 2022 17:56:13 GMT", "content-type": "application/x-amz-json-1.0", "content-length": "2", "connection": "keep-alive", "x-amzn-requestid": "NGJ9VAMKPNRFKI461KEOMVQ87VVV4KQNSO5AEMVJF66Q9ASUAAJG", "x-amz-crc32": "2745614147"}, "RetryAttempts": 0}}}

📄 Step 4 Create the Lambda function that will be in charge of the authorization validation

• Create and deploy a function named acme_api_authorizer with the code acme_api_authorizer.py

Switch to the API Gateway browser tab.

Return to the API Gateway service in the console and select APIs in the left navigation area. Your new API will be shown, copy the API ID. It will be a hash with random chars that will look like with oflu3t8kp8, copy this ID to a local notepad.

Now get the ID of your AWS account, you can get it by expanding the top navigation menu where there is your user information, just beside the region selector.

Replace the ACCOUNT ID and APIID in acme_api_authorizer.py

🎫 Step 4.1 Create an authorizer and attach it to the API methods.

• Choose the ACME Products API to enter into the API setup.
• In the left navigation menu, choose Authorizers.
• Now choose Create authorizer.
• Name: ACMEAPIAuthorizer.
• Type: Lambda
• Lambda Function acme_api_authorizer.
• Lambda Invoke Role: empty
• In the Lambda Event Payload area, inside Token Source field, write Authorization.
• Unchecks Authorization Caching

🎫 Step 4.2 Protect API methods by Lambda Authorizer

• Choose Resources in the left navigation menu.
• Select GET method related to /products resource. If you remember, this is the one who returns the list of products from your catalog.
• Inside /products — GET — Method Execution area, choose Method Request.
• In the Settings section, click on Edit.
• A selector should open and you should be able to select ACMEAPIAuthorizer . If the authorizer is not in the list, refresh the entire window and try again.
• Click Save.

Now you have one method protected by the Lambda Authorizer. The requests that send an Authorization with 123 as a value will be forwarded to the Lambda. Repeat the above steps for the remaining methods.

🎉Congratulations! We have successfully created the Lambda authorizers, methods, and resources for your API Gateway.

📄 Step 5 Deploying API

Before testing everything using a tool that simulates real life, you need to deploy your API. As your API will be also reached by web applications running in web browsers, you need to enable CORS before deploying. In this task, you will deploy your API.

• In your API Resources area, select /products resource.
• Click on Enable CORS.
• Choose default 4XX and 5XX.
• Select Access-Control-Allow-Methods: GET.
• Click on Save.

Now repeat the same for your /{uuid} resource:

In your API Resources area, select /{uuid} resource.

• Click on Enable CORS.
• Choose default 4XX and 5XX.
• Select Access-Control-Allow-Methods: GET.
• Click on Save.

Next, deploy the API.

• Select in the Resources area on top / in your API tree.
• Choose Deploy API.
• A Deploy API dialog window will pop up. Open the first selector called Deployment stage and select New Stage.
• For Stage name, type v1.
• Keep everything else empty and choose Deploy.

You will be redirected to Stages area. As you can see, it is possible to have different stages like development, production, and everything else that makes sense to your organization.

Now you also have the Invoke URL that will be used to reach your API. Copy it to your text editor for the final test.

Now it is time to test everything together.

🎉Congratulations! We have successfully deployed your API Gateway.

Go through the below Git Hub repo for step by step tutorial. https://github.com/Sudarkodi-Muthiah-repo/12weeksAWSworkshopchallenge/tree/main/Week8/mini-project

📄 Step 6 Final Test

As you want to validate your API as if an external application is accessing it, you built a web application used to test our API. We can test the API Gateway endpoint using third-party tools like Postman, CURL, Insomnia, etc.

To test our API endpoint in Insomnia, Paste the Invoke URL of our API in the URL section. Do not forget to add /products resource before passing your API endpoint. The final endpoint will be like the example below.

https://[YOUR-API-ID].execute-api.us-west-2.amazonaws.com/v1/products

Then, enter 123 in the Authorization token field. You can check the results on the right side window of Insomnia.

🎉Congratulations! We have successfully built a Product Catalog web API using Serverless at AWS.