AWS Lambda Easily Migratable to Traditional Servers
Background
As discussed in my previous article, “Bookmarks: Migration From AWS EC2 to AWS Lambda Reduced My Bills by 35 Times” [1], I encountered the need to implement a REST server using Lambda functions. This prompted me to consider the approach I should take for Lambda function implementation, keeping in mind the possibility of seamlessly switching back to a traditional web server with minimal effort. Serching a solution that adheres to best design principles, ensuring a clean and maintainable codebase, I decide to take some of the principles outlined in the widely recognized “Clean Architecture” by Robert Martin [2] book, approach I saw in one of the conferences [3] and various design patterns. Leveraging solutions I had previously implemented for traditional server solutions, I aimed to apply these principles to Lambda functions. If you’re curious about implementing a Lambda function that can effortlessly transition to a traditional serverbased solution with minimal adjustments, while adhering to the best design patterns, this article can be interesting for you.
Lambda Design Approach
The proposed design approach for the Lambda function is rooted in the principles outlined in [2]. The separation of layers is based on the responsibilities described in Fig. 1. I first encountered an implementation of this approach in a Node.js environment at a conference in Kharkiv, Ukraine, around 5 years ago, presented by Viktor Turskyi. You can find one of his presentations here [3]. My friends and I were impressed with this approach right from the moment we saw it. Since this moment, we have successfully applied it in our pet projects and some production solutions, witnessing its effectiveness in creating clean, maintainable, and flexible applications.
This architectural design facilitates swift changes in technologies for even large-scale projects. For instance, transitioning from Express to Koa or Fastify can be accomplished in a matter of hours, as you only need to rewrite the controller layer. The same flexibility extends to database changes.
Fig. 1. Schema described clean architecture [2]
Coming closer to the solution for a REST Server implemented in a Node.js project (I have also implemented services for ZMQ, WebSocket, and Telegram provided interfaces), the primary abstraction layers are:
- Router: This layer is responsible for directing incoming requests to the appropriate controllers or handlers based on the requested route.
- Controller: The controller handles input, processes application-specific business logic, and coordinates interactions between entities and use cases. It ensures the separation of concerns, maintaining a clear boundary between the application core and external frameworks.
- Usecase: Encapsulates the application’s business rules and logic, sitting at the core of the application. Surrounded by other layers, such as the interface adapters, entities, and outer layers like frameworks and drivers, the use case plays a crucial role in maintaining a structured architecture.
- Model: The model represents data structures, business rules, and methods for interacting with the data. Models are often used to abstract and represent real-world entities or concepts in the application.
- Repository: Responsible for abstracting the data access implementation, the repository provides an interface for the application’s core business logic to interact with the database or any external data source.
The interaction between described layers implemented as follows: the signal arrives at the router and, following the described rules, it is directed to the specific controller, which, in turn, calls the relevant usecases with the described business logic. This high-level business logic, as described in the usecase, manipulates models that implement lower-level data manipulation. The models collaborate with the repository to store, remove, and retrieve data from some storage. All these layers communicate with each other only at the interface level and know nothing about the implementation details of each other. This adherence to the main principle enables the application to remain loosely coupled and easy to maintain.
The core of the application resides at the models level, where the primary business logic is implemented. This layered architecture allows for independent changes to each layer without affecting others. As mentioned earlier, if you decide to change a technology, such as moving from Express to Fastify, you just need to modify the router layer and perhaps make some small enhancements. However, the models and repository will remain unaffected, offering significant flexibility. You can even switch from one protocol to another, transitioning from HTTP to WebSocket, without impacting the core functionality. And this is the greatest thing in this approach.
Lambda Example
Let’s move from theory to practice, and I want to illustrate a Lambda function example that follows the architectural approach I’ve described. As outlined in [1], I migrated from EC2 to Lambda functions, resulting in a 35-fold reduction in costs. You can find this solution on GitHub.
The Bookmarks application I described is a simple web server that provides a basic interface for storing, removing, and reading bookmarks, user tags (bookmark categories), users, and authentication. Since describing all of this interfaces is quite extensive, I will focus on explaining only one interface responsible for bookmarks data.
The Lambda function’s folder structure, reflecting its low-level architecture, can be described as follows:
bookmarks/ // Root project folder
|-- .github/ // GitHub-related configurations
| |-- ... // GitHub configuration files
|-- apps/ // Folder containing applications
| |-- client/ // Client application
| | |-- ... // Client application files
| |-- lambdas/ // Set of lambda functions
| | |-- auth/ // Auth Lambda function
| | | |-- ... // Auth Lambda function files
| | |-- bookmarks/ // Bookmarks Lambda function
| | | |-- models/ // Bookmarks models
| | | | |-- BookmarkCreateDto.js // Describes bookmark create DTO
| | | | |-- BookmarksRepo.js // Bookmarks repository
| | | | |-- BookmarkUpdateDto.js // Describes bookmark update DTO
| | | | |-- index.js // Folder entry point
| | | |-- controller/ // Bookmarks controller
| | | | |-- index.js // Controller implementation
| | | |-- usecases/ // Bookmarks usecases
| | | | |-- CreateBookmark.js // Create bookmark usecase
| | | | |-- DeleteBookmark.js // Delete bookmark usecase
| | | | |-- ReadBookmark.js // Read bookmark usecase
| | | | |-- UpdateBookmark.js // Update bookmark usecase
| | | | |-- index.js // Folder entry point
| | | |-- shared>/ // Symbolic link to a shared folder
| | | |-- constants.js // Lambda function constants
| | | |-- index.js // Lambda function entry point
| | | |-- package.json // Bookmarks lambda npm configuration
| | |-- tags/ // Tags Lambda function
| | | |-- ... // Tags Lambda function files
| | |-- spaces/ // Spaces Lambda function
| | | |-- ... // Spaces Lambda function files
| | |-- shared/ // Folder containing shared functionality
| | | |-- constants/ // Shared constants
| | | | |-- index.js
| | | |-- models/ // Shared models
| | | | |-- DatabaseClient.js // Database client
| | | | |-- Errors.js // Describes HTTP errors
| | | | |-- JWT.js // Functionality related to JWT
| | | | |-- index.js // Shared models entry point
| | | |-- system/ // Shared system functionality
| | | | |-- index.js // Router and system level utilities
| | | |-- usecases/ // Shared usecases
| | | | |-- index.js // Usecases entry point
| | | | |-- UserValidate.js // Validation usecase
| | | |-- utils/ // Shared utilities
| | | | |-- parsers.js // Parser functions shared between the lambdas
| | | | |-- formatters.js // Formatters shared between the lambdas
| | | | |-- index.js // Utilities entry point
|-- dev-ops/ // DevOps-related configurations
| |-- ...
|-- ...The /apps/lambdas/bookmarks/index.js serves as the entry point for the Lambda function. It initialises the database client and implements Lambda routing.
// /apps/lambdas/bookmarks/index.js
import { BookmarksRepo } from './models/index.js';
import { DatabaseClient, JWT } from './shared/models/index.js';
import { router } from './shared/system/index.js';
import { Controller } from './controller/index.js';
const JWT_SECRET = process.env.JWT_SECRET || 'JWT_SECRET';
JWT.setSecret(JWT_SECRET);
BookmarksRepo.setRepository(new DatabaseClient('prod_bookmarks_table'));
export const handler = async (event) => {
try {
return router(Controller)(event['httpMethod'], event);
} catch (error) {
return {
statusCode: 500,
body: JSON.stringify({ error: error.message }),
headers: {
'Content-Type': 'application/json'
}
};
}
};The JWT secret is read from environment variables and set as a static property in the JWT class, which implements methods such as token sign ing and token verification. This secret is then utilized to ensure the validity of user data during the request verification process.
// /apps/lambdas/shared/models/JWT.js
import jwt from 'jsonwebtoken';
export class JWT {
static secret = null;
static setSecret(secret) {
JWT.secret = secret;
}
constructor(payload) {
this.secret = JWT.secret;
this.payload = {
_id: payload._id,
email: payload.email,
...
};
}
sign() {
return jwt.sign(this.payload, this.secret);
}
verify(token) {
return jwt.verify(token, this.secret);
}
}The DatabaseClient represents low-level access to the database. In this particular example, it is a DynamoDB connection and implements basic methods. This implementation is slightly simplified and does not include any data locks to prevent race condition issues.
// /apps/lambdas/shared/models/DatabaseClient.js
import AWS from '/var/runtime/node_modules/aws-sdk/lib/aws.js';
export class DatabaseClient {
#table = null;
constructor(tableName) {
this.#table = new AWS.DynamoDB.DocumentClient({
params: { TableName: tableName }
});
}
async save({ pkValue, skValue, data }) {
try {
await this.#table.put({
Item: { PK: pkValue, SK: skValue, data },
}).promise();
return data;
} catch (e) {
console.error('[DatabaseClient] save: ', e);
}
}
async update({ pkValue, skValue, data }) {
try {
return this.#table.update({
Key: {
PK: pkValue,
SK: skValue,
},
UpdateExpression: 'SET #dataAttr = :dataValue',
ExpressionAttributeNames: {
'#dataAttr': 'data',
},
ExpressionAttributeValues: {
':dataValue': data,
},
ReturnValues: 'ALL_NEW',
}).promise();
} catch (e) {
console.error('[DatabaseClient] save: ', e);
}
}
async readByPk(pkValue) {
try {
const result = await this.#table.query({
KeyConditionExpression: 'PK = :pkValue',
ExpressionAttributeValues: { ':pkValue': pkValue }
}).promise();
return result['Items'];
} catch (e) {
console.error('[DatabaseClient] read: ', e);
}
}
async readByPkSk(pkValue, skValue) {
try {
return this.#table.get({ Key: { PK: pkValue, SK: skValue },}).promise();
} catch (e) {
console.error('[DatabaseClient] read: ', e);
}
}
async remove(pkValue, skValue) {
try {
await this.#table.delete({ Key: { PK: pkValue, SK: skValue } }).promise();
} catch (e) {
console.error('[DatabaseClient] remove: ', e);
}
}
}The DatabaseClient instance, upon Lambda function call, is set as a static property in the BookmarksRepo class and is then accessible during the model invocation.
// /apps/lambdas/bookmarks/models/BookmarksRepo.js
import { getTableKey } from '../shared/utils/index.js';
import { BOOKMARKS } from '../constants.js';
export class BookmarksRepo {
repository = null;
static repositoryInstance = null;
static setRepository(repository) {
BookmarksRepo.repositoryInstance = repository;
}
constructor() {
this.repository = BookmarksRepo.repositoryInstance;
}
async save(data) {
const pkValue = getTableKey(BOOKMARKS, data.owner);
const skValue = data.url;
return this.repository.save({ pkValue, skValue, data });
}
async update(data) {
const pkValue = getTableKey(BOOKMARKS, data.owner);
const skValue = data.url;
await this.repository.update({ pkValue, skValue, data });
return data;
}
async readByOwner(owner) {
const pkValue = getTableKey(BOOKMARKS, owner);
const records = await this.repository.readByPk(pkValue);
return records.map(record => record['data']);
}
async readByUrl(owner, url) {
const pkValue = getTableKey(BOOKMARKS, owner);
const record = await this.repository.readByPkSk(pkValue, url);
return record && record['Item'] && record['Item']['data'];
}
async readById(owner, id) {
const pkValue = getTableKey(BOOKMARKS, owner);
const records = await this.repository.readByPk(pkValue);
return records.map(record => record['data']).filter(bookmark => bookmark._id === id);
}
async remove(owner, skValue) {
const pkValue = getTableKey(BOOKMARKS, owner);
return this.repository.remove(pkValue, skValue);
}
}Once the DatabaseClient is initiated, and static properties for JWT and BookmarksRepo are set up, the Lambda function handles the signal. The function responsible for handling this signal is traditionally called a handler. This handler invokes the router to route to the specific business functionality through the controller. The Lambda function router is a reusable function across different Lambda functions and is stored as shared system components. In this file, there is also a described request handler function, which is the main function that invokes the usecases, handles errors, and logs request information. This function is called requestHandler and is used in the controller.
// /apps/lambdas/shared/system/index.js
import { HTTP_METHOD } from '../constants/index.js';
import { parseRequest } from '../utils/index.js';
export const defaultHeaders = {
'Content-Type': 'application/json',
'Access-Control-Allow-Origin' : process.env.BOOKMARKS_DOMAIN ? `https://${process.env.BOOKMARKS_DOMAIN}` : 'https://bookmarks.lat',
'Access-Control-Allow-Methods' : 'GET, OPTIONS, POST, PUT, DELETE',
'Access-Control-Allow-Headers' : 'Content-Type',
'Access-Control-Allow-Credentials': 'true',
}
export function router(Controller) {
return async function (method, event) {
const data = parseRequest(event);
switch (method) {
case HTTP_METHOD.GET: {
return Controller.get(data);
}
case HTTP_METHOD.POST: {
return Controller.post(data);
}
case HTTP_METHOD.PUT: {
return Controller.put(data);
}
case HTTP_METHOD.DELETE: {
return Controller.del(data);
}
default: {
return {
statusCode: 404,
body: JSON.stringify({}),
headers: {
...defaultHeaders
}
};
}
}
}
}
async function runUseCase(UseCase, { params }) {
// ... validation logic
return new UseCase().execute(params);
}
export async function requestHandler(UseCase, params, mapToResponse) {
function logRequest(params, result, startTime) {
console.log({
useCase: UseCase.name,
runtime: Date.now() - startTime,
params,
result,
});
}
try {
const startTime = Date.now();
const result = await runUseCase(UseCase, { params });
logRequest(params, result, startTime);
if (mapToResponse) {
return mapToResponse(result);
}
return {
statusCode: 200,
body: JSON.stringify(result),
headers: {
...defaultHeaders
}
};
} catch (err) {
console.error(`[ErrorHandler] ${JSON.stringify(err)}`);
return {
statusCode: err.statusCode,
body: JSON.stringify({ message: err.message }),
headers: {
...defaultHeaders
}
}
}
}The bookmarks router directs the signal to the bookmarks controller, which invoke usecases with the specific parameters.
// /apps/lambdas/bookmarks/controller/index.js
import { ReadBookmark, CreateBookmark, DeleteBookmark, UpdateBookmark } from '../usecases/index.js';
import { requestHandler } from '../shared/system/index.js';
export class Controller {
static async get({ body, cookie, param }) {
return requestHandler(ReadBookmark, { cookie })
}
static async post({ body, cookie, param }) {
return requestHandler(CreateBookmark, { data: body, cookie })
}
static async put({ body, cookie, param }) {
return requestHandler(UpdateBookmark, { data: { ...body, _id: param }, cookie })
}
static async del({ body, cookie, param }) {
return requestHandler(DeleteBookmark, { _id: param, cookie })
}
}One of the usecases is the “CreateBookmark” usecase. The usecases and their invocation are implementing the command pattern [4].
// /apps/lambdas/bookmarks/usecases/CreateBookmark.js
import { BookmarksRepo, BookmarkCreateDto } from '../models/index.js';
import { UnprocessableEntityError } from '../shared/models/index.js';
import { UserValidate } from '../shared/usecases/index.js';
export class CreateBookmark {
async execute({ data, cookie }) {
const jwtContent = await new UserValidate().execute(cookie);
const bookmark = await new BookmarksRepo().readByUrl(jwtContent.email, data.url);
if (bookmark) {
throw new UnprocessableEntityError(`Bookmark with url ${data.url} already exist`)
}
return new BookmarksRepo().save(new BookmarkCreateDto({ ...data, owner: jwtContent.email }));
}
}And that’s generally it. There are a couple of things that were not described, such as other usecases, error models, and the validation layer. You can find them in the Bookmarks [5] repository if you are interested.
Conclusion
Adopting the outlined approach to Lambda function design offers several significant advantages. One of the most significant benefits is the ease of transitioning from Lambda back to server base solution with minimal effort. This simplifyed process will involves rewriting the router, and if there’s a need to change storage, modifying the DatabaseClient without altering the interface. It is important to emphasise that the models, including usecases, remain unaffected. This level of flexibility is invaluable, particularly in real-world production scenarios.
Moreover, this architecture embodies various best practices, design patterns, and principles, notably adhering to the SOLID principles as described in [2]. The application of these principles ensures a robust, scalable, and maintainable design that stands the test of production experience. Overall, this Lambda function design approach not only facilitates seamless transitions between serverless and traditional server environments but also upholds fundamental principles that contribute to the maintainability of the solution.
Links
- Bookmarks: Migration From AWS EC2 to AWS Lambda Reduced My Bills by 35 Times
- Clean Architecture: A Craftsman’s Guide to Software Structure and Design. Robert C. Martin
- https://www.slideshare.net/fwdays/viktor-turskyi-effective-nodejs-application-development
- https://refactoring.guru/design-patterns/command
- https://github.com/yzhbankov/bookmarks