Creating Serverless Workboard with Netlify Lambda— Part One
Infrastructure running our applications has gone through many changes in recent years — from using dedicated machines to run even simple applications, virtual machines reducing the cost of infrastructure to containers that simplified deployment and maintenance. To provide more development time and simplify application development, large companies like Amazon, Google, and Microsoft now offer an entire ecosystem for all applications allowing you not to think in terms of servers, but instead, you think of infrastructure as part of the workflow of your application.
Term serverless has been described differently during the past few years. To make it clear, I will use term serverless to describe an application that runs in third-party stateless event-driven compute containers. In this architecture, a unit of application logic is a single function executed in these containers — which is where we get term FaaS (Function as a Service). The benefit of this approach is that the application developer is only focused on application, not on infrastructure. Infrastructure is in total control of third-party, scales automatically and charges per function execution (idle time is free of charge).
The downside is that we relinquish control over infrastructure to third-party making the debugging and monitoring process more difficult as we depend on how detailed are these services and how good is local development tooling provided by the third-party. Luckily, this is getting more robust every day.
The other downside is usually vendor lock-in, although functions should be easily deployable to different providers. From the business perspective, additional risk is an unpredictable expense that can increase significantly depending on the usage of functions.
Currently, most popular providers are:
Netlify Lambda
In this example, we will use Netlify Lambda to create functions needed to power client application that manages simple workboard application.
In the traditional approach, to run this application, we would require:
- Client application that contains client logic
- Service application that contains backend logic
- Server that runs the client application
- Scalable infrastructure for service application
- CDN service for security and speed optimisation
- Database cluster
- Preferably caching layer for service application
Using the serverless Netlify approach, we will try to reduce this to:
- Client application that contains client logic
- Functions that execute application logic and communicate with the database
- Netlify infrastructure
- Managed database service
Netlify infrastructure provides automatic atomic deployment of client application and functions from a git repository, offers powerful CDN out of the box as well as other features like branch deploys, split testing, client application pre-rendering, automatic deployment and packaging of functions to AWS, DNS management, proxy and redirect rules, unlimited snapshots and rollbacks, cache invalidation, etc.
Netlify uses AWS lambda behind the scenes, but enhances developer experience by versioning, building and deploying functions along with the client application. It also implements a custom API gateway that handles service discovery and the ability to rollback deploy. This process is totally seamless and the developer only needs to worry about the application logic inside functions.
Project Setup
We will start this project by creating a client application in Vue (popular JavaScript framework) which will be the base of the project. The entire project will be contained in a single repository.
To create a new Vue application, we will use vue-cli command:
vue create workboard-lambdaFrom the CLI wizard, choose features: Babel, PWA, Router (with history mode), Vuex, CSS pre-processors (SASS with Dart), Linter (ESLint with Airbnb config), Unit Testing (Jest). Choose to save the configuration to dedicated config files.
To keep the initial setup clean, we will remove About.vue from files and router and rename HelloWorld.vue component to WorkboardCard.vue and remove entire generated content. The initial setup can be checked on branch initialSetup inside git repository.
We will immediately connect the git repository with Netlify to have this initial project built. Go to Netlify Application and click New Site from Git button. Follow instructions to authorise Netlify with GitHub and choose git repository to connect to. Once connected, enter build command and deploy the site.
After the build, the site is available at a given domain. If you wish, you can set up a custom domain in Domain Settings section.
Although it is possible to develop functions without additional tools, we will use Netlify Dev (part of Netlify CLI) for local development.
Netlify Dev brings the power of Netlify’s Edge Logic layer, serverless functions and add-on ecosystem to your local machine. It runs Netlify’s production routing engine in a local dev server to make all redirects, proxy rules, function routes or add-on routes available locally and injects the correct environment variables from your site environment, installed add-ons or your netlify.toml file into your build and function environment.
To use Netlify Dev, first install Netlify CLI as a global package:
npm install -g netlify-cliThen log in to Netlify:
netlify loginYou can now connect to your remote Netlify site meaning that Netlify Dev will pull all settings locally, so your local environment is in sync with remote. Consequently, local development is much easier and without changes when it comes to pushing to production. Use link command to link to your remote site:
netlify linkChoose from a list of sites to connect. Once linked, Netlify Dev creates .netlify folder inside your project with state.json file that contains site identification. From now we can run all cli commands inside the project directory.
To start the project locally, use netlify dev command. It will detect your project and run internal development environment command (in our case this is vue-cli-service serve command). It also starts Netlify local server that serves as a proxy endpoint for the client application, functions, add-ons and redirects. By default, Netlify chooses a random free port to run the dev server. You can override this with port parameter:
netlify dev -p 8888Local server is now ready on http://localhost:8888.
Creating Simple Function
To create a function and serve it locally, it is required to create netlify.toml configuration file in the root directory of the project.
[build]
command = "npm run build"
functions = "lambda"
publish = "dist"Using build block inside configuration we are defining build command, functions directory and publish directory. Now we can create functions.
Netlify Dev offers commands for creating a function out of templates for easy learning. To create function this way use command:
netlify functions:createChoose hello-world template and name your function echo. Command will create sample function that you can run using command:
netlify functions:invoke echoAlternatively, you can visit http://localhost:8888/.netlify/functions/echo in your browser to get JSON result with hello message.
Since we wish to use the same JavaScript style (like using module syntax in Vue), we will not create functions directly in functions directory, but inside src/lambda directory where Netlify dev will perform a compilation of source code and output it to root lambda directory. For this, we can use any build tool we wish, but as a convenience, we will use netlify-lambda. First, install it as a project dependency:
npm install netlify-lambda --saveAdd build script to package scripts:
"build-lambda": "netlify-lambda build ./src/lambda"Now, when we execute npm run build-lambda, the tool will compile source functions to production-ready functions. Netlify Dev detects this script and recompiles function when the source changes.
Once we push changes to git, Netlify will start build and deploy process. When the process finishes, site and functions are available on the deployed site with the ability to revert to an older version in Deploys section of Netlify Application.
Project in this stage can be checked at setupLambda branch of the git repository. To execute remote function request URL ${baseSiteUrl}/.netlify/functions/echo.
Anatomy of a JavaScript Serverless Function
Serverless JavaScript function needs to export thehandler method. Name of the JavaScript file determines the name of the function. As demonstrated, every function can be invoked by requesting URL ${baseSiteUrl}/.netlify/functions/${functionName}. Function can also be triggered by internal Netlify events. We can use async to omit the last callback parameter recommended to return success or error:
Netlify provides two parameters in thehandler function. Event is an object that contains information about the request. It contains properties: path, headers, httpMethod, queryStringParameters, body which is JSON string representation of request payload and isBase64Encoded flag that indicates the encoding of payload. Inside context, Netlify provides information about the context in which function is invoked like clientContext in which we have identity and user information stored. Identity usage with functions will be shown in future parts of this article series.
To inspect theevent, we will add it to the echo function.
Once executed, this is the response:
${baseSiteUrl}/.netlify/functions/echo?testQuery=true:
In the response, data from headers is removed due to security reasons and for brevity. It contains standard headers along with the custom ones.
Connecting to Database Inside Function
To test getting data from the database we will connect to MongoDB named workboard and pull board collection documents to show in the output. You can create a free MongoDB database on Atlas. Create a database workboard and collection board that several documents with single property (along with _id) called name. Example of document:
{
"_id": {
"$oid": "5e0b43311c9d44000063556e"
},
"name": "Netlify Dev Release"
}Create new board.js function inside src/lambda directory:
It is visible from the code that we are using environment variables. Where do they come from? Environment variables come from netlify.toml configuration file or environment variables set on Netlify Application project Deploy Settings. Since this information is secret, we are setting it inside the Netlify Application project settings. Once netlify dev is started locally, remote environment variables are pulled locally so we can run functions without changing anything.
If we inspect bundled (compiled) board function we can see that it is pretty large, nearly 0.5MB. This is because it contains bundled dependencies, in this case, MongoDB node driver.
Netlify Application also offers functions view where we can check the execution log of each function. This is the example log where we can see that once started from a cold state, the function does not have initialisation duration added to execution time. It also shows memory usage per function execution.
2:34:24: Duration: 698.72ms Memory: 80MB Init Duration: 186.89ms
2:34:26: Duration: 631.54ms Memory: 81MB
2:35:03: Duration: 645.15ms Memory: 83MB
2:35:17: Duration: 634.43ms Memory: 83MB
2:35:18: Duration: 658.80ms Memory: 83MBIn this stage, project can be checked at setupDatabase branch of the git repository.
Optimising Function Deploy and Execution
The first thing before going further into the development of functions is to check for possible optimisations. It is best to follow AWS instructions on best practices to work with lambda functions.
In the current code, we are instancing MongoClient on each execution. We need to move this outside of the function handler so that the connection is cached and reused during the function lifecycle.
When we check the function execution log now, we get quite different results:
3:31:13: Duration: 704.53ms Memory: 80MB Init Duration: 193.64ms
3:31:17: Duration: 90.32ms Memory: 80MB
3:31:19: Duration: 89.91ms Memory: 80MB
3:31:20: Duration: 90.34ms Memory: 81MB
3:31:22: Duration: 95.28ms Memory: 82MBDuration time is decreased by almost 85% after the initialisation!
Another small optimisation we advise in this scenario is to set callbackWaitsForEmptyEventLoop property of context object to false. I have not seen any significant duration changes when this is set. The explanation is in AWS documentation and MongoDB best practices:
By default, the callback waits until the runtime event loop is empty before freezing the process and returning the results to the caller. Setting this property to false requests that AWS Lambda freeze the process soon after the callback is invoked, even if there are events in the event loop. AWS Lambda will freeze the process, any state data, and the events in the event loop. Any remaining events in the event loop are processed when the Lambda function is next invoked if AWS Lambda chooses to use the frozen process.
Other best practices can be found here.
In the Part Two we will create entire API for our application using Netlify functions.
