Serverless configuration for our URL shortener service
Details on our serverless.yml and serverless plugins used.
Earlier in Q1 2021 my colleague Mohamed Barakat gave an introduction to the custom URL shortener service we built at Homeday with an outlook towards more in depth articles as a technical series.
In this article we want to kick off this series by shedding some light on our serverless configuration file and the serverless plugins in use which enabled us to develop, test and deploy the service.
Recap
But before we take the deep dive let us have a quick recap of the service by utilizing some architecture map.
As depicted in (1) the λ application offers 2 public endpoints; resolve for doing a lookup of an shortened longUrl based on a shortId and shorten to store a longUrl in a dynamo db table and returning the shortUrl containing the shortId for later retrievals. In addition for the shorten funtion an authorization function verifyToken is used to only allow authorized calls.
Configuration
In case someone is interested in also building her own shortening service here is a listing of our current configuration:
Let us go through the sections one by one but I will try to keep the level of detail at a reasonable level.
provider.environment
Here we set some environment variables to access the dynamodb table based on the current stage. For stage dev the table name would be url-shortener-dev and the endpoint will default to http://localhost:8080 (although this is only relevant for offline mode).
More relevant is the HOSTING_URL which will be resolved by doing a function call to hosting.js to resolve the environment host by stage at deployment time (see the serverless guides) while creating the CloudFormation template.
This is because the shorten function does not know how the resolve function can be reached from the outside so it has to know this hosting detail in order to beeing able to return a valid shortUrl.
provider.iamRoleStatements
As the name implies we need to allow our λ application to access AWS ressources through specific IAM roles on action level:
- dynamodb: reading and writing to the table
- dynamodb index: reading the secondary index
- Systems Manager (SSM) Parameter Storage / Key Management Service (KMS): reading and decrypting the JWT secret
Reading and writing to the dynamodb table is quite obvious. But as soon as we wanted to check for existing entries for the same longUrl we noticed that this only can be done by introducing a secondary index for it (the main index is using the shortId). On IAM level the secondary indexes have their own name and needs to have specific access granted!
As mentioned above we are using a verifyToken function to authenticate calls to the shorten function by utilizing a JWT secret. In contrast to the hosting urls stored in plain text in the repository for the secret we wanted to have a more secure solution. Injecting it at deployment time would also be no option since this would still be a readable part of the CloudFormation template (and also only be changeable by doing a deployment). So we decided to go with an encrypted key in SSM Parameter Storage with corresponding IAM access settings (see the action sections for ssm::GetParamter, ssm::GetParamters, ssm::DescribeParamters and kms::Decrypt ).
functions
Nothing special in here. As mentioned we have verifyToken as authorizer function configured for shorten.
ressources
For SSM we do not have to configure anything, we just need to ensure that our chosen JWT secret key is present for every environment (see (2)). An advantage of having these keys there is that we can change keys at runtime (when a key gets corrupted for example).
So this caption only has configuration for the dynamodb table. The table setup is quite simple; we want to store a shortId and use it as key so we can do lookups through it.
Additionally we need another index to do lookups based on the original longUrl. Here is an example of how we use the index in our shorten function to check for existing entries:
Actually having this check is optional but we wanted to have some kind of flood prevention in place at least for existing original longUrls. (also not hardened against race contions though).
One could also discuss if the table setup itself should really be part of the CloudFormation template. So there is also the option to define and manage the dynamodb table outside of the project. But since we do not expect many changes and/or migrations of the table we sticked to having it part of the serverless.yml.
⚠️ But personally I ran already into some problems while changing the table rendering the deployment to fail in the development phase which I only could resolve by deleting and recreating the table; definitily something you do not want to do in production. (but this might also be due to my lacking knowledge though 😅 )
plugins & custom
While developing the service we ran into issues once in a while and it was an awesome experience that there are quite a lot of serverless plugin outside there which will help you to resolve those. They are mainly configured in the custom section. Lets go through them:
It is hard to fully recreate the infrastructure the λ functions are running in locally but still you strive for having a rapid feedback loop local development can offer to you.
This plugin helped us to skip the authenticator method for the shorten method since there is no local SSM/KMS:
serverless-offline:
noAuth: trueHaving the dynamodb at our hands locally allowed us to check our code for collision and the proper use of the read and write queries. Which is not quite trivial when comming from SQL at the start 😄
Combined with the above mentioned plugin you can have a handy script for starting the service locally:
jethroo-air ~/Workspace/url-shortener [stable]$ npm run
...
local
sls dynamodb start & sls offline
...For SSM/KMS IAM roles we needed to have the CloudFormation pseudo parameters AWS::Region and AWS::AccountId supported, which is currently not done by serverless natively. Thats there this plugin did a great job for us.
A nice plugin helping you to setup and run tests in jest.
package
The package configuration allows you to exclude folders and files from the deployment package and can reduce the size of it tremendously.
You can check the size yourself by running serverless package which will generate the zip file to be uploaded for deployment in the folder .serverless by default.
# with package configuration
$ ls -lah .serverless | grep url-shortener.zip
-rw-r--r-- 1 carstenwirth staff 341K 9 Mär 16:29 url-shortener.zip# without package configuration
$ ls -lah .serverless | grep url-shortener.zip
-rw-r--r-- 1 carstenwirth staff 17M 9 Mär 16:32 url-shortener.zip
So having 341 kilobytes instead of 17 megabytes!
Note: You do not need to care of the node_modules folder since this is handled by serverless for you already:
Serverless: Excluding development dependencies…
$ ls node_modules | wc -l
843Compared to the packaged service (approximation only though):
$ unzip -l .serverless/url-shortener.zip -W "*/package.json" | wc -l
26Although it seems that the mere package size itself has rather no effect on the cold start durations:
bigger deployment package size does not increase cold start time (acloudguru.com)
Indeed, the functions with many dependencies can be 5–10 times slower to start. Container image size does not seem to influence the cold start duration. (mikhail.io)
I would still see this as a clean habit to only ship the minimum needed for running the λ application. Because there are other storage limitations applied in AWS as well:
Conclusion and Outlook
Even though the URL shortener is quite a simple service there is still some thoughts needed for elaborating a production ready serverless configuration. In this article we showed our current configuration and the reasoning behind the different decisions. Furthermore we describe the plugins in use which helped us to develop, test and ship the service throug the different development stages. Hopefully these insights can help fellow engineers to build their own λ applications or resolve issues they are running into.
Further reads on the URL shortener service can be found here:
- How to build your own serverless URL shortener service
- Collision handling in our URL shortener service
Moreover we appreciate any feedback, and if you have a question, feel free to ask in the comments below. If you liked the article, give us a 👏 .
If you want a chance to work on cool projects like these at Homeday, we’re looking for engineers to join our team.
Thanks To
Ilyas Amezouar and Mohamed Barakat for pairing on this project. And Sinisa Grubor and Gepser Hoil for their reviews and awesome feedback!
