Serverless at LifeOmic
A high level overview of how LifeOmic is using serverless to quickly build a health cloud platform
Serverless has been gaining a lot of momentum lately. At LifeOmic, we are practical about the technologies we choose. We push boundaries, but not without some analysis to determine the benefits and trade-offs of picking shiny new tech.
My first project at LifeOmic was building a cloud-based whole genome analysis pipeline. I needed a way to kick off the pipeline after all of the data files from the DNA sequencer were uploaded to a S3 bucket. After looking at options, I determined that a Lambda function triggered by a S3 event could be used to start the pipeline. Deploying the function to AWS (Amazon Web Services) was quick, easy and cheap. We were not sequencing DNA around the clock and there was no need to have something running all of the time. It was my first taste of serverless development, and I really liked it.
Lambda: Amazon Web Services function as a service (FaaS)
S3: Amazon Web Services storage service
Months later, we started building the Precision Health Cloud platform. Several members of the development team had experience building cloud-based platforms using a microservice architecture. This time around, we knew we wanted to do something similar. However, deploying a microservice to those platforms involved a good deal of work: creating scaling policies, writing infrastructure provisioning scripts and verifying that the latest security patches were in place. It was a lot of overhead with a small development team. It was something we wanted to avoid if possible.
Containers or functions?
At first, we thought a container-based microservice approach deployed to Kubernetes or AWS Elastic Container Service would be best. We were already using Docker for the whole genome analysis pipeline with AWS Batch, and it felt like a natural fit. However, with the positive experiences we had with Lambda, we decided to give a serverless architecture a try. The event driven model was very appealing and simple to understand. We liked the cost benefits of not having services running when not being used. With a small team, we felt like we could get a prototype up and running pretty quickly to see if it could work.
Within a short amount of time we had a prototype up and running. We are using API Gateway to expose our public API’s in a secure manner. The API Gateway resources are wired up to Lambda functions that process the API requests. A microservice in this architecture can be a Lambda function and DynamoDB table. This is something that a single developer can get up and running in a very short amount of time. There’s also very little overhead in keeping a service like this updated and running. There’s no more applying security patches, or scale policy tweaking.
We are also using Lambda functions to consume events from different sources like CloudWatch, S3 and Kinesis. For example, our CloudWatch logs are sent to SumoLogic through a simple Lambda and Kinesis stream pairing. With the microservice design from our previous jobs, this type of event processing would likely be added to an existing microservice that also served another role. This was mainly due to the overhead of service ownership that I described above. One was hesitant to go through the work of deploying and having to maintain a new service. After some time, a few microservices were not that micro anymore. With the serverless approach, adding a new function for a specific task is very easy and helps to keep separation of concerns amongst the functions.
DynamoDB: Amazon Web Services NoSQL storage service
CloudWatch: Amazon Web Services monitoring service
Kinesis: Amazon Web Services data stream service
There are trade-offs
This article is painting a pretty nice picture for serverless. Just like any technology, there are some trade-offs. Cold starts are a thing, and chaos engineering overall needs to be considered when going with serverless. We have tried to apply the suggested techniques to work around cold starts, and it is an area where we continue to try to learn more and improve. With Lambda, you are also locking yourself into a specific set of runtime environments. The AWS Lambda team is improving in this area. Support for node 8.10 and Go were announced over the past few months. There are also hard and soft limits around things like maximum Lambda invocation duration. At LifeOmic, we decided the advantages of this approach far outweighed the disadvantages.
Where we plan to go
So far, we have been pleased with how well serverless has been working at LifeOmic. We typically stay in the free pricing tier for Lambda in our development environment. As a young startup, this cost savings is a huge benefit. The ease and speed of serverless development has allowed our small team to deliver a large set of features in a very short amount of time. We can worry about writing feature code that drives business value instead of deploying and mananging large infrastructure stacks. We have also been keeping an eye on new services like AWS Fargate to see if it makes sense to run some services there.
We have created some tools and libraries to make Lambda development easier. It is our hope to share some of these with the open source community in the near future. Please follow this blog for those announcements, and also check out some of the projects that we have already shared on GitHub.
If this kind of technology and company culture sounds interesting to you, please check out our jobs page. We are hiring!
