This story is in continuation to the previous one here (Technology Transitions in Legacy Systems(Mono to MicroServices)).
Published this article earlier in linkedin.
We launched multiple micro services and at each point we had choices on the implementation( Serverless or No-Serverless!!) We decided to go no-ops way(i.e. Serverless). This approach helped to focus only on the code and to choose the right tech stack for the application being developed.
Serverless (Function as a Service)
Serverless helps us decomposing our application into small chunks or units or work.You need not to manage the infrastructure if you are using serverless environment and developers too can concentrate on delivering business value.
Let’s start with serverless payment service which gave us a roller coaster experience. We have to serve things in a time bound manner and lambda needed to interact with so many other components like Dynamodb, Mysql,S3, another helper Lambda’s, API gateway and third party apis. Lets divide the services into components so that we can understand its functioning in a better way:
- App Lambda, It serves the index page and showcase the payment options available to the user.
- Helper Lambda, It helps in supporting functions like getting user payment details & perform relevant validations
- Distributor Lambda, It talks to different Payment gateways and does payment Initialisation, execution and maintain callback urls.
Challenges & Workarounds:
- Frustrated users because of slow response & Paying more money in billings.
Many a times we were not serving the payment within 100ms but after 10s. Quickly we realised that its a typical case of lambda cold start problems. It takes time to set up an execution context and do the necessary “bootstrapping”, which adds some latency each time the Lambda function is invoked. You typically see this latency when a Lambda function is invoked for the first time or after it has been updated because AWS Lambda tries to reuse the execution context for subsequent invocations of the Lambda function.
Keep the container in warm state by calling lambda after a fixed interval of time(say 5 mis). This way lambda will keep reusing the execution.
For more detail on execution context refer AWS Docs.
- Zombie MySQL Connections and Connecting Timeouts
During our load testing we figured out that mysql connections got drain out somehow and lambda is not able to make a new connection to mysql. Later on we got to know that containers which are not used were not relieving the connection and later connection will die eventually. This resulted in Zombie Mysql connections. Hence, its not easy to scale the lambda-mysql design pattern if there are left over connections.
There are few ways which can help in reducing this problem like Caching Mysql Queries, Lower Connection Timeouts, Limit Concurrent Executions, Increase Max Connections, Keep clearing the sleeping connections using some cronjob and using serverless-mysql npm package.
- Git Integration & Serverless function Creations
AWS Lambda doesn’t provide direct integration with git. At least if you are creating functions from console there are no options provided.
We use aws console and Serverless framework to create the functions. After the introduction of layers in lambda, people are using git-lambda-layer as well. However, AWS is recommending this way to integrate git with lambda functions.
Serverless is evolving and it will take sometime to replace the old stuff. But you will get the work arounds for the current problems as well.