Testing serverless applications #1: Unit testing AWS Lambda functions in Java

Testing Lambda functions in cloud with love

Here is my confession: I never like writing tests. It’s boring, tedious, and depressing. I always want to ship a new shiny feature.

Disclaimer
I Love My Local Farmer is a fictional company inspired by customer interactions with AWS Solutions Architects. Any stories told in this blog are not related to a specific customer. Similarities with any real companies, people, or situations are purely coincidental. Stories in this blog represent the views of the authors and are not endorsed by AWS.

However, I always write tests for production-grade applications. I often block a pull request without adequate tests and poke my teammate to add tests for each bug fixed. Why? Because we simply cannot sustainably build new features without tests. Writing production code without accompanied tests is like building a house without solid foundation. Sooner or later, the house will fall apart.

In the previous service we built, we compromised testing in order to ship new features quicker. After a few months, we were overwhelmed with issues in production and hot fixes. We caught regression bugs very late in the cycle (usually through manual testing or from customer complaints). At one point, we had to freeze building a new feature in order to write more automated tests.

To make things worse, the code that’s written without testing in mind was difficult to write tests for later. Refactoring the code was like walking in a landmine, a simple change request makes my palm sweaty as it might introduce more regression bugs. It was terrible.

Perhaps you have experience (or are experiencing) a similar situation, that’s why you are reading this blog.

Here’s the outline of this post.

1. Properties of good tests
2. Uniqueness of testing serverless applications
3. Writing testable Lambda handler
4. Mocking database object

In this post, we will start by exploring the properties of good tests, why not all tests are created equal. And how these are different in the serverless context. These two parts provide a good foundation for you to form a testing strategy for your serverless application.

Then we move on to the main topic of writing unit tests in a Lambda function. We will present some unique challenges that don’t exist in traditional applications. Then we show how to restructure our code to be more testable.

The examples in this post are written in Java with JUnit and Mockito. But the concepts are applicable to other programming languages.

Properties of good tests

We can divide tests broadly in to 3 levels:

1. Unit tests
2. Integration tests
3. End-to-End tests

The definition of these terms are blurry and debatable. For this blog series, unit tests are limited to public method level and do not call any external components. The integration tests include external components that we cannot control (e.g. calling an external services). The end-to-end tests emulate real user interactions to the application, and use every actual external components.

The Testing Pyramid, Mike Chon 2004

Before going further, let’s take a step back to examine what makes some tests better than others. Here are 5 properties we prefer to have in our tests.

1. System verification. This is the whole reason for writing tests. We want to make sure the application is working (not just that we’re seeing the tests pass). If the tests do not cover all test cases, we can’t trust that our existing features still works as they were. If we can’t trust the tests, we have to test manually, defeating the whole purpose of writing automated tests.
2. Fast. The slower the tests, the less likely we run it before pushing the code to CI tools, the more often we block each other with failed build. While we usually think of testing as a separate activity from coding, we actually do testing all the time. (Printing an output is also a form of testing.) Fast tests not only speed up the CI steps, they also speed up the development process.
3. Reliable. Some of us may experience tests being “flaky”. Running flaky tests is like throwing a dice. They often pass, but not always. There are many possible root causes (e.g. tests rely on wait time, share their states etc.). The flakiness appears more often in the higher level tests (End-to-end tests and Integration tests). Avoiding this issue takes careful consideration and experience.
4. Not brittle. Our tests are brittle when most existing tests fail after adding a small change. We have to make changes in several places to get them to work. One possible cause is leaky abstraction. Brittle tests are not harmful but expensive to maintain.
5. Easy to locate bug. When a test fails, we should be able to know in which part of the code that bug lives. In unit testing, the ideal scenario is to have only one test fail for a bug, instead of 90% of the tests.

Notice that each level of testing (unit, integration, end-to-end) has different tendencies of the 5 properties above. End-to-end testing is a great way to guarantee that everything works (System verification). But it’s slow, prone to flakiness (if badly-written), usually brittle, and very hard to pinpoint where things went wrong.

On the other end of the spectrum, unit testing is fast, usually reliable, and makes it easy to locate bugs. If written well, it is not too brittle and easy to maintain. However, we need to make sure that we cover all test cases. And even if we do, we cannot be sure that there is no bug when this unit of code interact with other parts. Many costly production issues actually come from configuration rather than code.

I said “tendency”, as it is not always true in all cases. We can write very robust end-to-end tests as well as a very brittle unit tests. That’s why we write this series. Some tips and know-hows can drastically improve these properties with little effort.

Uniqueness of testing serverless applications

There are some unique characteristics in serverless applications that affect our testing strategy:

1. Smaller code units. The code in our Lambda functions is usually small. Overall business logic can be split into multiple Lambdas and orchestrated by other AWS services (e.g. Step Functions, EventBridge)
2. More integration points. A Lambda function usually relies a lot on external services (e.g. API Gateway, database, queues and message buses, another API, etc.). Any failure usually comes from those integration points (e.g. Missing IAM permission, incorrect resource policy, incorrect parameters, mismatch format etc.)
3. Asynchronous side-effects. Asynchronous calls are more common in serverless architecture. The supporting services are usually asynchronous (e.g. SNS, Writing to DynamoDB that uses Stream, etc. ), increasing the chance of flaky tests.
4. Longer deployment time. Deploying a Lambda function to the cloud takes more time. We can emulate the function locally, but we won’t be able to verify that it works correctly within the actual context.

Due to the first three characteristics, many serverless practitioners favor integration testing (or End-to-End testing) over unit testing¹. Still, unit tests are not out of the picture. Integration tests that actually call real services are unavoidably slower. It will take very long time to cover all test cases. In additional, simulating failure scenarios are difficult. One does not simply switch AWS services off every time the test runs.

The last characteristic, longer deployment time, makes unit tests important for a development workflow. Given that deployment time usually takes 20+ seconds to several minutes, it is crucial to get quick feedback from unit tests to verify that the few lines of code I just added didn’t break anything. So I only need to deploy when I have successfully passed all unit tests.

Here is another confession. I used to test code change via deployment without writing unit tests first. I would end up either (1) getting distracted by Reddit or Twitter while waiting in that 20+ seconds. (2) running out of patience and opening an editor in the AWS Console to edit the code directly. The results were not very productive.

Of course, we can use some tools to reduce deployment time or even emulate Lambda infrastructure. Nevertheless, we still need a robust test suite that is fast and repeatable to check that future changes do not break anything.

Writing testable Lambda handlers

We are going to use the Delivery service API from previous posts² as an example. You should be able to follow this post without reading the prior posts. We start with a naive implementation of Lambda handler and see why it cannot be easily tested. Then we will propose a refactoring that allow us to inject a mock object into our tests.

In Lambda, we usually initialize reusable resources (e.g. Database connections) up-front for performance reason³. In this code, we have a GetSlots.handleRequest() as handler. It retrieves slot data from a database based on given parameters. To interact with the database, the method uses a SlotService object, which is created in the constructor.

Lambda will call the constructor when it is first initialized. Then the execution context of this function will be “kept warm”. This allows us to reuse the database connection inside the SlotService object in subsequent requests.

When we started writing unit tests, we found that it is impossible to unit test this handler without side effects. If we call the handler directly, a new SlotService() will be called inside the constructor. And the SlotService will try to call an RDS database. That isn’t what we expect unit tests to do.

Thus, we add another constructor for testing. This one takes a SlotService as a parameter.

With this overloaded constructor, we can pass a mocked SlotService in our test code.

In each test case, we use Mockito to intercept the call and return a canned response. This allows us to test the code in this handler without any side effects (e.g an external call to an RDS database).

With this setup, we can develop the handler without having to deploy anything to the cloud. When there is a change, we can rerun the unit tests to confirm that we didn’t break any existing functionality. Once we get all the tests green, we can then deploy and run integration tests.

What did we achieve with this design?

Let’s revisit the mentioned properties one by one and see how does this design perform.

1. System verification. Injecting mocked service improve this property. We can now simulate edge cases easily. Here’s a test case when a given parameter (farm id) does not exist:

Similarly, we can write test for other edge cases like required parameters, invalid id, etc. (See the whole test class).

2. Fast. Not calling external services makes the unit tests faster to run.

3. Reliable. Not calling external services makes the unit tests reliable. It will consistently pass or fail regardless of external environment.

4. Not brittle. we test at the handler level and only need to create the response to API Gateway. The test will last as long as we don’t change the API contract or switch out API Gateway.

5. Easy to locate bug. By having each case handled independently, we make it easy to locate bugs if any of the cases fail.

Mocking database connection to test SlotService

We started with the first attempt to unit test the handler. But the handler connects to Amazon RDS and we do not want to actually call any real database operations.

Instead of putting all the code inside the handler, we extract the business logic out into the SlotService class. The test for the handler injected a mocked SlotService where we can control the behavior.

However, we don’t have any tests for SlotService yet. The handler tests ensure that we return the correct status code and results based on the API contract. But the business logic for the database is inside SlotService. Also, the class needs to get data from Amazon RDS. So we need to mock the connection to write a proper test.

We extracted the code that is related to our database to a class DbUtil . This class has a createConnectionViaIamAuth() method that returns a Connection object back to the caller. Again, we can use a mocked DbUtil to inject a fake connection object that doesn’t really connect to RDS or simulate connection failure. The SlotService also has an overloaded constructor for testing purpose.

Conclusion

Automated testing is crucial for an application to live long. Not all tests are created equal and there are some properties we prefer to have in our tests. Low-quality tests can be as good as having no tests, or even liability for future changes. For serverless applications, integration tests have more important role than it was. The amounts of each type depend on your application. The bottomline is that we want to maximize system verification while not lowering any of its quality.

We demonstrated how to write good unit tests for AWS Lambda function in Java. The key idea is having a right abstraction so that you can inject mock object and test each class in isolation.

In the next post, we will visit the idea of Ports and Adapters architecture, use it in testing Lambda functions, and check how it affects the test properties. Also, we will discuss about when to draw the line to stop writing unit tests, and move to the integration tests.

We love to hear from you

The most exciting time for writer is hearing the feedback from readers. Do you like, dislike, or having any other topic to suggest? Please do let us know in the comment or in our Twitter.

Reference

[1] Yubl’s road to Serverless https://medium.com/hackernoon/yubls-road-to-serverless-part-2-testing-and-ci-cd-72b2e583fe64
[2] The Strawberry’s Journey: From Your Local Farm to Your Table https://medium.com/i-love-my-local-farmer-engineering-blog/the-strawberrys-journey-from-your-local-farm-to-your-table-70e80b5a4f51
[3] Optimizing your Java Lambda Cold Starts and Initializations https://medium.com/i-love-my-local-farmer-engineering-blog/optimizing-your-java-lambda-cold-starts-and-initializations-5ca24de2c078