Functionless First… Do you need that Lambda function?
AWS Lambda is the Swiss army knife of modern cloud development. When in doubt, write a Lambda function and stick it between two services. It can do nearly anything and has been revolutionary to our industry.
Back at re:Invent 2017, Dr Werner Vogels stated that the only code we will ever write is the business logic.
Take a look here: https://www.youtube.com/watch?v=nFKVzEAm-ts&t=6708s
But we often use Lambda to do much less than the business logic, filling the gaps between services and performing low-value data transformations.
Reflecting on the statement over five years after Dr Vogels made it, we find ourselves in a much better situation to fulfil the idea of only writing the business logic, but to achieve that, it means adopting more native integrations between AWS services and altering our architectural thinking to embrace these integrations and their behaviours.
Amazon API Gateway
You can create direct integrations with numerous services with API Gateway, which provides strong validation and authentication to our API requests. We often only save the request into a table.
API Gateway also offers proxy integrations for many services:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html
Amazon EventBridge Pipes
A relatively recent addition to the toolkit, EventBridge Pipes is an excellent way of transforming data, once one of the main jobs of AWS Lambda, unmarshalling DynamoDB streams and dispatching the data somewhere.
EventBridge Pipes can be invoked from several source services:
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-source.html
And target a range of services:
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html
It is also worth mentioning that you can also integrate a Lambda function with the pipe to perform data enrichment with custom business logic:
https://docs.aws.amazon.com/eventbridge/latest/userguide/pipes-enrichment.html
AWS StepFunctions
AWS Step Function has had a massive investment in this area; with native integrations and SDK integrations, Steps can now invoke nearly any AWS service directly.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-services.html
AWS Step Functions has become a powerhouse in recent years, and the team behind the product should be incredibly proud. It stands above any other service for orchestrating complex business logic across microservices and breaking them into manageable chunks, running complex parallel workloads and many more use cases.
Case study
Recently, I worked with one of our clients to implement “click tracking.” this was an API request from a client application on every click a user made.
Initially, we had gone straight to using a Lambda function to write the data to a DynamoDB table and then stream that out to another service.
Click tracking is a great candidate for direct integration:
- It is fire and forget; we don’t require a response other than a 200.
- It has the potential for high throughput and could push service limits very high to the detriment of business logic used elsewhere in the application.
There was a much better, much more resilient solution,
By integrating API Gateway directly with Kinesis Data Stream, we could store the collected click data in S3, eliminating any potential Lambda concurrency issues arising from the high throughput of click events. Kinesis Data Firehose handles batching the events, so we reduce the total amount of records written to S3.
I have a much better solution for the problem, which removed both the cost of Lambda invocations and the risk associated with a high throughput Lambda and the risk of custom code.
Benefits
Service Quotas: reducing the number of Lambda functions running at any time will reduce the possibility of hitting a concurrency quota.
Cost: removing the Lambda function cost from your architecture will make things more cost-efficient. Cost awareness and optimisation featured highly in Werner Vogels Keynote at re:Invent 2023.
https://thefrugalarchitect.com/laws/make-cost-a-non-functional-requirement.html
Speed: Direct integrations have speed benefits over invoking a Lambda to invoke the service. Your application will be more performant.
Drawbacks
Testing: The lack of ability to unit test is a significant drawback. Unit testing should focus on testing the custom business logic we are reducing using native integrations. We also invest heavily in running integration and end-to-end testing in the cloud.
Check out https://github.com/leighton-digital/eventnet for an insight into how we test Event-Driven Applications that use Amazon EventBridge.
Perceived difficulty: This can also be a significant boundary. Using a Lambda function is easy, but some native integrations can be challenging to implement. Nobody likes making their job harder. Ask any developer who has worked with VTL if they enjoyed it.
Asychronosity: Synchronous thinking can make you shy away from direct integrations. Embracing Asynchronicity in your design will enable you and your teams to make the most of direct integrations and improve reliability.
When building an application driven by a user interface, embracing technologies like AppSync and WebSockets and building our clients to be ‘fatter’ to accommodate Async behaviour can be much more complicated than a thin client over synchronous REST services.
Having a Lambda function as the main building block for our applications is safe when adapting to change; we can alter the Lambda logic to do anything. If we need to enrich data, we can do it at the source, adding it directly to our function. With this, we must figure out where to enrich the request in the Asynchronous chain of events.
Functionless first mindset
Having converted to a serverless first mindset, I encourage people to think functionless first. Do we need that function, or can it be done with a native service integration? Do we gain anything by using a Lambda function? Can we be more confident?
Optimise your applications to use better technological solutions, reduce cost, and improve reliability. Iterate towards only having the business logic that native AWS services can’t do be the only bits running in Lambda functions.
