While building an integration platform with AWS serverless components, the issue with logging, tracing was something important. Our platform receives API requests through API gateway and lambda invokes a number of 3rd party APIs for each request. Logging the input and output of the 3rd party request is a requirement from a support team perspective to see if there are any issues later.
So, we have around 5 million API calls received in a day and for each, we are invoking around 5 third party APIs (mostly in parallel), getting the response, process it and respond back to the consumer who invoked our API. Simple Arch diagram is given below.
Our Requirement : Store all request and response payloads sent/received to/from external systems for later reference by support team in case of any issues.
Implementation : If you have a very generic storage requirement in AWS serverless, whats the first service comes to mind ? Yes! its AWS S3. It is very cheap, it has different storage tiers which looks like a very good fit. And we proceeded with that approach.
Things were looking very good when we started, with a very less number of requests per day. Kind of 10k — 20k.. Then AWS S3 cost started shooting up. then we started digging deep into different components of AWS S3 cost.
- Amazon Simple Storage Service USE2-Requests-Tier1 — $0.005 per 1,000 PUT, COPY, POST, or LIST requests
- Amazon Simple Storage Service USE2-Requests-Tier2 — $0.004 per 10,000 GET and all other requests
- Amazon Simple Storage Service USE2-Select-Returned-Bytes — $0.0007 per GB — for bytes returned by S3 Select in Standard
- Amazon Simple Storage Service USE2-Select-Scanned-Bytes — Amazon Simple Storage Service USE2-Select-Scanned-Bytes
- Amazon Simple Storage Service USE2-TimedStorage-ByteHrs — Amazon Simple Storage Service USE2-TimedStorage-ByteHrs
As you can see, the first item — “Amazon Simple Storage Service USE2-Requests-Tier1” costed us 85% of the cost! this points to the write requests to S3.
Conclusion:
We discussed how our team didn't account for write costs to S3 in the start of the project. We all think about storage in case of S3m not the write costs. We are working on fine tuning our code to have lesser number of writes to S3.
If you have any suggestions, please comment.
Thank You!
