Reformat/Resize Prescriptions on the fly with AWS S3, Lambda and API Gateway

When you are working on a scale with millions of requests everyday, it becomes quite essential to optimise your architectural flow. Along with various other optimisation techniques, here is an interesting one that we practise in 1MG.

Introduction

When users place an order, they are allowed to upload prescriptions (image/pdf) on our platform which we save in our S3 bucket. Now, these prescriptions are used in various places in different formats and sizes. So, generating all these combinations without any information of how many of these will actually be needed isn’t a good practice.

To cite an example from everyday life, we generally do not book hotels or return-tickets unless our vacation is confirmed. This is done to avoid the unnecessary expenditure we may have to face in case the trip gets cancelled. Similarly, we generate prescriptions of different formats and sizes only when it is demanded by any user (100% surety).

Services Used

• S3 Bucket — a data store to save and retrieve files over internet
• Lambda Function — provides a serverless computing environment
• API Gateway — used for invoking lambda function
• Nginx — used as a reverse proxy and load balancer

Creating image on the fly

Suppose a user has uploaded a prescription X and requests for thumbnail of his/her prescription Y, which is of different size than X. So, in the current scenario X is present in S3 while Y is not.

Flow for creating image on the fly

Explanation of each step shown above:

1. User requests for prescription Y providing the resolution and other details
2. Server routes the request to the concerned S3 bucket
3. If the required resource is found in S3, it will respond with status code 200. If not found, it will respond with a status code of 301 and a redirection url. (in the above considered scenario it will respond with 301)
4. Server forwards the request to the user’s browser
5. If the response was 200 then prescription Y will appear on the screen and the flow is complete. If it is 301, the browser will redirect to the redirection url.
6. This time, the server will call API gateway
7. API gateway will then invoke Lambda and Lambda will in turn generate prescription Y of desired format and resolution from X.
8. Prescription Y then gets saved in the S3 bucket.
9. Lambda will respond with a success code
10. And then it passes the request to server
11. Server will now gather prescription Y from S3
12. And respond to the redirection request along with prescription Y

After the completion of the above process whenever Y is demanded again, it will be available in the first four steps since in step 3, S3 will respond with 200 rather than 301 because the resource is already available in the bucket. A standard implementation of lazy propagation algorithm.

Several Upsides

• Security: We are able to work without involvement of nginx and deal directly with S3 and API gateway. Although, this helps in reducing 4 extra hops, it leads to security vulnerability as we end up exposing our S3 and API gateway to the public.
• Storage Cost: Since we are generating required prescription lazily i.e. only when needed, it helps us reduce the storage cost.
• Scalable: Entire process can be done without using Lambda and API gateway by just handling these tasks along with other business logic in an EC2 server but that would become a bottleneck as processing images is quite a heavy process.
• Computing Cost: In case of EC2, scaling would be our responsibility, whereas in Lambda it is the responsibility of AWS to scale the server up/down. Irrespective of the traffic, we will be charged only for the amount of computation actually used.

Reference

• Amazon’s blog on image resize

If you liked this blog post, hit the 👏 and stay tuned for the next one!