Cloud’s silver lining: how to make content management systems more effective with AWS

Offloading images from Adobe Experience Manager to Lambda functions can reap several benefits

We’re always exploring different way we can improve our quality and performance. Recently we’ve experimented with externalising image renditions generation using Amazon Web Services’ Lambda functions, and integrating them with Adobe Experience Manager (AEM), our content management system. Below is a guide of two ways you can do this and the benefits it can bring.

The AWS Lambda code used in the examples below is available on request. The main ingredients are a nodejs javascript function, which is using grahicsmagick & async libraries to generate renditions. (See the approach diagrams below for more details). AEM code can also be provided on request.

Approach 1

1. The user uploads an image to AEM, AEM triggers the DAM (Digital Asset Management) Update Asset workflow, which uploads an image to an S3 bucket using an AWS Java SDK.
2. The upload to S3 event triggers a Lambda function that generates thumbnail renditions.
3. The Lambda function generates the thumbnail renditions, stores them in another S3 bucket and writes a message to SQS (Amazon’s Simple Queue Service).
4. AEM polls the queue to receive the message. If the message says “success”, AEM deletes the message and continues to the next step. If the message doesn’t say “success”, it deletes the message and suspends the workflow.

For each of the 7 web renditions (one per device type), AEM repeats the following steps as a part of a custom workflow step:

It calls the web renditions Lambda with a different width and height.

The Lambda generates the web rendition and stores it in the other S3 bucket.

The Lambda then returns the message to AEM.

AEM checks the message to see if the rendition generation is successful. It moves on to next step or suspends the workflow, depending on whether it is a success or not.

5. AEM downloads all 11 renditions (the 7 web renditions and 4 thumbnails) to AEM and creates renditions nodes.

Pros

• Less CPU load as AEM doesn’t need to generate renditions.
• It’s scalable.
• It’s cheap compare to any other offloading services.

Cons

• We found that it took a very long time to finish the workflow and have renditions available back in AEM. It’s 10 times slower when bulk images (of medium to large size) are uploaded to AEM using the CMS’s out of the box workflow.

Reasons for slowness and potential solutions

• Reason: The current workflow has a lot of steps to generate renditions.
• Solution: Implement fanout in AWS which can generate all renditions in parallel, reducing the number of workflow steps.
• Reason: It is using SQS, which is a polling queue, and AEM has to poll to see if the thumbnail renditions have been generated.
• Solution: Remove SQS completely from the design by decoupling S3 and Lambda.
• Reason: AEM is calling the web rendition generation Lambda function 7 times sequentially to generate web renditions, hence we are not taking advantage of AWS’s parallel execution using this design.
• Solution: Fanout should reduce the number of steps and we can take advantage of AWS parallel execution capability.
• Reason: AEM downloads all 11 renditions from S3 and creates nodes in AEM DAM.
• Solution: Images from web pages can be served from S3 instead of serving from DAM. Some research would be required to support crop and blur if you wanted to do this.

Approach 2

1. The user uploads an image to AEM, AEM triggers the DAM Update Asset workflow, which uploads an image to S3 bucket using an AWS Java SDK.
2. AEM then calls the master Lambda function, which invokes 8 Lambda functions in parallel. Each web Lambda generates a web rendition of different sizes and stores it in another S3 bucket. The thumbnail Lambda generates 4 thumbnail renditions and stores them in another S3 bucket.
3. AEM gets a response message from the master Lambda, which it checks to see whether the result is a success. If it is successful, it moves on to next step. If not, it suspends the workflow.
4. AEM downloads all 11 renditions back to AEM and finishes the workflow.

Note: In the above approach, rather than having an individual Lambda function per web rendition, we would have used just one Lambda and made parallel requests for renditions with different widths & heights. The reason behind not doing this is, “At any point of time, we can have 1000 parallel requests per lambda”. To maximize number of parallel requests, multiple web lambdas were used.

Comparisons with AEM after implementing the above approach

AEM

We uploaded 78 images (of medium to large size) to DAM using AEM, which took around 2–3 minutes.

AEM-AWS

We uploaded the same 78 images using Approach 2, which took around 1–2 minutes.

How to deploy the AWS Lambda code

• Download the code from the Stash repository.
• Make a zip using the js & node_modules, rename it to match the name of the js file.
• Login to an AWS account.
• Navigate to Lambda.
• Go to the Code tab and upload the zip file.

Configuring AWS parameters in AEM can be done via OSGI configuration.