How to add an SQS queue to your Amplify CLI bootstrapped project
Before jumping in, you need to know that everything about AWS Amplify is constantly changing and this problem might not even exist in the future or my solution might be outdated. I only hope it helps someone until they add SQS queues to their supported resources.
Amplify CLI is the new shiny toolchain by AWS which has lots of benefits if you’re developing on AWS cloud. However, it still doesn’t support all resources on AWS. One of the main resources, which is kind of vital to Serverless projects these days is SQS. Here is a step-by-step guide of how to add an SQS queue to your Amplify CLI bootstrapped project and use it as a trigger for one of your lambda functions:
Step 1: Adding the queue to your Amplify project.
You need to follow this guide on AWS Amplify document to create your queue. Remember to run “amplify env checkout YOUR_ENVIRONMENT” wherever it’s mentioned. It’s really important in the process to let Amplify CLI know that you have some manual changes.
When you want to choose a name for your queue, be aware that Amplify uses your category name as a prefix for the queue, so you won’t need to include “queue” in the name.
Here’s a sample of your queue “template.json”:
Make sure your template & parameter JSON file names are unique among all resources you have in Amplify. It’s not mentioned anywhere in their document, but it’ll give you a serious headache if you miss it and want to have more than one queue in your project. For example, this could be a good pattern: “YOUR_QUEUE_NAME-template.json” & “YOUR_QUEUE_NAME-parameter.json”. The reason is Amplify is compressing the whole stack resources in a zip file and put it on S3. The file name is not based on the resource folder name, but based on the template filename!
Notice the “Outputs” section. We’re going to use these outputs in our next steps as input parameters, so make sure you’ll export them here.
To make sure if the output parameters are being generated correctly, you can simply log in to your AWS Console and find the queue stack among all your CloudFormation stacks and check the output tab. There, you should be able to see both Name & Arn of your queue as output parameters.
This step finishes with deploying your changes to AWS, whether by running a direct “amplify push — yes” or by merging changes into your source code and triggering Amplify Console App “build” script. (or any other CI/CD you might have used).
Step 2: Adding consumer lambda function(s) to your Amplify project.
When we’re talking about an SQS queue, normally it means that we need a lambda trigger (except FIFO queues) and also we need its URL to send a message via AWS SDK. Creating a lambda function is currently supported by Amplify CLI, so you just need to run “amplify add function” command and create your lambda function to be used as the lambda-to-be-triggered by the queue. It’s up to you if you want to add “Expressjs” in your function or just create a simple “hello world!” function working as your queue’s trigger. All we need in this step is your lambda function template, generated by Amplify CLI.
Step 3: Modifying backend-config.json to add the queue dependency
Remember the “Outputs” part? Good! Because if like me you need to address the deployed queue’s ARN or URL, you’ll need to connect the queue’s output parameters to your resource’s input ones. On this article, I’m assuming that like me, you want to send a message to your queue from one of your other lambda functions. All you need to do is let amplify know that we expect two input parameters that depend on our queue’s Outputs:
At this point, because you made a manual change in your Amplify CLI auto-generated config, you need to run a “amplify env checkout YOUR_ENVIRONMENT” command to make sure it knows about the new dependency. However, that’s not enough for Amplify to update the Lambda Function stack and add the new dependency. That’s why you don’t need to run “amplify push” command at this stage, we need another change on the Lambda template to make sure Amplify updates the stack along with the dependency.
Step 4: Modifying your consumer Lambda Function’s template and add new input parameters, policies , and variables
If you’re following my example, it means that you need the queue’s URL to send a message to it. To achieve that, open your lambda function’s JSON template in your editor of choice. You’ll find it inside the root folder of your lambda function with a name like this: “YOUR_LAMBDA_NAME-cloudformation-template.json”. You need to add your queue’s Name & ARN that we managed to expose on previous steps as your lambda parameters:
Notice that I added only 2 new actions to the lambda’s execution role policy, the ones necessary to consume an SQS queue.
I know you’re tempted to change the “Resource” property inside the new policy to {“Ref”: “queueYOUREXACTQUEUENAMEArn”}, but don’t! Because at this point, Amplify might not be able to apply both the dependency and policy changes at the same time. Once you run “amplify env checkout YOUR_ENVIRONMENT” & “amplify push — yes” command and it is successful, feel free to come back and change it.
This step ends with “amplify env checkout YOUR_ENVIRONMENT” and then “amplify push — yes”.
To make sure if input parameters are being generated correctly, you can simply log in to your AWS Console and find the queue stack among all your CloudFormation stacks and check the parameters tab. There, you should be able to see both Name & Arn of your queue as input parameters. You can also open your Lambda Function and make sure that your ENV variable is correct. It should be filled with the queue URL which helps us send a message to it.
EDIT: Sometimes, you’ll still get an error like “ARN must be in the correct format…” Whenever you see this, it means that the dependency part in backend config hasn’t been applied yet. Try amplify update function and select the consumer lambda and then answer y to the question that if you want to access any other resources in the project, then select queue from the list and it’ll try to show you the list of queues, but it can’t and it will throw an exception. However, it’ll solve the problem. Now your `amplify push — yes` will work.(Weired)
Step 5: Modifying your trigger Lambda Function’s template and add new input parameters, policies , and variables
This step looks like the previous step except for the part we add a “LambdaFunctionEventSourceMapping” section to our Lambda’s template and the Lambda Execution Role Policy:
If you take a closer look at the template above, you’ll notice that I’ve added a new output parameter and also a dependency to it in the EventSourceMapping object. Without it, you’ll get an error while Amplify is trying to add the event source mapping, why? because the lambda which is supposed to be trigger by our queue, needs to have certain policies like “sqs:ReceiveMessage” and if you don’t include the dependency, it’ll try to add the EventSourceMapping at the same time or before the policy and you know the rest.
This final step ends with “amplify env checkout YOUR_ENVIRONMENT” and then “amplify push — yes”. (Probably just a push would suffice though)
Don’t hesitate to leave a comment if you have any questions or you want to offer an improvement.
Happy coding!
