How to build a serverless Laravel queue using AWS Lambda
Prerequisites
- Laravel 6+
- bref.sh
- serverless
Quick Steps
- Install bref
- Deploy to AWS Lambda using serverless
- Configure Laravel queue to use SQS
- Bonus!
Step 1: Install bref
Installing bref into existing Laravel application is quite simple, since it provides Laravel’s package already:
$ composer require bref/bref bref/laravel-bridge
Then create aserverless.yml
configuration file:
$ php artisan vendor:publish --tag=serverless-config
It will bring the default configuration that require to modify a bit to handle Laravel queue jobs as the following example:
service: your-laravel-app-nameprovider:
name: aws
# The AWS region in which to deploy (us-east-1 is the default)
region: us-east-1
runtime: provided
# The stage of the application, e.g. dev, prod, staging
stage: prod
environment:
APP_ENV: production
SQS_QUEUE:
Ref: YourQueueName
iamRoleStatements:
# Allows our code to interact with SQS
- Effect: Allow
Action: [sqs:SendMessage, sqs:DeleteMessage]
Resource:
Fn::GetAtt: [ YourQueueName, Arn ]functions:
worker:
handler: worker.php
timeout: 900 # in seconds
layers:
- ${bref:layer.php-74}
events:
- sqs:
arn:
Fn::GetAtt: [ YourQueueName, Arn ]
# Only 1 item at a time to simplify error handling
batchSize: 1package:
exclude:
- .env
- node_modules/**
- public/storage
- resources/assets/**
- storage/**
- tests/**plugins:
- ./vendor/bref/brefresources:
Resources:
# The queue
YourQueueName:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 900
RedrivePolicy:
maxReceiveCount: 3 # jobs will be retried up to 3 times
# Failed jobs (after the retries) will be moved to the other queue for storage
deadLetterTargetArn:
Fn::GetAtt: [ YourQueueFailedName, Arn ]# Failed jobs will go into that SQS queue to be stored, until a developer looks at these errors
YourQueueFailedName:
Type: AWS::SQS::Queue
Properties:
MessageRetentionPeriod: 1209600 # maximum retention: 14 days
On the part of configuration code above, you will notice worker.php
which is used to handle SQS events in AWS Lambda, then just create it on your Laravel project folder:
<?php declare(strict_types=1);use Bref\LaravelBridge\Queue\LaravelSqsHandler;
use Illuminate\Foundation\Application;require __DIR__ . '/vendor/autoload.php';
/** @var Application $app */
$app = require __DIR__ . '/bootstrap/app.php';$kernel = $app->make(Illuminate\Contracts\Console\Kernel::class);
$kernel->bootstrap();return $app->makeWith(LaravelSqsHandler::class, [
'connection' => 'sqs', // this is the Laravel Queue connection
'queue' => getenv('SQS_QUEUE'),
]);
Step 2: Deploy to AWS Lambda using serverless
The magic will be done by serverless in just few commands and it will setup all for you. Install it by using npm or yarn:
$ npm install -g serverless
# or
$ yarn global add serverless
Once installed if you haven’t setup aws configure
command before, you need to create AWS access keys and configure it by the following a command:
$ serverless config credentials --provider aws --key <key> --secret <secret>
In somehow I faced an error of “AWS provider credentials not found …” when run it on my root of Laravel’s project folder, and get fixed by running on the outside of the folder, very strange!
Deploy the Laravel application on your AWS account by running:
$ serverless deploy
If you are lucky then you won’t see any error and shows the lambda’s function name instead.
Step 3: Configure Laravel queue to use SQS
If the serverless’s deployment is going well, it will generate 2 SQS queues on your AWS account that you need to take a look for the main queue’s name to put it on .env
as SQS_QUEUE
, thus you need to update .env
file accordingly and also switch the use SQS connection:
# .env
QUEUE_CONNECTION=sqs
SQS_PREFIX=https://sqs.us-east-1.amazonaws.com/<your-account-id>
SQS_QUEUE=<generated-queue-name-from-aws>
AWS_ACCESS_KEY_ID=<key>
AWS_SECRET_ACCESS_KEY=<secret>
AWS_DEFAULT_REGION=us-east-1
Add a token
config to config/queue.php
:
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'token' => env('AWS_SESSION_TOKEN'),
...
Finally, you can dispatch the job as general Laravel job, for example:
ProcessPodcast::dispatch($podcastId);
Step 4: Bonus!
In real world implementation, we had some issues! Here are some tips and tricks to handle them:
- Debugging is important!
bref is really awesome, it ships with nice dashboard to see any things from the AWS Lambda’s logs locally by running:
$ vendor/bin/bref dashboard
# or with console view
$ serverless logs -f <generated-lambda-function-name> --tail
- My lambda has no internet access!
Don’t worry it is not end of the world :) I was surprise also on the first time. Basically, you need to create a NAT Gateway in the VPC that Lambda uses. This tutorial shows how to do. - Connect to AWS RDS
In my case I use RDS for the database, all you need is by using same Security Group between RDS instance and Lambda function. See more details from bref’s docs. - Don’t expose secret variables!
You might store API tokens, password etc at serverless.yml file which is required to be committed into your git repo. A best approach is by using SSM parameter store, follow from bref’s official docs for more details.