Photo by Adrien Delforge on Unsplash

How to build a serverless Laravel queue using AWS Lambda

Prerequisites

• Laravel 6+
• bref.sh
• serverless

Quick Steps

1. Install bref
2. Deploy to AWS Lambda using serverless
3. Configure Laravel queue to use SQS
4. Bonus!

Step 1: Install bref

Installing bref into existing Laravel application is quite simple, since it provides Laravel’s package already:

$ composer require bref/bref bref/laravel-bridge

Then create aserverless.yml configuration file:

$ php artisan vendor:publish --tag=serverless-config

It will bring the default configuration that require to modify a bit to handle Laravel queue jobs as the following example:

service: your-laravel-app-name
provider:
    name: aws
    # The AWS region in which to deploy (us-east-1 is the default)
    region: us-east-1
    runtime: provided
    # The stage of the application, e.g. dev, prod, staging
    stage: prod
    environment:
        APP_ENV: production
        SQS_QUEUE:
            Ref: YourQueueName
    iamRoleStatements:
        # Allows our code to interact with SQS
        -   Effect: Allow
            Action: [sqs:SendMessage, sqs:DeleteMessage]
            Resource:
                Fn::GetAtt: [ YourQueueName, Arn ]
functions:
    worker:
        handler: worker.php
        timeout: 900 # in seconds
        layers:
            - ${bref:layer.php-74}
        events:
            -   sqs:
                    arn:
                        Fn::GetAtt: [ YourQueueName, Arn ]
                    # Only 1 item at a time to simplify error handling
                    batchSize: 1
package:
    exclude:
        - .env
        - node_modules/**
        - public/storage
        - resources/assets/**
        - storage/**
        - tests/**
plugins:
    - ./vendor/bref/bref
resources:
    Resources:
        # The queue
        YourQueueName:
            Type: AWS::SQS::Queue
            Properties:
                VisibilityTimeout: 900
                RedrivePolicy:
                    maxReceiveCount: 3 # jobs will be retried up to 3 times
                    # Failed jobs (after the retries) will be moved to the other queue for storage
                    deadLetterTargetArn:
                        Fn::GetAtt: [ YourQueueFailedName, Arn ]
# Failed jobs will go into that SQS queue to be stored, until a developer looks at these errors
        YourQueueFailedName:
            Type: AWS::SQS::Queue
            Properties:
                MessageRetentionPeriod: 1209600 # maximum retention: 14 days

On the part of configuration code above, you will notice worker.php which is used to handle SQS events in AWS Lambda, then just create it on your Laravel project folder:

<?php declare(strict_types=1);
use Bref\LaravelBridge\Queue\LaravelSqsHandler;
use Illuminate\Foundation\Application;
require __DIR__ . '/vendor/autoload.php';
/** @var Application $app */
$app = require __DIR__ . '/bootstrap/app.php';
$kernel = $app->make(Illuminate\Contracts\Console\Kernel::class);
$kernel->bootstrap();
return $app->makeWith(LaravelSqsHandler::class, [
    'connection' => 'sqs', // this is the Laravel Queue connection
    'queue' => getenv('SQS_QUEUE'),
]);

Step 2: Deploy to AWS Lambda using serverless

The magic will be done by serverless in just few commands and it will setup all for you. Install it by using npm or yarn:

$ npm install -g serverless
# or
$ yarn global add serverless

Once installed if you haven’t setup aws configure command before, you need to create AWS access keys and configure it by the following a command:

$ serverless config credentials --provider aws --key <key> --secret <secret>

In somehow I faced an error of “AWS provider credentials not found …” when run it on my root of Laravel’s project folder, and get fixed by running on the outside of the folder, very strange!

Deploy the Laravel application on your AWS account by running:

$ serverless deploy

If you are lucky then you won’t see any error and shows the lambda’s function name instead.

Step 3: Configure Laravel queue to use SQS

If the serverless’s deployment is going well, it will generate 2 SQS queues on your AWS account that you need to take a look for the main queue’s name to put it on .env as SQS_QUEUE, thus you need to update .env file accordingly and also switch the use SQS connection:

# .env
QUEUE_CONNECTION=sqs
SQS_PREFIX=https://sqs.us-east-1.amazonaws.com/<your-account-id>
SQS_QUEUE=<generated-queue-name-from-aws>
AWS_ACCESS_KEY_ID=<key>
AWS_SECRET_ACCESS_KEY=<secret>
AWS_DEFAULT_REGION=us-east-1

Add a token config to config/queue.php :

'sqs' => [
    'driver' => 'sqs',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'token' => env('AWS_SESSION_TOKEN'),
    ...

Finally, you can dispatch the job as general Laravel job, for example:

ProcessPodcast::dispatch($podcastId);

Step 4: Bonus!

In real world implementation, we had some issues! Here are some tips and tricks to handle them:

• Debugging is important!
  bref is really awesome, it ships with nice dashboard to see any things from the AWS Lambda’s logs locally by running:

$ vendor/bin/bref dashboard
# or with console view
$ serverless logs -f <generated-lambda-function-name> --tail

• My lambda has no internet access!
  Don’t worry it is not end of the world :) I was surprise also on the first time. Basically, you need to create a NAT Gateway in the VPC that Lambda uses. This tutorial shows how to do.
• Connect to AWS RDS
  In my case I use RDS for the database, all you need is by using same Security Group between RDS instance and Lambda function. See more details from bref’s docs.
• Don’t expose secret variables!
  You might store API tokens, password etc at serverless.yml file which is required to be committed into your git repo. A best approach is by using SSM parameter store, follow from bref’s official docs for more details.