Caching S3 metadata requests in Laravel

Laravel (a PHP framework) is great and it supports Amazon S3 out of the box, eg. you can do stuff like:

 Storage::disk(‘s3’)->files(‘folder’); 

To see a list of files in a remote folder in your S3 bucket. Then, there are size() and lastModified() methods to see file timestamps and sizes. But consider this: imagine you have a folder with many many files and you want to filter them by modification time, you only want to see the new files.

One issue with Laravel’s default S3 implementation is that even though files() method receives file metadata (size, modification time, etc) along with the names, Laravel completely discards those. When you call lastModified() or size(), Laravel is going to make new remote requests to S3. If you have hundreds of files — this is going to take ages! Oh no!

In other words, this code:

$files = Storage::disk('s3')->files('folder');
foreach ($files as $file) {
echo Storage::disk('s3')->lastModified($file);
}

Will make more than 100 requests if the folder ‘folder’ has more than 100 files.

Luckily, League’s Flysystem already has a caching decorator for file system drivers, it’s just not included in Laravel. Let’s install it:

composer require league/flysystem-cached-adapter

Now let’s create a new service provider:

<?php

namespace
App\Providers;

use Illuminate\Support\ServiceProvider;
use League\Flysystem\Cached\Storage\Memory;
use Storage;
use League\Flysystem\Filesystem;
use League\Flysystem\Cached\CachedAdapter;

class CachedFileSystemProvider extends ServiceProvider
{
/**
* Bootstrap the application services.
*
*
@return void
*/
public function boot()
{
Storage::extend('s3-cached', function ($app, $config) {
$adapter = $app['filesystem']->createS3Driver($config);
$store = new Memory();

return new Filesystem(new CachedAdapter($adapter->getDriver()->getAdapter(), $store));
});
}
}

We created a new driver called ‘s3-cached’, now head to config/filesystems.php and switch your S3 disk from ‘s3’ to ‘s3-cached’ and include this new service provider in config/app.php.

In this example we are using a memory cache so file metadata will only be remembered for one single request, but Flysystem has extra drivers like Memcached, if you require.

Running the same code from above will now only make a few requests.

Enjoy your speed boost!