AWS Lambda Size: PIL+TF+Keras+Numpy?

Kees C. Bakker
Apr 28 · 5 min read

Setup

The images come from the DAM. A trigger will make sure the images are processed.

Lambda: the sky is the limit?

tensorflow==1.8.0
Keras==2.2.4
pillow==5.4.1
numpy==1.16.3
colorgram.py==1.2.0
webcolors==1.8.1
boto3==1.9.137

AWS Layer for TF+Keras+PIL

Side loading packages

numpy==1.16.3
colorgram.py==1.2.0
webcolors==1.8.1

Package ’em up

#!/bin/bash
name=$(basename -s .git `git config --get remote.origin.url`)

if [ -d "deploy" ]; then rm -Rf deploy; fi
mkdir deploy

pip install -r requirements-lambda.txt -t deploy/requirements-lambda/

cd deploy/requirements-lambda
rm -r PIL
rm -r Pillow*

zip -9 -r ../$name-requirements.zip .

cd ..
rm -r requirements-lambda

Un(z/sh)ip it

import os
import sys
import zipfile

pkgdir = '/tmp/requirements'
zip_requirements = 'lambda-requirements.zip'

if os.environ.get("AWS_EXECUTION_ENV") is not None:
if not os.path.exists(pkgdir):

root = os.environ.get('LAMBDA_TASK_ROOT', os.getcwd())
zip_requirements = os.path.join(root, zip_requirements)
zipfile.ZipFile(zip_requirements, 'r').extractall(pkgdir)

sys.path.append(pkgdir)

Too big to ship with the package?

import boto3
import os
import sys
import zipfile

REQUIREMENTS_BUCKET_NAME = ''
REQUIREMENTS_KEY = ''

pkgdir = '/tmp/requirements'
zip_requirements = '/tmp/lambda-requirements.zip'

sys.path.append(pkgdir)

if os.environ.get("AWS_EXECUTION_ENV") is not None:
if not os.path.exists(pkgdir):

s3 = boto3.resource('s3')
bucket = s3.Bucket(REQUIREMENTS_BUCKET_NAME)
bucket.download_file(REQUIREMENTS_KEY, zip_requirements)

zipfile.ZipFile(zip_requirements, 'r').extractall(pkgdir)
os.remove('zip_requirements')

sys.path.append(pkgdir)

Lazy-loading the models from S3

import boto3
import keras
import os

MODEL_BUCKET = ''
cache = {}

def get_model(key):

if key in cache:
return cache[key]

local_path = os.path.join('/tmp', key)

# download from S3
if not os.path.isfile(local_path):
bucket=s3.Bucket(MODEL_BUCKET)
bucket.download_file(key, local_path)

cache[key] = keras.models.load_model(local_path)
return cache[key]

So…

AWS Lambda Size Layout

wehkamp-techblog

We'll try to keep up and post on the stuff we're doing and discovering. Interesting in working @wehkamp? Check out https://werkenbij.wehkamp.nl/

Kees C. Bakker

Written by

I work as a Lead Developer for one of the biggest web-shops in the Netherlands: wehkamp. I ❤️ C# and I like to solve nifty problems.

wehkamp-techblog

We'll try to keep up and post on the stuff we're doing and discovering. Interesting in working @wehkamp? Check out https://werkenbij.wehkamp.nl/