How to get fbprophet working on AWS Lambda
I assume you’re reading this post because you’re looking for ways to use the awesome fbprophet (Facebook open source forecasting) library on AWS Lambda and you’re already familiar with the various issues around getting it done. I will be using a python 3.6 example, but the approach is applicable to other runtimes as well as other large ML libraries. Without further ado:
- Install Docker locally. An alternative is to run a EC2 instance and copy files between the instance and local machine, which feels more complicated and less reliable to me.
- Run a shell on a relevant Lambda image (take a look at lambci) from the working directory with your code, that is supposed to run on Lambda (which has the lambda handler):
docker run --rm -it -v “$PWD”:/var/task lambci/lambda:build-python3.6 bash
For an explanation of the commands used here go to Docker docs or here.
3. Create a virtual environment and install relevant packages:
python3 -m venv venv
pip install --upgrade pip
pip install fbprophet --no-cache
4. You can check if your code is working:
5. Reduce the venv size to be below the 250Mb limit:
pip uninstall -y matplolib
find "$VIRTUAL_ENV/lib/python3.6/site-packages" -name "test" | xargs rm -rf
find "$VIRTUAL_ENV/lib/python3.6/site-packages" -name "tests" | xargs rm -rf
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/lib/python3.6/site-packages/pystan/stan/lib/stan_math/lib"
echo "venv size $(du -sh $VIRTUAL_ENV | cut -f1)"
What happens here is that you’re getting rid of matplotlib (if you don’t need plotting), tests, and the pystan source codes (the heaviest part of the package). Last command displays the size of the venv, which should be below 250Mb.
6. Make sure your code is still working after the cleanup:
7. Create a .zip to upload to Lambda:
zip -r -9 -q /var/task/name_of_your_zip.zip *
zip -9r name_of_your_zip.zip name_of_your_code_folder/
zip -9 name_of_your_zip.zip lambda_handler.py
8. You now have a .zip in your working directory that you can use for Lambda. You can stop/quit Docker now — the zip will still be in your folder.
Great post on preparing ML libraries for Lambda: https://medium.com/@maebert/machine-learning-on-aws-lambda-5dc57127aee1
The discussion on preparing fbprophet for Lambda: https://gist.github.com/CarstVaartjes/77dbe8249d171e592bd17847f7009272
Reducing the size of scipy: https://gist.github.com/wrwrwr/4a38a17113e3845f3385e9388effc064