Local pip package and Google App Engine

Google’s App Engine sandbox is an amazing solution for creating applications in no time. You’ve got everything you need and LOTS of resources, samples and quickstarts.

However, while developing multiple services in Python, you might find out that you should create at least one common lib for your domain, utils etc.

Coming from the .NET world, where private NuGet feeds is a common thing for organizations, I would have guessed that pip had the same — especially since pip has existed for a long time.

I was wrong.

I started googling this, and there were basically just simple solutions such as:

“Create your package and host it manually on your local computer with SimpleHTTPServer”

“Use buckets”

“$ pip install .”

I get it, it’s the open source spirit — things should be open, so perhaps I should have done this using the submodule feature in Git if I really want to keep things closed to the public. However, the thing is, I don’t want to expose the app’s domain and features, yet. Also, I did this because I really wanted to learn more about pip.

So I started moving some modules to the new package I had created and read this simple guide which I would like to recommend to everyone in the need to understand pip.

Using this guide I learned to

1. Create a setup.py with a simple module.
2. Install the package locally using '$ pip install .'
3. Install the package locally as a symlink '$ pip install -e .'
4. Create an actual package using '$ python setup.py sdist'

In theory I had everything under control, I made my modules, managed to pack them within the same package named ‘myapp-shared’ and I was happy — the dist-package after running the sdist-command seemed nice.

However, I had trouble making the package work with virtualenv and the GAE runtime.

I created a source distribution and tried to install the package using pip’s find-links option pointing to the location of the dist-folder and things seemed okay in the shell, but this didn’t seemed to work properly when I updated the shared package and ran sdist again — when I looked inside lib folder I found the old one, as if it just installed it from a local cache (???).

However, after a lot of back and forth I learned that the easiest way was to ignore the sdist-command and simply run this command from the GAE project’s root source directory:

pip install ../../../myapp-shared -t lib --upgrade

So I added this line to my existing install.sh script:

#!/bin/sh
virtualenv env
source env/bin/activate
pip install -r requirements.txt -t lib
pip install ../../../myapp-shared -t lib --upgrade

So in the end, the most simple solution was the best.

Cheers.