Managing Python Project with Conda

This post describes an easy to use but effective workflow for building a python project from empty to be deploy-ready.

Development

First, create a project folder and enter it.

Next, create an environment to manage the packages used by your project. Instead of using the mix of pip and virtualenv, we will use conda. conda combines package management and virtual environment in the same tool. I found it very intuitive to use.

# create env named my_proj
conda create -n my_proj
# activate env
source activate my_proj
# install pip
conda install pip

Now, do the real work, writing your code. You can use pip install or conda install to install your packages.

Remember to test your project before deployment.

Deployment

We will use a docker container as our deployment option so you can easily run it almost anyway and rest assure the same behavior of your code.

The process is quite straightforward. Since we already have our environment nicely sandboxed, it is easy to export it. You can always use conda env export > env.yaml to save the env configuration to a file. In the Dockerfile, simply recreate the conda env create -f env.yaml and activate it. However, if you choose the pip route to install packages, just do pip freeze > requirements.txt and pip install -r requirements.txt to recover. Deploy using conda or pip takes almost the same effort (little). So pick whichever one you want.

A sample Dockerfile using conda is shown below:

FROM continuumio/miniconda
# set up work directory.
WORKDIR /home/me/dev/
# copy necessary files.
COPY ./*.* ./
# install packages.
RUN conda env create -f env.yaml
# run script.
CMD python my_script.py

Build the image using docker build -t some_tag -f path/to/dockerfile [context directory] .

Run your docker container on the cloud platform you like, all major vendors have container service in place.