Efficient Python Development: Combining the Powers of Docker, Conda, and VS Code

Previous Read: The Power of Docker in Modern Development

This blog outlines a modern approach to Python development by leveraging Docker for containerization, Conda for environment management, and Visual Studio Code (VS Code) for an integrated development environment (IDE). This guide specifically focuses on using the full Anaconda distribution within Docker to create a consistent and reproducible development environment, facilitating collaboration and easing the process of package management.

To run Python scripts inside a Docker container with a Conda environment using Visual Studio Code (VS Code), you’ll need to create a Docker image that includes Conda and use that environment to run the scripts.

This process involves several steps: setting up Docker, creating a Dockerfile that includes Conda, building a Docker image based on this Dockerfile, and finally, running the Python script inside a Docker container.

Here’s how to do it:

Step 1: Prerequisites

• Install Docker: Download and install Docker Desktop from Docker’s official website for your operating system.
• Install Visual Studio Code (VS Code): Download and install VS Code from the official Visual Studio Code website.
• VS Code Extensions: Install the “Docker” and “Dev Containers” (search for its extension ID: ms-vscode-remote.remote-containers.) extensions from Microsoft, available through the VS Code Marketplace.

Step 2: Setup Your Project

Create a new project directory and include the following:

• Your Python script(s) or Jupyter notebook file(s).
• An environment.yml file that specifies your Conda environment and dependencies. Here's an example of what it might contain:

Step 3: Create the Dockerfile

In your project directory, create a Dockerfile to specify the setup of your Docker container. Since the Anaconda official Docker image may be too large or not perfectly suited to your needs, starting with a Miniconda base image and setting up your environment can be more efficient:

This Dockerfile uses the continuumio/miniconda3 image for a lightweight base, installs your dependencies from the environment.yml, and set up Jupyter Lab to run when the container starts.

Dockerfile Overview

Here’s a breakdown of the Dockerfile key components:

a. Base Image: FROM continuumio/miniconda3 starts with a Miniconda image, which includes Conda.

b. Working Directory: WORKDIR /app sets /app as the working directory in the container.

c. Environment Creation:

• COPY environment.yml . copies your environment.yml from your host to the container's current directory (which is /app due to the previous WORKDIR directive).
• RUN conda env create -f environment.yml creates the Conda environment as defined in environment.yml.

d. Copying Files: COPY . /app copies all files from your project directory (on your host) to /app in the container. This includes any Python (*.py) and Jupyter Notebook (*.ipynb) files.

e. Conda Initialization:

• RUN echo "source activate myenv" > ~/.bashrc ensures the Conda environment is activated when you open a shell.
• ENV PATH /opt/conda/envs/myenv/bin:$PATH adds the Conda environment to the PATH, making it the default environment.

f. Exposing Port 8888: EXPOSE 8888 makes port 8888 available outside the container. Jupyter Notebook uses this port by default.

g. Starting Jupyter Notebook:

• CMD ["conda", "run", "-n", "myenv", "jupyter", "notebook", "--notebook-dir=/app", "--ip='*'", "--port=8888", "--no-browser", "--allow-root"] starts a Jupyter Notebook server when the container runs, making notebooks available in /app accessible through a web browser.

Step 4: Build Your Docker Image

Open a terminal in your project directory and build your Docker image with the following command:

docker build -t my-anaconda-project .

Step 5: Run Your Docker Container

Start your Docker container by running:

docker run -p 8888:8888 my-anaconda-project

or click the run button on your docker window — (on the “Actions” Column)

Resulting window:

Run the code below to get the token — in the terminal

jupyter notebook list

If you want to interact with a running Docker container (e.g., to execute commands inside it or to manage it in some way), you first need to know the container’s ID or name. Here’s how to proceed step by step:

1. List Running Containers

First, find out the ID or name of your running container. Open a terminal and type:

docker ps

This command lists all currently running Docker containers. Look for your container based on the image name or container name if you assigned one when you started it. You’ll see a list similar to this:

CONTAINER ID   IMAGE                COMMAND                  CREATED        STATUS        PORTS                    NAMES
xxxxxxxxxxxx   my-anaconda-project  "start-notebook.sh"      10 minutes ago Up 10 minutes 0.0.0.0:8888->8888/tcp   funny_dhawan

In this list, xxxxxxxxxxxx is the container ID, and funny_dhawan is an example of a container name Docker automatically assigns if you haven't specified one.

2. Execute Commands Inside the Container

To run commands inside this container, use docker exec. If you want to open a bash shell inside the container, you can run:

docker exec -it <container_id_or_name> /bin/bash

Replace <container_id_or_name> with your container's actual ID or name from the docker ps output.

3. Dealing with Jupyter Token Issues

If your goal is to deal with Jupyter token issues:

• Inside the container’s bash shell, you can generate a new Jupyter token or list running Jupyter notebook servers to get the current token:

Generate a new token:

jupyter notebook list

• To set a password for Jupyter Notebook access (instead of using a token), while still in the container’s bash shell, start Python and use the following commands:

from notebook.auth import passwd
passwd()

Follow the prompts to set a password, then exit Python.

• To make this password effective, you'll need to update Jupyter's configuration or restart the Jupyter server with password authentication enabled. This might involve modifying your Docker setup or Jupyter starts command.

Exiting the Container

After you're done, you can exit the bash shell inside the container by typing exit.

Running your development in VS Code while using Docker

Continuing your development in VS Code while using Docker, especially for projects like Jupyter notebooks or any Python-based projects, can be significantly enhanced by using the “Remote — Containers” extension. This extension allows you to open any folder inside a container and use VS Code to develop directly inside the Docker environment. This approach offers a consistent development environment, regardless of your local machine’s setup.

Step 1: Install the Remote — Containers Extension

1. Open VS Code.
2. Go to the Extensions view by clicking on the square icon on the sidebar or pressing Ctrl+Shift+X.
3. Search for “Remote — Containers” / “Dev Container”.
4. Click on the Install button.

Step 2: Opening Your Project in a Container

After installing the Remote — Containers extension, follow these steps to open your project within a Docker container:

1. Start your Docker Container: Before you can open your project in a container, you need to ensure your Docker container is running. Use the docker run command with the necessary options to start your container if it's not already running.
2. Attach to the Running Container:

• With the Docker container running, open the Command Palette in VS Code by pressing F1 or Ctrl+Shift+P.
• Type “Remote-Containers/ Dev Container: Attach to Running Container…” and select it.
• You’ll see a list of running containers. Select the container you’re working with.

Opening the Project Folder:

• Once attached to the container, you can open the folder containing your project files directly within the container. Go to File > Open Folder, and navigate to your project folder within the container’s filesystem.
• This action will reopen VS Code scoped to the container’s filesystem, where you can work on your files directly.

Step 3: Continue Development

• Edit, Run, and Debug: With your project folder open in VS Code while attached to the container, you can edit your files as if they were on your local filesystem. You can also use VS Code’s built-in terminal to run commands inside the container, including starting your Jupyter notebook server, running Python scripts, or managing packages.
• Install Extensions Inside the Container: Some extensions may need to be installed directly within the container to work correctly. If VS Code prompts you to install any extensions inside the container, go ahead and do so.
• Port Forwarding: If your development involves running servers (like Jupyter notebooks) and you need to access them from your browser, ensure the Docker container’s ports are mapped to your host. VS Code may also offer to automatically forward ports detected in use by the container.

Step 4: Save Your Work

• Commit Changes: If you’re working with a version-controlled project, remember to commit your changes as you would normally. Even though you’re working inside a container, your git commands and extensions will function as expected.
• Stop the Container: When you’re done, you can stop the Docker container from the terminal using docker stop <container_name_or_id>. If you've mounted volumes for persistent data, your changes will be saved outside the container.

Additional Tips

• Debugging: If the problem persists, you can access the container directly to debug further. First, run the container in interactive mode with a shell:

docker run -it --entrypoint /bin/bash my-anaconda-project

• Once inside the container, try manually activating the Conda environment and starting the Jupyter Notebook server to see if any errors occur:

conda activate myenv jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-rootl

• This can provide additional insights into what might be going wrong.
• Check for Typos: Ensure there are no typos in your environment.yml, Dockerfile, or the Docker commands you're using. Typos can often lead to unexpected errors.