Setting a Private PyPI Server With NGINX

In this post, we will set up a private PyPI server using the pypiserver Docker image, which will be wrapped by NGINX for caching and performance.

Maroun Maroun

Published in

Geek Culture

4 min readMar 2, 2021

What is PyPI?

The official docs:

The Python Package Index (PyPI) is a repository of software for the Python programming language.

PyPI is the official 3rd party software repository for the Python language.

To install packages from the PyPI repository, you will need a package installer. The most common and recommended one is pip.

PyPI is public, anyone can access it and download any package listed there. However, sometimes you want to create your own private repository, mainly for security reasons.

Docker Setup

To set up a PyPI server, you will need a server — it can be your own machine, your friend’s, or a machine in the cloud (AWS, GCP, …). In this tutorial, I assume that you already have a server that will host your private repository.

Since we’ll use Docker to run the server, we need to install Docker:

Docker installation

PyPI Server Setup

Let’s create a new directory for our PyPI configurations.

# replace with your name
mkdir /home/maroun/pypi-server

Create docker-compose.yml:

Note that we are currently allowing unauthorized access by specifying the -P . -a . command.

Now, we should configure our NGINX server. From within pypi-server, create the following directories:

mkdir nginx nginx_cache
mkdir nginx/conf.d

Now, add pypi.conf the conf.d with the following content:l

We are listening on port 80 (default port), and passing all requests to “pypi-server:8080”, where our PyPI server is running.

By default, Compose sets up a single network for our application. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name. That’s why we can use the pypi-server host in the configuration above.

Finally, let’s add (the default) nginx.conf in the nginx directory, which includes our custom configuration above (line 27):

nginx.conf

Starting the Server

By now, we should be able to run the following from the pypi-server directory:

docker-compose up -d

Accessing the home route from the server:

$ curl localhost:8080
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Welcome to pypiserver!</title>
  </head>
  <body>
    <h1>
      Welcome to pypiserver!
    </h1>
    <p>
      This is a PyPI compatible package index serving 11 packages.
    </p>
    <p>
      To use this server with <code>pip</code>, run the following command:
      <pre>
        <code>pip install --index-url http://pypi-server:8080/simple/ PACKAGE [PACKAGE2...]</code>
      </pre>
    </p>
    <p>
      To use this server with <code>easy_install</code>, run the following command:
      <pre>
        <code>easy_install --index-url http://pypi-server:8080/simple/ PACKAGE [PACKAGE2...]</code>
      </pre>
    </p>
    <p>
      The complete list of all packages can be found <a href="/packages/">here</a> or via the <a href="/simple/">simple</a> index.
    </p>
    <p>
      This instance is running version 1.4.2 of the <a href="https://pypi.org/project/pypiserver/">pypiserver</a> software.
    </p>
  </body>
</html>

Downloading Packages

Using pip, you can now download packages from your own private repository!

pip install --index-url http://<IP>:8080 my_package --trusted-host <IP>

Publishing packages to our newly created server can be done using Twine. Go ahead and check its documentation for further information.

Authenticating Requests

While we are now working with our private repository, anyone who has access to our server can interact with the repository. To protect it, we can use htpasswd. Install the httpd-tools:

sudo yum install -y httpd-tools

Now, create a folder for authentication:

mkdir authentication

and create a username:

# create .htpasswd file and force SHA encryption
htpasswd -sc .htpasswd pypi-user

you will be prompted to insert a password. Once done, .htpasswd will create an entry for the new user, with a hashed password:

pypi-user:{SHA}s91+2DUySxI8K+YpcFDJPbI596c=

We should next update our Docker compose file:

Docker compose YML

Note that we changed the command to require authentication for the update, download, and list commands. You can check the manual of the pypi-server command:

pypi-server understands the following options:

  -p, --port PORT
    Listen on port PORT (default: 8080).

  -i, --interface INTERFACE
    Listen on interface INTERFACE (default: 0.0.0.0, any interface).

  -a, --authenticate (update|download|list), ...
    Comma-separated list of (case-insensitive) actions to authenticate.
    Requires to have set the password (-P option).
    To password-protect package downloads (in addition to uploads) while
    leaving listings public, use:
      -P foo/htpasswd.txt -a update,download
    To allow unauthorized access, use:
      -P . -a .
    Note that when uploads are not protected, the `register` command
    is not necessary, but `~/.pypirc` still need username and password fields,
    even if bogus.
    By default, only 'update' is password-protected.

Accessing the PyPI server will now require authentication.

Summary

Sometimes you don’t want to host your packages in a public repository. Creating a custom PyPI server is a good idea for increasing security and not exposing your packages to everyone.

Using NGINX, we introduced a caching mechanism, and we can easily configure a load-balancer when needed.