Different ways of specifying shared library paths for Python and system level on Ubuntu

This article will explain the exact difference between /usr/bin/python and /usr/local/bin/python, the shared library paths these two kinds of python looking for, dist-packages and site-packages, methods of specifying shared library paths at the system level.

Photo by Johannes Plenio on Unsplash

The difference between /usr/bin/python and /usr/local/bin/python

/usr/bin: contains executable programs that are part of the operating system and installed by its package manager.

/usr/local/bin: default location for executable programs not part of the operating system and installed there by the local administrator, usually after building them from source with the sequence configure;make;make install. The goal is not to break the system by overwriting a functional program by a dysfunctional or one with a different behavior.

When the same program exists in both directories, you can select which one will be called by default by rearranging the order of thedirectories in your PATH.

Thus, the /usr/bin/python is usually the system python managed by the system package manager like apt, while the /usr/local/bin/python or some executable like /usr/local/bin/vtkpython is usually manually compiled from source.

The difference between the shared library paths different python looking for

Python’s shared library paths are specified in sys.path.

sys.path contains a list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory in inserted before the entries inserted as a result of PYTHONPATH.

In a word, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module, which is automatically imported when you start Python.

To check it, just run

import sys
print ('\n'.join(sys.path))

And here is the running results of this code for different python versions on my system, just an example:

python:

/home/lyl
/home/lyl/DoorGym/DoorGym-Unity/python_interface
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/home/lyl/.local/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages

python3:

/home/lyl
/home/lyl/DoorGym/DoorGym-Unity/python_interface
/usr/lib/python36.zip
/usr/lib/python3.6
/usr/lib/python3.6/lib-dynload
/usr/local/lib/python3.6/dist-packages
/usr/lib/python3/dist-packages

vtkpython:

/usr/local/lib/python3.6/site-packages
/home/lyl
/home/lyl/DoorGym/DoorGym-Unity/python_interface
/usr/lib/python36.zip
/usr/lib/python3.6
/usr/lib/python3.6/lib-dynload
/usr/local/lib/python3.6/dist-packages
/usr/lib/python3/dist-packages

We can see that the system python only looks for its shared libraries in the ~ directory, some specified in the PYTHONPATH and some directories in /usr/lib and /usr/local/lib (mainly /usr/lib), and only looks for the dist-packages instead of site-packages.

In contrast, the vtkpython in /usr/local/bin, on the basis of fully including the above paths, also covers more content, which is /usr/local/lib/python3.6/site-packages in my case.

Please note that it contains site-packages in addition to dist-packages.

So there may be some packages you installed locally can only be found by the python in /usr/local/bin.

But there are also some ways to solve this problem

For example,

import sys
sys.path.append ('/usr/local/lib/python3.6/site-packages')

or just edit ~/.bashrc and add in the following lines:

export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.6/site-packages

The difference between dist-packages and site-packages

dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location:

/usr/lib/python3.6/dist-packages

Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here:

/usr/local/lib/python3.6/dist-packages

Furthermore, Third party Python software installed from Debian packages goes into dist-packages, not site-packages. While if you manually compile and install Python interpreter from source, like vtkpython, it uses the site-packages.

This is to reduce conflict between the system Python, and any from-source Python build you might install manually, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.

For example, if the pip you are using corresponds to the system python, then its own packages would be located in /usr/lib/python3.6/dist-packages, but the packages you installed by it is in /usr/local/lib/python3.6/dist-packages.

If the pip you are using corresponds to your manually compiled python from source, then its own packages and the packages installed by it would both be located in /usr/local/lib/python3.6/site-packages.

The differences between different methods of specifying shared library paths at the system level

There are two ways for system to look for the shared library, especially for C++ codes in my case.

The first one is by editing files in /etc/ld.so.conf.d.

The second one is by using the LD_LIBRARY_PATH environment variable.

Under Linux, the search and loading of shared libraries is implemented by /lib/ld.so. ld.so looks for shared libraries used by the application in the standard path (/lib, /usr/lib).

If the shared library that needs to be used is in a non-standard way, the general Linux method is to add non-standard paths to /etc/ld.so.cache. When ld.so loads the shared library, it will look it up from ld.so.cache.

To do this, you can add these directories to /etc/ld.so.conf, or create a .conf file in /etc/ld.so.conf.d, and add these directories into it. For example, create a file /etc/ld.so.conf.d/usr-local.conf, put /usr/local/lib in it and then ran sudo ldconfig. To check it, just run

$ cat /etc/ld.so.conf.d/*.

ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib (on some 64-bit architectures such as x86–64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries).

The cache is used by the run-time linker, ld.so or ld-linux.so. ldconfig checks the header and filenames of the libraries it encounters when determining which versions should have their links updated.

To use the LD_LIBRARY_PATH variable is another way to specify a non-standard path. When ld.so loads the shared library, it will also look up the path set by this variable. However, many voices advocate avoiding the use of the LD_LIBRARY_PATH variable, especially as a global variable.

Here are a few things to note:

1. Add things to /lib and /usr/lib, you don’t need to modify /etc/ld.so.conf, but you need to adjust ldconfig after finishing, otherwise the library will not be found.
2. When you want to add something to some specific location other than the above two directories, you must modify /etc/ld.so.conf, and then call ldconfig, otherwise you will not find it. For example, if you install a mysql to /usr/local/mysql, MySQL has a lot of libraries under /usr/local/mysql/lib. At this time, you need to add a line /usr/local/mysql/lib under /etc/ld.so.conf. After saving, ldconfig can be used for the new library. Found when the program is running.
3. All these things done by ldconfig are related to running the program, not at all at compiling. When compiling, you should add -L.

References

How does python find packages?

28.1. sys - System-specific parameters and functions - Python 2.7.18 documentation

28.14. site - Site-specific configuration hook - Python 2.7.18 documentation