Gaining a better understanding of Python modules and packages

Akshar Raaj

Follow

Apr 1, 2020 · 4 min read

This post attempts to clear frequently occurring confusion around Python modules and packages.

You should read this post if you frequently grapple with the following errors:

• Attempted relative import in non-package
• No module named package.module
• You want to understand how relative imports work

Modules

We will create two Python modules. These modules are without a package.

A Python package doesn’t come into picture unless there is an __init__.py defined.

Let’s create a directory called anydir in your home directory.

╰─$ mkdir anydir
╰─$ cd anydir

Let’s create a Python module called hello.py with following code:

def hello():
    return 'hello'

Let’s create another Python module called say_hello.py with the following code:

from hello import hellodef say():
    return 'saying %s' % hello()

We can start a Python shell and invoke say.

In [1]: from say_hello import sayIn [3]: print say()
saying hello

Very often developers think that since hello.py and say_hello.py are in the same directory so they could use relative import instead of an absolute import.

They change from from hello import hello to from .hello import hello.

Let’s try that. Make say_hello.py look like:

from .hello import hellodef say():
    return 'saying %s' % hello()

Restart Python shell and invoke say again.

In [1]: from say_hello import say
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-c19beb0ca355> in <module>()
----> 1 from say_hello import say/Users/akshar/anydir/say_hello.py in <module>()
----> 1 from .hello import hello
      2
      3 def say():
      4     return 'saying %s' % hello()ValueError: Attempted relative import in non-package

Shell would have raised a ValueError with message Attempted relative import in non-package.

Relative imports can only be used in a package. Since there is no package here, so it failed.

Packages

Let’s create a package called mathematics inside anydir.

╰─$ pwd
/Users/akshar/anydir╰─$ mkdir mathematics╰─$ touch mathematics/__init__.py

We created an __init__.py inside mathematics, so mathematics becomes a package.

Let’s create a module called simple.py in package mathematics.

╰─$ vim mathematics/simple.py

Add the following content to this module.

def sum(a, b):
    return a + b

Let’s create another module called advanced.py in package mathematics.

╰─$ vim mathematics/advanced.py

Add the following content to this module.

from simple import sumdef add_five_to_sum(a, b):
    return 5 + sum(a, b)

Let’s start a Python shell and invoke add_five_to_sum.

In [1]: from mathematics.advanced import add_five_to_sumIn [2]: add_five_to_sum(3, 4)
Out[2]: 12

Let’s remove mathematics/__init__.py and try to invoke add_five_to_sum.

╰─$ rm mathematics/__init__.py╰─$ ipythonIn [1]: from mathematics.advanced import add_five_to_sum
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-452b1a8afbc5> in <module>()
----> 1 from mathematics.advanced import add_five_to_sumImportError: No module named mathematics.advanced

Let’s add mathematics/__init__.py back. Things should work as earlier then.

╰─$ touch mathematics/__init__.py

Let’s use relative import in mathematics/advanced.py. Modify the import statement to look like the following:

from .simple import sum

Let’s invoke add_five_to_sum.

╰─$ ipythonIn [1]: from mathematics.advanced import add_five_to_sumIn [2]: add_five_to_sum(3, 4)
Out[2]: 12

Let’s try to use add_five_to_sum from inside the package i.e from the directory mathematics.

╰─$ cd mathematics╰─$ ipythonIn [1]: from advanced import add_five_to_sum
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-300480d69d38> in <module>()
----> 1 from advanced import add_five_to_sum/Users/akshar/anydir/mathematics/advanced.py in <module>()
----> 1 from .simple import sum
      2
      3 def add_five_to_sum(a, b):
      4     return 5 + sum(a, b)ValueError: Attempted relative import in non-package

Now we are working in a module context and not in a package context. So relative imports of advanced.py raised an error.

This suggests that if you have a package then you need to work from the directory containing the package. You cannot work in the package directory itself.

Let’s come back to our base directory.

╰─$ cd ..╰─$ pwd
/Users/akshar/anydir

We can create a subpackage inside package mathematics. Let’s create a package complex inside mathematics.

╰─$ mkdir mathematics/complex╰─$ touch mathematics/complex/__init__.py

Since complex is a package, so we created an __init__.py in it.

Let’s create a module called all.py in subpackage complex.

╰─$ vim mathematics/complex/all.py

Let’s add the following content.

from mathematics.simple import sumdef add_ten_to_sum(a, b):
    return 10 + sum(a, b)

Let’s invoke add_ten_to_sum.

╰─$ ipythonIn [1]: from mathematics.complex.all import add_ten_to_sumIn [2]: add_ten_to_sum(10, 5)
Out[2]: 25

We can use relative imports in mathematics/complex/all.py.

from ..simple import sumdef add_ten_to_sum(a, b):
    return 10 + sum(a, b)

Restart shell and things should keep working as earlier.

All this setup is only going to work properly as long as you are working from anydir. We can consider anydir as our base directory.

If you try to work from directory mathematics things will break. It’s important to give a proper thought to your base directory and define packages in it and do imports accordingly.

Feel free to ask any question on modules and packages and I would be glad to answer.