Python Import — subpackages and circular dependencies.

If this article or this write up is any indications — it’s fair to say the Python import system is tricky. Fluent Python had a really nice walk through of module import process as well, which inspired me to do a similar exercise here.

For me, my motivation for this article was trying to skirt Lux default behavior of overriding all pandas dataframes; I wanted to explicitly control behavior by invoking LuxDataFrame directly.

I tried doing something like from lux.core.frame import LuxDataFrame ; It didn’t work.

To help me understand what’s going on — I developed a lightweight package to experiment with imports:

The folder structure looks as follows:

Directory Structure

Experiment 1: What happens if I try to directly access object in nested subpackage

>>> from ddii.subpackage1.subpackage1b import hello_nested_1b
In top level import statement for ddii
Top Level Import Statement Completed
in subpackage 1
hi from subpackage1
in nested subpackage of 1 in b

What did I learn… even though I only refer to nested subpackage — all import statements at each level run.

Experiment 2: What exactly happens if I import top level package in the subpackage:

#added import into subpackage1b - 
print("in nested subpackage of 1 in b")
import ddii
def hello_nested_1b():
    print('hello world from nested 1b')

…

>>> from ddii.subpackage1.subpackage1b import hello_nested_1b
In top level import statement for ddii
Top Level Import Statement Completed
in subpackage 1
hi from subpackage1
in nested subpackage of 1 in b

What did I learn — top level import does not ‘rerun’ it just leverage cache register itself into the subpackage namespace. This feels intuitive.

Experiment 3: What exactly happens if I add the import of subpackage to top level package.

# In Top Level Package added
from ddii.subpackage1.subpackage1b import hello_nested_1b

…

>>> from ddii.subpackage1.subpackage1b import hello_nested_1b
In top level import statement for ddii
in subpackage 1
hi from subpackage1
in nested subpackage of 1 in b
in nested subpackage of 1 in b - after top level import ddii call
Top Level Import Statement Completed

What did I learn, theimport ddii in subpackage doesn’t complete the full import; in this case, python completes the subpackage import before completing top level import process. This feels Intuitive as well.

Experiment 4: Circular reference. What happens if I add a top level function before and after I import subpackage from top level…

# Top Level Module

def function_top_level_one():
    print('I am function from top level before call to subpackage')
from ddii.subpackage1.subpackage1b import hello_nested_1b 
def function_top_level_two():
    print('I am function from top level after call to subpackage')

Now, I get the ‘circular reference’ issue — at point in time — subpackage has no awareness of second function.

>>> from ddii.subpackage1.subpackage1b import hello_nested_1b
In top level import statement for ddii
in subpackage 1
hi from subpackage1
in nested subpackage of 1 in b
in nested subpackage of 1 in b - after top level import ddii call
I am function from top level before call to subpackage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\lgarzia\Documents\GitHub\deep_dive_into_imports\src\ddii\__init__.py", line 7, in <module>
    from ddii.subpackage1.subpackage1b import hello_nested_1b #hello_world
  File "C:\Users\lgarzia\Documents\GitHub\deep_dive_into_imports\src\ddii\subpackage1\subpackage1b\__init__.py", line 6, in <module>
    ddii.function_top_level_two()
AttributeError: partially initialized module 'ddii' has no attribute 'function_top_level_two' (most likely due to a circular import). Did you mean: 'function_top_level_one'?

What did I learn, this confirms the earlier intuition that order does matter with these imports — it’s all starting to feel intuitive.

In Summary, it’s a reminder that import system is a bit confusing and it helps to just experiment a bit. This little exercise help me reorganize my mental models.

1: Can’t import a module from a subpackage without running the import __inits__ for top level packages

2. Reminder to be a bit more explicit with imports within packages to avoid any potential traps of import system. Order matters…