Avoiding Circular Imports in Python

If you’ve written enough Python code, you’ve almost certainly found yourself staring down a stack trace that that ends with:

ImportError: cannot import name '<name>' from partially initialized module '<module>' (most likely due to a circular import)...

In my experience, running into such an error usually prompts a long and tedious inspection of the codebase. The end result is almost always some sort of refactor–ranging from small adjustments of imports all the way to large changes in a project’s directory structures.

The reason why circular imports in Python result in errors has been explained by others much more eloquently than I could manage — instead of focusing on the reason behind the problem, in this post we will provide some practical advice that might help you avoid circular imports in large Python projects (or at least make them easier to debug!).

Not all Circular Imports are Created Equal

Some circular imports in Python are “real”, and cannot be resolved without moving where your objects are defined. An example of this can be seen in this repository. If you clone the repository locally, check out the circular-imports-require-refactor branch and try to run main.py, you’ll see the import error show up:

$ python3 ./main.py                                                   
Traceback (most recent call last):
  File "/Users/andremenck/repos/personal/circular_import_examples/./main.py", line 1, in <module>
    from child import Child
  File "/Users/andremenck/repos/personal/circular_import_examples/child.py", line 4, in <module>
    from parent import Person
  File "/Users/andremenck/repos/personal/circular_import_examples/parent.py", line 4, in <module>
    from child import Child
ImportError: cannot import name 'Child' from partially initialized module 'child' (most likely due to a circular import) (/Users/andremenck/repos/personal/circular_import_examples/child.py)

So, what is going on here? Well, this case is (deliberately) pretty simple:

• The Child class in the child file depends on the Person class in the parent module
• The Parent class in the parent file depends on the Child class in the child module

In this contrived example, the solution is really quite simple: move the Person class definition into a new file, and have child and parent pull from that. The notable thing here is that in order to resolve this circular import, we actually had to move a class definition into another file (the solution can be seen in another branch in the same repository).

Importing from Parent Modules

If you asked me a couple of weeks ago, I would have said that pretty much all circular imports would require such refactors (ie: changing where objects are defined) — of course, had I been correct, this would be a much less interesting blog post. In fact, as our Python libraries at Brex grow and become more complex, we started to notice that many circular imports had a very different root cause: they were happening because we were importing objects from the wrong place, not because we were defining objects in the wrong place. Rather than me fumbling for a description, let’s consider an example. Take the following directory structure and files (this is replicated in another branch of our repo):

./
├── main.py
└── objects
    ├── __init__.py
    ├── child.py
    ├── parent.py
    └── person.py

This is exactly the same directory structure we had in the “fixed” branch for our imports. However, we have added some slight changes:

1. We added imports for all of our classes in the objects/__init__.py file
2. The imports in person.py and child.py were changed to import from objects, instead of importing from each submodule

In other words, we have:

The classes above are still defined in the same place as in the “fixed” branch, but the imports have been slightly modified (we now import from the module root, instead of importing from each specific file). So, what happens when we try to run main.py here?

$ circular_import_examples % python3 ./main.py                                                
Traceback (most recent call last):
  File "/Users/andremenck/repos/personal/circular_import_examples/./main.py", line 1, in <module>
    from objects.parent import Parent
  File "/Users/andremenck/repos/personal/circular_import_examples/objects/__init__.py", line 1, in <module>
    from .child import Child
  File "/Users/andremenck/repos/personal/circular_import_examples/objects/child.py", line 3, in <module>
    from objects import Person
ImportError: cannot import name 'Person' from partially initialized module 'objects' (most likely due to a circular import) (/Users/andremenck/repos/personal/circular_import_examples/objects/__init__.py)

Wait, what? Well, the problem here is that loading the __init__.py file requires loading the child.py file, which in turn requires loading the __init__.py file in order to import the Person class. This is pretty obvious when you think about it for a second, but the key insight here is that the “circular” import exists only because of how objects are imported! We can trivially fix this particular circular import by changing our imports to import from the module where the object is actually defined. You can see this fix in the following diff:

Does this actually happen?

Again, this example might seem a bit contrived… it would be pretty natural to ask at this point “does this actually happen in practice?” — to which I would answer: yes, most definitely!

We noticed this starting to become a pretty big problem at Brex with one of our larger Python libraries called “fractal” — this is a library built and maintained by our Data Platform team, and used mainly by our Data Science team. In order to allow our clients to easily import all the library’s objects, we adopted the convention of importing the public API into the top-level level __init__.py file of the project. In addition to being user-friendly, this practice ensures that our client’s imports won’t break if we refactor the internals of our library. So, why does this matter? Well, as more and more developers contributed to this library, we found that in many instances folks were importing from the root of the library from within the library itself. This is the exact same practice as shown in the example above, but in a real-world scenario where the less-than-ideal import strategy came about naturally, out of the ever-growing complexity of a large Python library.

The pattern we observed internally was that developers would import from the “root” module while writing their code (due to convenience — after all, everything is right there!), and that would work just fine for a while. However, when someone else came along and made subsequent code changes, they would end up hitting a circular import that could be resolved if we simply changed all the other imports to import directly from where classes are defined. This kind of issue significantly slowed down developers, as they had to unwind a very complicated dependency chain in order to fix circular imports. It would be much better for developer efficiency (and sanity) if we could simply make sure to import objects from the files they are defined in in the first place!

What can we do about this?

At this point, we have already pointed out a common cause of circular imports and how to fix them — we could very well leave it at that and simply establish the following principle:

“Always import objects from the files where they are defined.”

– Brex Engineering, 2022

While that is wise advice, at Brex we try harder than that — so we wanted to provide you with something more practical. Specifically, we implemented a function which can help you enforce the principle above in your Python code. We accomplish this by searching the code and detecting cases where classes are not being imported from the module they are defined in. In the interest of being practical, I won’t spend time walking through the implementation here (if you’d like to learn more about how it works, I recommend reading through it — it should be quite straight-forward to follow along). However, we can apply this to the example shown above. After checking out the test-imports-from-source branch of our repository, we can run:

pytest -vv ./objects/tests/test_import_from_source.py

You should see the following error in the pytest output:

E           AssertionError: Imported Child from objects, which is not the public module where this object is defined. Please import from objects.child instead.

And there we go! Switching out the problematic import to import directly from objects.child instead will make this error go away.

How can I use this?

The code linked to above provides a function that will automatically scan your project for imports that violate the “import from definition” principle. If you’d like to try this out in your own project, feel free to add that bit of code to it and see how many instances of “indirect imports” you have to deal with. In our case, we had quite a few imports that failed this test, so fixing them up involved a fairly significant refactor. The good news, however, is that the refactor was quite safe, since we really were limited to only changing the import locations.

In our case over at Brex, once we actually did this refactor, not only did we avoid circular dependencies that are due to “indirect imports”, but the “real” circular dependencies also became quite a bit easier to debug. This was due to the significantly simplified import chain, now that we were forcing ourselves to use only direct imports. As a way to visualize this simplification, I used pydeps to generate a before/after graph of how our Python files depend on each other. Here is the after graph (ie: after we simplified our imports):

Yes, this is the after graph! I attempted to generate a before graph to show that this is actually a lot simpler than our previous situation… but about 2h of waiting, pydeps crashed and refused to produce anything usable. I guess that alone should tell you something about the previous state, and how using direct imports improved our codebase’s sanity!