How to reset your Django migrations?

Within three years of development in our company our Django project accumulated 934 migrations. Most of them were just adding/ removing a few fields here and there, but some also had a lot of custom migration code in them. Why we had to adjust the database schema that often over the span of just three years, you may ask our business team.

Why less migrations?

Generally Django handles a lot of migrations extremely well and is actually built for this.

You are encouraged to make migrations freely and not worry about how many you have; the migration code is optimized to deal with hundreds at a time without much slowdown. — Django documentation

That being said, every time we ran our automated tests, the testing database needed to apply all 934 migrations which slowed them down a lot. How much you may ask? On my machine (MacBook Pro 2021, M1 Max, 32GB RAM, Python 3.11, Django 4.1.10) starting the tests took 49.79s. In our CI/CD pipeline this step took over 5 minutes. 5 MINUTES! Just for preparing a testing database. However we are not the first ones to think of optimizing this.

Why not use squashmigrations?

Django has a built-in command called squashmigrations. It can be used to squash migrations of one app into a single new one, which contains the same migration steps plus some optimizations. However it has it’s limitations. If you created a model and deleted it later, it will include two commands (CREATE, DELETE), so it is not that efficient.

The primary reason we couldn’t use squashmigrations were our apps interdependencies. Most of the time we encountered CircularDependencyErrors and after a substantial amount of time, we moved on.

In a future release of Django, squashmigrations will be updated to attempt to resolve these errors itself. — Django documentation

How did we resolve it then?

We tested a couple of ways, but in the end we settled with this procedure:

1. Delete the django_migrations table
2. Remove all migration files
3. Redo the migrations
4. Run migrate

The steps are pretty similar to this answer (StackOverflow), but the answer only refers to one app, with no interdependencies and 3rd party migrations. Also it is at the time of writing 8 years old and a lot has changed in Django since then.

Please only proceed if you know what you are doing and have a backup you can roll back to!

1. Delete the django_migrations table

Firstly we deleted the django_migrations table using this command:

python manage.py shell -c "from django.db import connection; cursor = connection.cursor(); cursor.execute('DROP TABLE django_migrations;');"

Alternatively one could specify the apps for which the migrations should be deleted like this:

python manage.py shell -c "from django.db import connection; cursor = connection.cursor(); cursor.execute(\"DELETE FROM django_migrations WHERE app IN ('app1', 'app2', ...);\")"

We tried with the second command and selected our apps, but it led to anInconsistentMigrationHistory error, on our 3rd party migrations like celery etc. If you want to reset only one or two apps you could (and probably should) use squashmigrations.

2. Remove all migration files

This step was pretty straight forward, after removing the memory for Django on which migration it currently is, we can remove all the migration files.

find . -not -path "./.venv/*" -path "*/migrations/*.py" ! -name "__init__.py" -delete

Notice how we excluded our virtualenv here, because we had issues with 3rd party migrations, which you shouldn’t delete, unless you have a specific reason.

3. Redo the migrations

Pretty obviously we now need new migrations. This step now creates the minimal possible migrations for Django. In a lot of apps this will be just one migration. Interdependencies between apps will however add additional ones.

python manage.py makemigrations

4. Run migrate

Once the migrations are created, you can apply them to a local test database. In our case Django didn’t import some validators correctly, so we had to adjust those.

Now we need to migrate the changes. We can’t just run migrate, because Django “thinks” that the database is blank as of now (because of step 1.) but it is actually populated. That is where fake migration comes in handy:

python manage.py migrate --fake

Running this command, the django_migrations table is brought up to the latest migrations, without actually touching the other tables. After this step we have a functional database again, with the same schema as before but a lot fewer migrations.

Note: fake-initial throws InconsistentMigrationHistoryerrors here, which is why we didn’t use it.

Did it work?

We reduced the number of migrations from 934 to just 84 (down 91%). The 49.79s testing setup was reduced to just 9.55s (80% saved). In the CI/CD pipeline we went from 14m 56s to 7m 46s (GitHub cloud runners are not extremely consistent, so take this with a grain of salt). Even though this is more than the 5 minutes initially assumed, it is reproducible and it takes around 8 minutes ever since.

Before the migration reset

After the migration reset

Additionally we lost about 20k lines of code, which in most projects is irrelevant, but I thought it was pretty neat.

So would I recommend it to anyone with a Django project? No. Smaller projects don’t need this kind of optimization and there are quite a few mistakes that could occur on the way. It is a complex procedure, however it is possible to easily test this with a local database. If the performance difference is minuscule, then don’t worry about applying this to your databases. If it is as significant as for us (i.e. cuts time of testing in half) then you might want to apply it. Make sure that you have a tested backup on hand before starting the procedure and that everyone in the team is on board with the change. Merging any other migrations during the application of the update is not possible and will lead to problems with your database.

We reset the migrations and applied the changes from a local machine to our development and production database. After that we just committed the new migrations. If you have a lot of different environments, I would advise of thinking of a deployment strategy for this change. We just told all our developers that they needed to delete their local databases after the update.

When we started with this I was hesitant; from a lot of blogs and documentation I got the feeling it is not a good idea to meddle with databases on your own while using Django. We thought about scaling up our GitHub workers (and other solutions), but in the end this was a good solution to our problem.