Django + Mongo = Pytest FTW! A clean way to manage connecting and dropping of database between tests.

Sometime ago I started working on my own more of a playground project where I’m mostly trying to learn new things, but I also hope to create something useful from it that’s also in my field of interest. I figured out that it might be a good project for learning how to use integrate Django with Mongo database which I plan to use for some of the models.

Incorporating a new technology in your stack is never easy - in this case, besides learning new way to model the data, I also had to find a good ORM, came up with proper provisioning for my Vagrant box and TravisCI, and finally figure out how to do testing in this new environment.

In my project I’m using MongoEngine as an ORM which is pretty similar to Django’s ORM, well maintain, more or less easy to setup, and supported by many other plugins and libraries. I also use Pytest. Let’s look at most basic way to setup this environment for tests:

### FILE: testing_settings.py
import mongoengine
from .settings import *
# [...] some settings overrides
mongoengine.connection.disconnect() # disconnect main db first

In my main settings.py file I’m connecting to proper Mongo database, the above file is run only by Pytest, so first I’m disconnecting. This looks the same in every other case that I will later write about. After that we need to connect to our testing database, so still in testing_settings.py:

connect('testdb', host='mongomock://localhost')

From now on every call made by MongoEngine will be made to this database. However if you are used to Django testing you probably would want to have a clean database in each test case. We can achieve this by calling drop_database() method on DB instance which is returned on proper connect() so one way to connect and drop on each test is to use Pytest’s so called xUnit stylesetup_method and teardown_method:

import mongoengine as me
from ..models import Site
class SiteTests:
  def setup_method(self):
    self.db = me.connect(
              ‘testdb’,
              host=’mongodb://localhost’
            )
  def teardown_method(self):
    self.db.drop_database(‘testdb’)
    self.db.close()
  def test_object_creation(self):
    site = Site(name=’test_site’)
    site.save()
    assert Site.objects.first().name == site.name

So in our setup method we are connecting to a testing database, and that database is then bound to an attribute so it can be easily dropped and the connection closed on the teardown. Test is passing, and everything is ok, but…

I usually write my tests in several files, most often these are test_models.py test_forms.py test_views.py In this files usually there are several classes, when you multiply it by many apps it becomes obvious that you need to use that teardown and setup few times… Let’s come up with a better way of doing it (besides OOP methods).

Pytest fixtures for the rescue! Pytest assertions are very clean, there are plenty of configuration options, but fixtures feature is the biggest deal - if you still aren’t using them, then you definitely are missing a fantastic tool. Let’s look how we can use them in this case:

### FILE: fixtures.py
import pytest
import mongoengine as me
@pytest.fixture(scope=’function’)
def mongo(request):
   db = me.connect(‘testdb’, host=’mongodb://localhost’)
   yield db
   db.drop_database(‘testdb’)
   db.close()

I usually create this file in utils or common directory, so then I can import it everywhere where I need it. This fixture connects to the database, yields it, and after it scope finishes db is dropped, and connection closes. I set the scope to function explicitly, but it’s a default value. If I set it to module then all tests within a single module would use the same database. Fixture is used like a dependency injection:

from utils.fixtures import mongo
from ..models import Site
class SiteTests:
   def test_object_creation(self, mongo):  # use the fixture
     site = Site(name=’test_site’)
     site.save()
     assert Site.objects.first().name == site.name
   def test_object_creation2(self, mongo):   # use the fixture
     site = Site(name=’test_site2')
     site.save()
     assert Site.objects.first().name == site.name

Both tests pass. In the same way we can import this fixture to other modules, and everything will work. If I needed the database to be persistent for couple of tests then I could create another, almost the same fixture but with scope set to for example class. Prior to Pytest 2.10 creating fixtures with teardowns (dropping the db in our case) was a bit more complicated, but now using yield is pretty neat, I love it!

So we still have to inject the fixture as one of the arguments for each test, it will be quite a lot of writing, so… can it get any better? It actually can :)

from utils.fixtures import mongo
@pytest.mark.usefixtures('mongo')
class SiteTests:
   def test_object_creation(self):
     site = Site(name='test_site')
     site.save()
     assert Site.objects.first().name == site.name

By decorating the class Pytest automatically uses given fixtures in each method. Keep in mind that in the decorator the fixture must be given as a string (yet it still needs to be imported), otherwise tests in this class won’t run, no warnings, watch out for that.

MongoEngine’s documentation suggests using Mongomock in tests, which is built around excellent Mock module from the Python standard library. To use it - after it’s installation - just change the connection URI like this:

# mongodb => mongomock
db = me.connect(‘testdb’, host=’mongomock://localhost’)

And that’s it! Since it’s not a real db tests are faster and we no longer need to have Mongo itself installed (which might be a case for some CI tools), but everything works fine! At least when test cases aren’t to complicated - well, this is just a mock, and not every Mongo functionality is implemented, but that’s an issue, which can be easily managed with fixtures: just create one fixture for “real” db, and one for the mock, then decorate your classes accordingly.

That’s not all: I haven’t mention yet that fixtures are modular, which means that you can build couple of them into one, and that they can also get parameters, but I will write more about it some other time, meanwhile I suggest going through the docs.

I hope this was helpful, cheers.