Integrating a Django project with Elastic App Search

Django + Elastic App Search

App Search is an Elastic product which helps developers to quickly build refined search experiences for users, using the power of Elasticsearch. It also provides a dashboard where you can quickly adjust and tune search results. Everything is well documented here.

Tuning relevance is a breeze with Elastic App Search

App Search was initially one of their cloud services, but now we’re also given the option of having a self managed app search instance with their basic license, which is free!

This story is focused on guiding you with integrating a Django project with Elastic App Search. So from here onwards, I will assume that you have a basic understanding of the concepts/features of Elastic App Search, and that you have a new Elastic App Search instance setup locally.

Let’s get started.

Integrating with Django

So let’s say you have setup a brand new self managed instance of App Search, and now you’re looking to integrate it with your existing Django project. Django Elastic App Search to the rescue! “Django Elastic App Search” is a mouthful, but it will make up by helping you to quickly integrate your Django project with Elastic App Search.

Step 1 — Installing

Let’s install and configure it on our Django project. We’ll use our favourite python package manager for this:

Using pip

$ pip install django_elastic_appsearch

or using Pipenv

$ pipenv install django_elastic_appsearch

Next step is to add it to the INSTALLED_APPS setting in our settings.py file.

Then we tell the details about our app search instance by adding these settings to our settings.py

Replace the API key and host with your values. Since we’re running Elastic App Search locally, it will be running over HTTP, so we want to configure that on the settings.py file.

This is not recommended for production use, so remember to change this on your production app.

That’s it for the installation step. Now our Django project knows which app search instance it should be talking to.

Let’s look at how we’re going to use it now. As an example, let’s say we have a Django project which helps us manage the inventory of a book store, with the following Django models.

Now we want to index our Books and Authors to our Elastic App Search instance, to make them searchable. Let’s see how we’re going to do that.

Step 2 — Setting up engines on our App Search instance

Let’s setup the engines for Books and Authors on our brand new Elastic App Search instance. Since Books and Authors are different object types, we will create two separate engines called authors and books. Then we have the option to tune search settings for authors and books, and we also get two sets of API endpoints on app search for authors and books.

Step 3 — Defining serialisers

Now we have to define how our model objects will be indexed to Elastic App Search. We use python object serialisers for that. We define serialisers by importing the base serialiser AppSearchSerialiser from django_elastic_appsearch.serialisers and writing our own serialiser class extending that. AppSearchSerialiser is based off the popular Python object serialiser serpy, and all serpy features like MethodField are available for use too.

Now let’s say we don’t want to index author’s name as two different fields on App Search. We can utilise the serpy MethodField to achieve that.

We can use serpy attr to rename a field name on App Search too.

Above code would mean that the model field country_of_origin would be stored on App Search as country .

Now we write a serialiser for the Book model too.

Note that we’re specifying author_name as a MethodField and not a serpy nested field. As of the day I’m writing this story (2019–08–07), Elastic App Search doesn’t support nested objects. But hopefully soon!

Step 4 — Changing the model classes

Okay, so now we have defined serialisers for our Book and Author models. Now we have to configure our models. We do that by modifying our Django model to extend from django_elastic_appsearch.orm.AppSearchModel. This inherits some extra methods which will help us index objects to our Elastic App Search instance easily.

Then, we define a meta class named AppSearchMeta within our model class, and we specify which app search engine to index to, and which serialiser class to use within that meta class. Everything else remains the same.

That’s it, we’re done! Next, we look at how we’re going to index our authors and books to our Elastic App Search instance.

Step 5 — Indexing and deleting model objects on App Search

Now all our model objects will have methods named index_to_appsearch and delete_from_appsearch. Using these methods, we can index our model objects to our Elastic App Search instance.

Now we go in to our App Search dashboard on the books engine, and into the documents section. We should now see our book object.

You can also index an entire queryset as well.

Now when we go back to our app search dashboard for the books engine, we should see that all our books have been indexed.

Let’s do the same for all our authors too.

Great, now in the dashboard for the authors engine, we should see all our authors have been indexed too.

We can delete objects too, individually or as a queryset, similar to indexing.

Going back to the App Search dashboard, we should now see those objects are now deleted.

Step 6 — Tune searches and build a frontend

Then we can use Elastic App Search features on the dashboard to tune our search application. We can tune relevance, add keywords/synonyms, curate results, and even see analytics.

We can also build search frontends easily for our authors and books using the Reference UI tools.

We can make use of the search API on our App Search instance to build our own custom fronted too. All done! Now we have a fully functional search engine for our Books and Authors.

Advanced usage

In order to achieve the index_to_appsearch and delete_from_appsearch functionality on querysets of App Search Models, we use Django custom queryset managers. So if there’s a custom queryset manager in place on one of your App search indexed models, you would run into errors when you try above methods, because those methods wouldn’t exist on your custom querysets.

Using the same book inventory project example, let’s say we have a custom BookManager defined for the Book model, which adds extra some extra functionality.

With this setup, if we try to do the following, we would run into issues.

To get around this issue, we can base our custom queryset manager on Django Elastic App Search’s custom queryset manager.

With this setup, we’d get our some_cool_method and also the index_to_appsearch and delete_from_appsearch methods on our querysets.

Writing Tests

Untested Code is Broken Code.

So now we have our functionality, but we want to run automated tests against the app search models. During our automated tests, we wouldn’t have an App Search instance running, so any calls to index_to_appsearch or delete_from_appsearch would certainly fail. How do we get around that issue? Worry not, Django Elastic App Search comes with a base test case mixin you can use with your test cases which mocks all calls to the app search instance. Then you can use the following methods to check call counts to different mocked app search methods.

• self.assertAppSearchQuerySetIndexCallCount — Check the number of times index_to_appsearch was called on a appsearch model querysets.
• self.assertAppSearchQuerySetDeleteCallCount — Check the number of times delete_from_appsearch was called on an appsearch model querysets.
• self.assertAppSearchModelIndexCallCount — Check the number of times index_to_appsearch was called on an appsearch model objects.
• self.assertAppSearchModelDeleteCallCount — Check the number of times delete_from_appsearch was called on an appsearch model objects.

Note that we’ve used MockedAppSearchTestCase as a mixin class in combination with Django’s TestCase . We can use the MockedAppSearchTestCase in combination with any test case class that derives from python unittest.TestCase . Also note that the inheritance order is important, so we have to define MockedAppSearchTestCase before TestCase .

That’s it! That should cover everything you need to integrate a Django project with Elastic App Search. I will keep updating this story with any new features that’s added to the Django Elastic App Search package.