Django, Docker and Elasticsearch

Let’s create Django application which will utilize Elasticsearch engine what can do a lot of things like full-text search, fuzzy search, autocomplete, spell check and etc.

If you’re new to Django framework already trying to do something with it I think you know what Django has great tutorials with some real projects. One of them is a Polls application.

What we gonna do here is to extend that app to have nice search, powered by Elasticsearch.

Django Poll application

This application is pretty simple and it basically has two models: Question and Choice. You can copy-paste sources from Django Tutorial at Django page or just clone it from my GitHub repository:

git clone https://github.com/inoks/django-polls-elasticsearch
cd django-polls-elasticsearch
docker-compose up

If you see some errors or you don’t really understand what is going please take a look at Django Tutorial.

If you see your server up and running — go to the admin page and add some Questions.

Elasticsearch instance

We will use Docker to run Elasticsearch. Those steps are pretty well documented on the Elasticsearch website, except we need to use Elasticsearch 2.x instance since the newest versions are not really supported by Django-Haystack.

So let’s add new services section to our docker-compose.yml file:

elasticsearch:
  image: launcher.gcr.io/google/elasticsearch2
  ports:
    - "9200:9200"
    - "9300:9300"

Elasticsearch uses two ports by default: 9200 for RESTful API and 9300 for node communication (in case you run multiple instances).

Let’s run our Docker configuration with docker-compose up and test if everything works well:

curl http://127.0.0.1:9200/_cat/health
1542963817 09:03:37 elasticsearch yellow 1 1 0 0 0 0 0 0 - 100.0%

You may have some other values in the response, but if you were able to receive some response — this is a good sign that Elasticsearch works.

Connect Django with Elasticsearch

We will use Django Haystack app to do all integrations with Elasticsearch. In order to use Haystack we also need to install Elasticsearch official client for Python — elasticsearch-py. Unfortunately, Haystack only supports 2.x version of the client. So let’s install Haystack and latest 2.x version of Elasticsearch package:

pip install django-haystack elasticsearch>=2.0.0,<3.0.0

Add haystack to your INSTALLED_APPS list at mysite/settings.pyfile:

INSTALLED_APPS = [
    'polls.apps.PollsConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'haystack',
]

And provide basic configuration, so Haystack will know which search engine we want to use and where the server is located:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Create Search configuration

In order for our search engine to work we need to configure which data and fields we want to send to the search engine and how the search engine needs to treat them. Django-Haystack allows us to do that via SearchIndex classes.

Let’s create search_indexes.pyinside our polls application:

class QuestionIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='pub_date')
    def get_model(self):
        return Question
    def index_queryset(self, using=None):
        """Used when the entire index for model is updated."""
        return self.get_model().objects.filter(pub_date__lte=timezone.now())

And as we use a template for our text field, we also need to create a question_text.txt template file, which is as simple as:

{{ object.question_text }}

That means Haystack should only index values from the fieldquestion_text .

Adding Search page to the Polls app

So we almost ready now to use new Elasticsearch with our app and we need to create a view, where we will utilize it.

Django-Haystack comes with a predefined view for search, so we need just add it url patterns:

urlpatterns = [
    path('', include('polls.urls')),
    path('search/', include('haystack.urls')),
    path('admin/', admin.site.urls),
]

And create a template search/search.htmlfor it, which should contain the form and can display paginated results:

<h2>Search</h2>
<form>
  {{ form.as_p }}
  <br/>
  <input type="submit" value="Search">
</form>
<ul>
  {% for result in page.object_list %}
    <li>{{ result.object }}</li>
  {% endfor %}
</ul>

Populate Elasticsearch index

Haystack comes with some tools to populate and update Elasticsearch index. For the initial index build we need to run:

./manage.py rebuild_index

To update the index after we added or updated entries we need to run:

./manage.py update_index

Using search

In order to use the search, you need to visit http://localhost/search/ page and enter a query.

That’s it!

Complete source code and Docker configuration available at Github.

In the next article, we will add Autocomplete to the search input field.