Setting Up Django and Elasticsearch in Vagrant on MacOS

Disclaimer: This post assumes that the reader possesses basic working knowledge of Python.

Using Django with Elasticsearch on Vagrant can be quite tricky. This article aims to provide quick workarounds for some of the most common issues. All resources that were used during debugging have been listed at the very end.

Getting started with Vagrant

Vagrant is a software that allows developers to build and manage virtual development environments. This post talks about setting up Django and Elasticsearch in Vagrant, so if you are unfamiliar with Vagrant, follow the link here to install and set it up on your machine. You can also install Vagrant using Homebrew. We will be using Homebrew in this tutorial as well, so if you do not have it installed already, open up your terminal and run the following:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Alternatively, you can copy and run the command provided on the Homebrew website. Once that is done, you can follow the link here to install and set up Vagrant.

One thing to note is that Vagrant is just a development server. If you are in the process of building a web app that you are looking to deploy in the future, you will need to use a different solution. Django has specific deployment requirements as well, that you can find here.

Once you are done with the process of installing Vagrant and VirtualBox, the next step is to provision your virtual machine. For that, go to your terminal and navigate to your home directory. From there, run the following commands:

cd vagrant
vagrant up

After the provisioning process has been completed, run the following command to access the virtual machine through a secure shell:

vagrant ssh

This will allow you to access the virtual machine that you can use for development.

Installing Django

Now that you are inside your virtual environment, you can start installing Django. However, before you run the installation command, you will need to make a few changes in the Vagrantfile. You can find the Vagrantfile in the vagrant folder in your home directory. Pull up the file in the preferred text editor of your choice and uncomment the following line:

# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

Now, change the guest port to the following instead and save the file:

config.vm.network "forwarded_port", guest: 8080, host: 8080, host_ip: "127.0.0.1"

This will allow you to access the development server on http://127.0.0.1:8080 from the browser. However, the server will not work until you actually build your web app. You will also notice a line in the Vagrantfile that says config.vm.box = “bento/ubuntu-18.04” or something similar. This means that because Vagrant is provisioned to work on an Ubuntu system, and Python is pre-installed on Ubuntu, you will not need to reinstall Python inside Vagrant. Once the Vagrantfile is edited, run the following command in your terminal:

vagrant reload

This will reload the virtual machine along with the changes that you just made to the Vagrantfile. Next, you will have to provision the machine once again, and fire up the virtual environment. However, you will need to have virtualenv and pip3 installed before moving on. If you do not have them pre-installed, click here and here to download them outside Vagrant. Then, run the following commands again:

vagrant up
vagrant ssh

The next step is to set up an isolated virtual environment wherein you can start downloading Django and Elasticsearch. We create isolated virtual environments for each individual project, so that the packages that we download remain specific to those projects only. To create a sample project and install Django, run the following commands:

cd src
mkdir sample_project
cd sample_project

Then, run the following in your terminal to fire up a virtual environment:

virtualenv env
source env/bin/activate

You will notice that your command line now shows an (env) sign before the file path. This means that your virtual environment has been activated. Once you are inside the virtual environment, you can start the installation for Django by simply running the following command:

pip3 install django

To check the latest version of Django after the installation has completed, run:

python -m django --version

Now that you have Django installed, you can finally start building the web app. I followed this tutorial by Corey Schafer that was super helpful, and would recommend that you do the same if this is your first time. One distinct feature that you will have to keep in mind going forward is that using the command python3 manage.py runserver will not run your server, and will likely error out because we changed the port address in the Vagrantfile. You will have to always specify the port, so be sure to run the following every time you want to kickstart your server:

python3 manage.py runserver 0:8080

Setting up Elasticsearch

To work with Elasticsearch, you will need the Java Development Kit along with Elasticsearch downloaded on your machine. To do so, open up another terminal window and run the following command:

brew cask install java

To check the details of the installed version, run:

brew cask info java

Next, you can move on with the Elasticsearch installation process. Run the following in your terminal:

brew install elasticsearch

If the installation is successful, move on to the next step. However, if it fails, run:

brew cask install homebrew/cask-versions/adoptopenjdk8

This should ensure that Elasticsearch is installed on your machine. Refer to this article to check if your files are downloaded to the correct folders. Remember that, just like Python, Elasticsearch needs to be installed on your machine and not just in a virtual environment. The reason why we need to do this is because when we install Elasticsearch in the virtual environment and use it as a library, it acts merely as a Python client that communicates between the Python script and existing Elasticsearch cluster. Downloading Elasticsearch on a machine will allow you to connect your client to the cluster and communicate with it. To check if the installation was successful, run the following in your terminal:

elasticsearch

You will notice that the server has started loading. Do not interrupt the server or exit the process. Also, be sure to note the publish_address in the terminal, as we will need it later. It should be published to port 9200, and not 9300. If you are confused as to which one to note, run a quick search for publish_address in your terminal window, and save the one that is bound to port 9200. It should look something like this:

publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}

It may not be bound to 127.0.0.1, but instead to another IP address with the port as 9200, and that is also fine. Navigate to the provided publish address on your browser. If it does not work, try navigating to localhost:9200. It might take a few minutes for the site to load, but once it does, it should look similar to this below:

The next step is to set up Elasticsearch in Vagrant. Open up the Vagrantfile again, add the following line to your list of existing ports, and save the file:

config.vm.network "forwarded_port", guest: 9200, host: 9200, host_ip: “127.0.0.1”

Once that is done, go back to the first terminal window, and deactivate your virtual environment by running the following command:

deactivate

Because of the changes that were just made to the Vagrantfile, you will have to reload Vagrant. Exit out of Vagrant by pressing Ctrl+D, and then run:

vagrant reload
vagrant up
vagrant ssh

Navigate to your Django project directory, and set up the virtual environment as shown above. Then, run the following:

pip3 install elasticsearch

Ensure that your Elasticsearch server is up and running in the second terminal window. You may need to change the Elasticsearch settings on your computer if your port fails to connect. To do so, search for the elasticsearch.yml file on your machine. I found mine in usr/local/etc/elasticsearch/elasticsearch.yml. Open up the file and uncomment the following line:

# network.host: 0.0.0.0

If you do not find that line in the file altogether, add it in the Network section under Set the bind address to a specific IP (IPv4 or IPv6). Ensure that the custom port for HTTP is set to 9200. We do this so that Elasticsearch can run bootstrap checks that check whether all Elasticsearch settings are set correctly. For more information on bootstrap checks and their types, click here.

To quickly check if Elasticsearch is working, create a new file in your Django project folder:

touch documents.py

Open up the file in your text editor and add the following lines of code:

from elasticsearch import Elasticsearch
es = Elasticsearch(["publish_address"]) # add your publish_address here
print(es.cluster.state())

Don’t forget to add your local publish_address provided in your terminal. Once that is done, run the file:

python3 documents.py

If you run into a Connection Error, check if your Elasticsearch is running in the second terminal window. If it is, then wait about a minute and run the file again. You should see a bunch of lines printed, and it should look somewhat similar to this:

This means that your Elasticsearch has been set up in your virtual environment, and you can now start using it to build your web app. Hope this was helpful!

Resources:

1. https://samulinatri.com/blog/django-vagrant-development/
2. https://stackoverflow.com/questions/58022783/elasticsearch-error-bootstrap-checks-failed-binding-non-loopback-address
3. https://qbox.io/blog/series/elasticsearch-python-django-series

Note: Imen Graja helped me during the debugging process. Please show her some love too!