Setting up Apache Superset on Dockers and connecting it to a MySQL Docker container: A foolproof guide

Setting up Apache Superset is relatively easy. The challenge actually lies in connecting the database, in this example, MySQL to Superset.

So, let’s start with the setup.

Step 1: Prerequisites

Prerequisites are dependent on where you’re setting Superset up.

In this article, we are using

• Computer- 8 GB RAM; OS: Ubuntu
• Docker, Docker Compose

[You could also use Windows with a Ubuntu virtual machine and give the virtual machine up to 8 GB RAM. However, from personal experience, at least 10 GB RAM needs to be allocated to the VM for Superset to run smoothly.]

So, assuming you don’t have Docker engine installed, you can run these commands in your terminal. [You can find them in more detail here.]

There! The repository is set up. Let’s move on and install docker engine.

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

We’ll run the Hello world image to see if the engine has been properly installed.

sudo docker run hello-world

The output of the program will be something like this:

For more settings, like for example, if you want to run docker as a non-root user and so on, you can find detailed instructions here.

Time to install Docker-compose. We’ll use curl, so if you don’t have it, you can install it using these commands:

sudo apt update
sudo apt install curl

So, verify the installation by typing curl in your terminal. You’ll get an output like this.

Moving on, for Docker-compose, start by running this command

sudo curl -L “https://github.com/docker/compose/releases/download/1.28.2/docker-compose-$(uname -s)-$(uname -m)” -o /usr/local/bin/docker-compose

This downloads the current stable release of Docker Compose. After that,

sudo chmod +x /usr/local/bin/docker-compose

Finally, check the version.

docker-compose — version

Now, that we’ve got Docker and Docker Compose installed and set up, let’s get started with Superset.

Step 2: Setting up Superset and the MySQL container and connecting them

We’ll be following the official documentation, find the link here.

Clone Superset’s Github repository, after which a new folder named “incubator-superset” will be created in your current directory.

git clone https://github.com/apache/incubator-superset.git

Go to the folder, and create a file called requirements.txt where we’ll fill in the dependencies of the app along with specific versions we’ll need. So, here, we’re only going to require “mysqlclient”, however, the version we’ll use is 1.4.6. So, let’s add it in!

cd incubator-superset
touch ./docker/requirements-local.txt
echo “mysqlclient==1.4.6” >> ./docker/requirements-local.txt

Let’s build Superset now, (and remove the prebuilt containers) from the dockerfile, which you can find in the folder, “incubator-superset”.

docker-compose build — force-rm

Compose it up. (I’ve run it in detached mode). Then we move on to set up the MySQL container.

docker-compose up -d

Move out of the incubator-superset directory, and then create the MySQL container.

cd 
docker run --detach --network="incubator-superset_default" --name=vedamysql --env="MYSQL_ROOT_PASSWORD=vedashri" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

There are a lot of flags attached to the run, so let’s break it down to understandable bits:

--detach: to run the container in detached mode, i.e. start it up and run it in the background.

-- network=”incubator-superset_default”: we’re connecting the container to the incubator-superset network by a simple bridge network connection.

--name=vedamysql: it is the name of the container; you could name it anything of your choice. But remember the name for later.

--env=”MYSQL_ROOT_PASSWORD=vedashri”: set the password to login to the server as an environment variable, so that you won’t have to login to mysql every time. Remember this too!

-- publish 3306:3306: map the container’s 3306 to the host’s 3306 and make the port available to other containers and services.

mysql: name of the image to be pulled

--default-authentication-plugin=mysql_native_password:

“A plugin that performs native authentication; that is, authentication based on the password hashing method in use from before the introduction of pluggable authentication in MySQL”[1]

And voilà! Our MySQL container is ready!

Now, we’ll create a test database in MySQL, to be used further. Here’s how you do it.

docker exec -it vedamysql mysql -uroot –p

Enter the password(which you were asked to remember). You’ll enter the MySQL terminal:

create database test1;
exit

Now we’re mostly done. What we’ve got to do now is rebuild the containers in the ‘incubator-superset’ network.

cd incubator-superset
docker-compose build — force-rm

Then, the limit of the number of files monitored by Docker-compose must be increased, else an error is thrown up; so here’s what we can do.

echo fs.inotify.max_user_watches=524288 > /etc/sysctl.conf
sysctl -p

Then restart the containers to let the changes take effect, and compose it up!

docker-compose restart
docker-compose up

Then you’ll encounter a huge log of data and when the stream slows down, you can go to http://localhost:8088 and open up Superset.

Step 3: Checking the connection

Login with default username and password, both of which are ‘admin’. You can create more users later.

Then go to Data-> Databases-> the +Database on the top right

And then enter the database name and the SQLAlchemy URI in this way:

mysql://root:vedashri@vedamysql:3306/test1

or

mysql://mysql-user:mysql-password@container-name:port/database-name-as-stored-in-mysql

Test the connection.

If you get a ‘connection looks good’ message, then your connection is successful!

You can then go the tabs next to connection, and fill in the details according to your needs, and then go ahead and add the new database to Superset.

This was all about making and connecting a MySQL container to Apache Superset, on Dockers.

Extras: Tips to troubleshoot-

We posted our issues on

• Stack Overflow
• Slack
• Gitter
• Cold email to the developers of Superset( did not work, as they’re really busy people)
• Quora too( we were getting desperate 😄)
• Github ( they responded pretty fast!)

I have run the entire script as a root user; if you run it as a non root user, please add sudo to the commands as and when required.