IoT with an ESP32, InfluxDB and Grafana

Part 2: Cloud Hosting

In the first post, we got set up with streaming data securely to Influx from an embedded device. That server was just on my laptop, which isn’t much use if I switch it off or am not connected to WiFi. It could be run on a Raspberry Pi or a standard server, but I decided to set it up on a cloud server. It avoids having anything else running in my house, and it’s all doable in the free tier of AWS. I picked AWS because it has the biggest knowledge base, and I know a guy who knows AWS. (I’d like to try it on a platform run by a company I don’t dislike).

Creating an AWS Instance

The basic compute service is EC2, which I used to set up a Linux machine which I can then install InfluxDB and Grafana on. Then it can run 24–7 and I can check it whenever. I can also connect to the machine itself over SSH, and use the Influx CLI from my Macbook, in the same way as when the server was local. The AWS free tier includes 750 hours / month a small EC2 server, which just happens to be an entire month.

Before creating the instance itself, we need to create a security group, which controls inbound and outbound traffic. On the main EC2 dashboard, under the Network and Security heading, click on the Security Groups link. Create a new Security Group, calling it something sensible, and then add the following inbound rules:

• SSH, so we can connect remotely over the command line
• Port 8086, the (default) port InfluxDB uses for its HTTP API
• Port 3000, the (default) port Grafana uses.

Next, go to the EC2 dashboard and select “launch instance”. Select an Amazon Linux 2 machine image and then pick the t2.micro option (note it says “free tier”). At this point, you can just click “review and configure”, since most of the options are defaults. However, on that page, be sure to click “edit security groups” and select the one you just created. Click “launch”, and then create a new key pair. MAKE SURE TO DOWNLOAD THIS KEY PAIR BECAUSE IT’S THE ONLY TIME YOU CAN AND YOU NEED IT TO CONNECT. So also, save it somewhere safe but sensible. The SSH client doesn’t like if private keys have loose permissions: chmod 400 /path/to/<keyname.pem> fixes this.

Finally, launch the instance. On the “instances” menu of the EC2 dashboard, you’ll see that it’s initialising.

Configuring the Instance

When the status says “ready”, you can start working with it. Make a note of the IPv4 Public IP. For most things you can also use the DNS address, but I found that it takes a while for that to become active and you can’t connect right away. The Arduino code needs the plain IP anyway.

Sweet ASCII art

Connect to the instance: ssh -i /path/to/<keyname.pem> ec2-user@<IP Address>. You’ll get a warning about the authenticity of the key: type yes. Now we can install stuff.

Amazon’s Linux implementation uses the yum package manager. Install InfluxDB, Grafana and tmux:

sudo yum update
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.4.2.x86_64.rpm
sudo yum localinstall influxdb-1.4.2.x86_64.rpm
sudo yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.3-1.x86_64.rpm
sudo yum install tmux

InfluxDB needs to be set up like in part 1. sudo influxd and, in another tab (you don’t need to SSH in for this), influx -host <ip-address> allows you to access the InfluxDB server on AWS directly from your Mac. Real unsafe, see? Luckily, we’re fixing this now. CREATE USER <your-name> WITH PASSWORD '<your-password>' WITH ALL PRIVILEGES. Now shut down the influx server before anyone starts lurking. To secure it, we need to upload the Influx certificate and private key.

We can do this using Secure Copy. Unfortunately, we can’t put the files in the correct folders because of privileges, but we can dump them in the home directory and move them. As with ssh we need to pass the AWS key pair, and then specify the local directory of the file to transfer, and the directory of where we want to put it on the remote server (in our case, ~).

scp -i /path/to/<keyname.pem> /usr/local/etc/ssl/<influx-cert-name>.crt ec2-user@<ip-address>:~
scp -i /path/to/<keyname.pem> /usr/local/etc/ssl/<influx-cert-name>.key ec2-user@<ip-address>:~

You can check the files are there by running ls on the tab you ssh’d into. Now, move each to its appropriate directory.

sudo mv influxdb-selfsigned.crt /etc/ssl
sudo mv influxdb-selfsigned.key /etc/ssl

Finally, edit the config file: sudo nano /etc/influxdb/influxdb.conf and, as before, set the following under the [http] header:

auth-enabled = true
https-enabled = true
https-certificate = "/etc/ssl/influxdb-selfsigned.crt"
https-private-key = "/etc/ssl/influxdb-selfsigned.key"

Now, test it by running sudo influxd again, and in another tab check you can connect remotely via the CLI: influx -ssl -unsafeSsl -host <ip-address> -username <yourname> password <yourpass>

Shut down the InfluxDB daemon, because instead we’re going to run it as a service. This runs it in the background, so when you disconnect from SSH it still runs. Alternatively you can use Screen or tmux, but they’re a bit clunky and I imagine less reliable. sudo service influxdb start gets InfluxDB up and running. Note that this looks for the .conf file in the default location.

Running the Server

Now we can use the example Arduino sketch to test the connection. In the CLI, CREATE DATABASE aws_test (or whatever), then fill out the WiFi details, IP address and database name. Once that’s been flashed, check the Serial monitor for good news, and in the CLI, USE aws_test then SELECT * FROM /.*/ (which returns everything from all measurements) and we should see a few rows of data.

We can start Grafana in the same way: sudo service grafana-server start . In a browser tab, go to your instance’s address at port 3000, and you should see the Grafana log-in page. Now it can be set up exactly as in part 1. Remember that as InfluxDB and Grafana are on the same server, you can treat InfluxDB as being on localhost. Seen enough, you’ll be able to visualise the data coming in.

Environment Monitoring

And that’s about it! For the environment sensor, I used a Bosch BME680 from Pimoroni, which measures temperature, pressure, humidity and air quality. As of now, it’s difficult to integrate Bosch’s closed source Air Quality Index source code, but apparently that’s coming soon. The examples directory contains a template for running this sensor, just connect it to the I2C pins (remember your pullups), power and ground and you’re good to go.

Part 3?

I’ve got a few ideas of where to go with this next — if you have any suggestions, let me know.

Firstly, I should set up Grafana for HTTPS because you are entering passwords and such.

Secondly, while streaming directly over HTTPS is nice, what probably makes more sense for IoT is setting up an MQTT broker and publishing data to that. Then, you can forward that data to InfluxDB. If you’re fancy, you could even have a couple of databases on different servers to add some redundancy. Each EC2 instance could run Telegraf to listen to the MQTT topic, and ingest the data as it comes in.