IoT Made Easy: ESP-MicroPython-MQTT-ThingSpeak

Using MQTT protocol, we will get captured data from sensors, logging them to an IoT service, ThingSpeak.com and to a mobile App, Thingsview.

Marcelo Rovai

Published in

MJRoBot.org

13 min readJun 15, 2018

1. Introduction

In my previous article, MicroPython on ESP using Jupyter, we learned how to install and run MicroPython on an ESP device. Using Jupyter Notebook as our development environment, we also learned how to read from sensors (Temperature, Humidity, and Luminosity), using several communication protocols and methods as: Analog, Digital, 1-Wire and I2C, this last one to display our captured data on an OLED display.

Now, on this article and using an MQTT protocol, we will get all captured data, send them to an IoT service, ThingSpeak.com and to a mobile App, Thingsview, where is possble to log and play with them.

Here, a general overview regrading our project:

2. The Hw

The Hw that will be used, is basically the same used on the previous project: Micropython on ESP Using Jupyter. Please, refer to it for a detailled HW explanation (the exception is the servo, that will not be used in this project).

Above you can see the full HW. Connect the devices as shown.

3. Micropython, REPL and Jupyter Notebook

At this point , you should have a Micropython interpreter loaded on your ESP device and so, it is possible to program it using any of the available IDEs, like:

REPL /Ampy
Jupyter Notebook (*)
Mu
ESPCut (Windows only)
… etc

On my article Micropython on ESP Using Jupyter, I have detailed how to download and install the MicroPython interpreter, ESPTool to manage ESP devices and how to use Jupyter Notebook as a Development Environment. Feel free to use what is more comfortable for you.

(*) I usually make all development on Jupyter Notebook, and once I get the final code, I copy them to Geany, saving it as an python script. The script is upload on ESP using Ampy. I found this method very easy to create more complex and professional projects.

4. Sensors

Let’s install the libraries, define GPIOs, create objects and functions for all sensors individually:

A. DHT (Temperature and Humidity)

Install the DHT library and create an object:

from dht import DHT22
from machine import Pin
dht22 = DHT22(Pin(12))

Create a function to read DHT sensor:

def readDht():
    dht22.measure()
    return dht22.temperature(), dht22.humidity()

Test DHT function:

print (readDht())

The result should be a tuple as the one below:

(17.7, 43.4)

B. DS18B20 (External Temperature)

Install the libraries and create an object:

import onewire, ds18x20
import time# Define the pin to be used with 1-wire bus ==> pin 2 (D4)
dat = Pin(2)# create the onewire object
ds = ds18x20.DS18X20(onewire.OneWire(dat))

Scan for devices on the bus

sensors = ds.scan()
print('found devices:', sensors)

The printed result is not really important, what we will need is the first detected sensor: sensors[0]. And now, we can build a function to read sensor data:

def readDs():
    ds.convert_temp()
    time.sleep_ms(750)
    return ds.read_temp(sensors[0])

It is always important to test the sensor using the created function

print(readDs())

If you get a temperature value, your code is correct:

17.5

C. LDR (Luminosity)

The LDR will use the analog pin of our ESP (it is only one in the case of ESP8266 and several to ESP32).

Same as done before:

# import library
from machine import ADC# Define object
adc = ADC(0)

A simple function: adc.read() can be used to read the ADC value. But remember that the internal ADC will convert voltages between 0 and 3.3V in correspondent digital values, varying from 0 to 1023. Once we are interested in “Luminosity”, we will consider Max light as the maximum captured value from the sensor (in my case 900) and minimum light that in my case is 40. Having those values we can “map” the value from [40 to 900] to [0 to 100%] of luminosity. For that, we will create a new function:

def readLdr():
    lumPerct = (adc.read()-40)*(10/86) 
    return round(lumPerct)

You should test the function using print (readLDR()). The result should be an integer between 0 and 100.

D. Push-Button (Digital Input)

Here we are using a Push-Button as a “digital sensor”, but it could be an “echo” of an actuator (A pump that was turned ON/OFF, for example).

# define pin 13 as an input and activate an internal Pull-up resistor:
button = Pin(13, Pin.IN, Pin.PULL_UP)# Function to read button state:
def readBut():
    return button.value()

You can test the button using the function print(readBut()). Whithout pressing it, the result should be “1”. Pressing the button, the result should be “0”.

5. Capturing and Displaying Locally All Sensor Data

Now that we have created one function for each sensor, let’s create another one, responsable for reading all of them at the same time:

def colectData():
    temp, hum, = readDht()
    extTemp = readDs()
    lum = readLdr()
    butSts = readBut()
    return temp, hum, extTemp, lum, butSts

Now, using:

print(colectData())

Will result in a tuple that includes all captured data from sensors:

(17.4, 45.2, 17.3125, 103, 1)

We can also optionally, show those data on a local display:

# import library and create object i2c
from machine import I2C
i2c = I2C(scl=Pin(5), sda=Pin(4))# import library and create object oled
import ssd1306
i2c = I2C(scl=Pin(5), sda=Pin(4))
oled = ssd1306.SSD1306_I2C(128, 64, i2c, 0x3c)# create a function:
def displayData(temp, hum, extTemp, lum, butSts):
    oled.fill(0)
    oled.text("Temp:    " + str(temp) + "oC", 0, 4)
    oled.text("Hum:     " + str(hum) + "%",0, 16)
    oled.text("ExtTemp: " + str(extTemp) + "oC", 0, 29)
    oled.text("Lumin:   " + str(lum) + "%", 0, 43)
    oled.text("Button:  " + str(butSts), 0, 57
    oled.show()# display data using the function
displayData(temp, hum, extTemp, lum, butSts)

As an option, we can use the LED to be ON when we start reading sensors, going OFF after data are displayed. This will help to confirm that the program is working when we have the ESP disconnected from PC and running autonomously.

So, the “main function would be:

def main():
    led.on()
    temp, hum, extTemp, lum, butSts = colectData()
    displayData(temp, hum, extTemp, lum, butSts)
    led.off()

So, executing main(), we will get the sensor data displayed on OLED as shown in the last picture.

6. Running the Local Station Code on ESP Start-up

If you have using Jupyter Notebook to create and test the functions so far, it is to time to have all that were developed so far on a single file script to be executed by our ESP autonomously.

Let’s open any text editor and past on it all code (I like to use Geany):

from machine import Pin
import time# LED
led = Pin(0, Pin.OUT)# DHT
from dht import DHT22
dht22 = DHT22(Pin(12))
def readDht():
    dht22.measure()
    return dht22.temperature(), dht22.humidity()# DS18B20
import onewire, ds18x20
dat = Pin(2)
ds = ds18x20.DS18X20(onewire.OneWire(dat))
sensors = ds.scan()
def readDs():
    ds.convert_temp()
    time.sleep_ms(750)
    return round(ds.read_temp(sensors[0]), 1)# LDR
from machine import ADC
adc = ADC(0)
def readLdr():
    lumPerct = (adc.read()-40)*(10/86)
    return round(lumPerct)# Push Button
button = Pin(13, Pin.IN, Pin.PULL_UP)
def readBut():
    return button.value()# Read all data:
def colectData():
    temp, hum, = readDht()
    extTemp = readDs()
    lum = readLdr()
    butSts = readBut()
    return temp, hum, extTemp, lum, butSts# I2C / OLED
from machine import I2C
import ssd1306
i2c = I2C(scl=Pin(5), sda=Pin(4))
oled = ssd1306.SSD1306_I2C(128, 64, i2c, 0x3c)
def displayData(temp, hum, extTemp, lum, butSts):
    oled.fill(0)
    oled.text("Temp:    " + str(temp) + "oC", 0, 4)
    oled.text("Hum:     " + str(hum) + "%",0, 16)
    oled.text("ExtTemp: " + str(extTemp) + "oC", 0, 29)
    oled.text("Lumin:   " + str(lum) + "%", 0, 43)
    oled.text("Button:  " + str(butSts), 0, 57)
    oled.show()# Main function
def main():
    led.on()
    temp, hum, extTemp, lum, butSts = colectData()
    displayData(temp, hum, extTemp, lum, butSts)
    led.off()'''------ run main function --------'''
main()

Save it, for example as localData.py.

To run this code directly on your terminal you will need Ampy.

First, on Terminal let’s inform Ampy our Serial port:

export AMPY_PORT=/dev/tty.SLAB_USBtoUART

Now, we can see the files that are inside our ESP root directory:

ampy ls

As a response, we will get boot.py, that is the first file that will run in the system.

Now, let’s use Ampy to load our python Script LocalData.py as /main.py, so our script will run just after boot:

ampy put localData.py /main.py

If we use the command amp ls now, you will see 2 files inside the ESP: boot.py and main.py

Resetting your ESP, will make the program localData.py (uploaded as /main.py) run automatically, displaying the sensor data on display.

The above Terminal print screen shows what we have done.

With above code, the display will be shown only once, but we can define a loop on main() function, that will show data on every defined time interval (PUB_TIME_SEC), and for example, until we press the button:

# loop getting data until button is pressed
while button.value():
    led.on()
    temp, hum, extTemp, lum, butSts = colectData()
    displayData(temp, hum, extTemp, lum, butSts)
    led.off()
    time.sleep(PUB_TIME_SEC)

The variable PUB_TIME_SEC must be declared by the time that you want your samples.

To enhance more our code, would be good to inform that we will go out from the loop, for that we will define 2 new general functions, one for clear the display and another to blink the LED on a certain number of times.

# Clear display :
def displayClear():
    oled.fill(0)
    oled.show()# create a blink function
def blinkLed(num):
    for i in range(0, num):
    led.on()
    sleep(0.5)
    led.off()
    sleep(0.5)

So, we can now, rewrite our main() function:

while button.value():
    led.on()
    temp, hum, extTemp, lum, butSts = colectData()
    displayData(temp, hum, extTemp, lum, butSts)
    led.off()
    time.sleep(PUB_TIME_SEC)
    blinkLed(3)
    displayClear()

The final code can be downloaded from my GitHub: localData.py and also the Jupyter Notebook used for development of full code: Jupyter Local Data Development.ipynb.

7: Connecting the ESP to Local WiFi

The network module is used to configure the WiFi connection. There are two WiFi interfaces, one for the station (when the ESP8266 connects to a router) and one for the access point (for other devices to connect to the ESP8266). Here, our ESP will be connected to local network via a router. Let’s call the library and define our network credentials:

import network
WiFi_SSID = "YOUR SSID"
WiFi_PASS = "YOUR PASSWORD"

The function below can be used to connect the ESP to your local network:

def do_connect():
    wlan = network.WLAN(network.STA_IF)
    wlan.active(True)
    if not wlan.isconnected():
        print('connecting to network...')
        wlan.connect(WiFi_SSID, WiFi_SSID)
        while not wlan.isconnected():
            pass
    print('network config:', wlan.ifconfig())

Running the function, you can get as a result the IP address:

do_connect()

The result will be:

network config: (‘10.0.1.2’, ‘255.255.255.0’, ‘10.0.1.1’, ‘10.0.1.1’)

Were, in my case, 10.0.1.2, is the ESP IP address.

8. The ThingSpeak

At this point, we have learned how to capture data from all sensors, displaying them on our OLED. Now, it is time to see how to send those data to an IoT platform, the ThingSpeak.com.

“ThingSpeak is an open source Internet of Things (IoT) application to store and retrieve data from things, using REST and MQTT APIs. ThingSpeak enables the creation of sensor logging applications, location tracking applications, and a social network of things with status updates”

Let’s begin!

First, you must have an account at ThinkSpeak.com. Next, follow the instructions to create a Channel and take note of your Channel ID and Write API Key.

Above you can see the 5 fields that will be used on our Channel.

9. MQTT Protocol and ThingSpeak Connection

MQTT is a publish/subscribe architecture that is developed primarily to connect bandwidth and power-constrained devices over wireless networks. It is a simple and lightweight protocol that runs over TCP/IP sockets or WebSockets. MQTT over WebSockets can be secured with SSL. The publish/subscribe architecture enables messages to be pushed to the client devices without the device needing to continuously poll the server.

The MQTT broker is the central point of communication, and it is in charge of dispatching all messages between the senders and the rightful receivers. A client is any device that connects to the broker and can publish or subscribe to topics to access the information. A topic contains the routing information for the broker. Each client that wants to send messages publishes them to a certain topic, and each client that wants to receive messages subscribes to a certain topic. The broker delivers all messages with the matching topic to the appropriate clients.

ThingSpeak™ has an MQTT broker at the URL mqtt.thingspeak.com and port 1883. The ThingSpeak broker supports both MQTT publish and MQTT subscribe.

In our case, we will use: MQTT Publish.

The figure above describes the topic structure. The Write API Key is required to publish. The broker acknowledges a correct CONNECT request with CONNACK.

The MQTT protocol is supported in a built-in library in the Micropython. This protocol can be used to send data from your ESP8266, over WIFI, to a free cloud database, as ThingSpeak.

We will use the umqtt.simple library:

from umqtt.simple import MQTTClient

And knowing our SERVER ID, it is possible to create our MQTT client object:

SERVER = "mqtt.thingspeak.com"
client = MQTTClient("umqtt_client", SERVER)

Now, having your ThingSpeak credentials on hand:

CHANNEL_ID = "YOUR CHANNEL ID"
WRITE_API_KEY = "YOUR KEY HERE"

Let’s create our MQTT “Topic”:

topic = "channels/" + CHANNEL_ID + "/publish/" + WRITE_API_KEY

Get the data to be sent to ThingSpeak IoT Service, using the created function and associate its response to specific data variables:

temp, hum, extTemp, lum, butSts = colectData()

With those variables updated, we can create our “MQTT Payload”:

payload "field1="+str(temp)+"&field2="+str(hum)+"&field3="+str(extTemp)+"&field4="+str(lum)+"&field5="+str(butSts)

And that’s it! We are ready to send data to ThinsSpeak, simply using the 3 lines of code below:

client.connect()
client.publish(topic, payload)
client.disconnect()

Now, if you go to your channel page (as mine below) you will see that each one of the 5 fields will have data related to your sensors.

10. Sensor Data Logger

Now, that we know that with only a few lines of code it is possible to upload data to an IoT service, let’s create a loop function to do it automatically at a regular interval of time (similar to what we have done with “Local data”).

Using the same variable (PUB_TIME_SEC) declared before, a simple main() function can be created to continuously capture data, logging them on our channel:

while True:
    temp, hum, extTemp, lum, butSts = colectData()
    payload = "field1="+str(temp)+"&field2="+str(hum)+"&field3="+str(extTemp)+"&field4="+str(lum)+"&field5="+str(butSts)
    client.connect()
    client.publish(topic, payload)
    client.disconnect()
    time.sleep(PUB_TIME_SEC)

Note that only the “payload” must be updated, once “topic” is related to our channel credential and will not change.

Looking for your ThingSpeak channel page, you will observe that the data will be loading continuously to each field.

Cover the LDR, put your hand on temp/hum sensors, press the button, etc. and see how the channel will be automatically “logging” those data for future analysis.

Usually, for Data Logging, we should try to use as less power as possible, so, we would not use the LED or a local display. Also, it is common with ESP devices, to put them on “deep sleep”, where the microprocessor will stay on its state of minimum energy until it is time to capture data and send them to the IoT platform.

But, once here the idea is to learn, let’s also include the display and LED as we did before. Doing that, our “logger” function will be:

while button.value():
    led.on()
    temp, hum, extTemp, lum, butSts = colectData()
    displayData(temp, hum, extTemp, lum, butSts)
    led.off()
    payload = "field1="+str(temp)+"&field2="+str(hum)+"&field3="+str(extTemp)+"&field4="+str(lum)+"&field5="+str(butSts)
    client.connect()
    client.publish(topic, payload)
    client.disconnect()
    time.sleep(PUB_TIME_SEC)blinkLed(3)
displayClear()

The complete microPython script can be found here: dataLoggerTS_EXT.py and the Jupyter notebook that was used for development can be also found here: IoT ThingSpeak Data Logger EXT.ipynb.

To upload the script on ESP, on your terminal use the command:

ampy put dataLoggerTS.py /main.py

And press the ESP reset button.

You will have the ESP capturing data and logging them on ThingSpeak.com until the bottom is kept pressed (wait for the LED to blink 3 times and the OLED turns off).

11. The ThingView App

The data logged can be view directly on ThingSpeak.com site or via an APP, for example, ThingsView!

ThingView is an APP developed by CINETICA, that enables you to visualize your ThingSpeak channels in an easy way, just enter the channel ID and you are ready to go.

For public channels, the application will respect your windows settings: color, timescale, chart type and the number of results. The current version supports line and column charts, the spline charts are displayed as line charts.

For private channels, the data will be displayed using the default settings, as there is no way to read the private windows settings with the API key only.

The ThingView APP can be download for ANDROID and IPHONE.

12: Conclusion

As always, I hope this project can help others find their way into the exciting world of electronics!

For details and final code, please visit my GitHub depository: IoT_TS_MQTT

For more projects, please visit my blog: MJRoBot.org

Saludos from the south of the world!

See you in my next article!

Thank you,

Marcelo