Mainflux Bootstrap Service — Automatically configure devices on power up

How to use Mainflux bootstrap service to automatically configure devices on power up

Mainflux is a performant, scalable and secure open-source IoT platform for development of Internet of Things solutions, applications and smart connected products. Mainflux can be deployed on the edge, on premise and in the cloud. It is built as a set of microservices written in Go, containerized by Docker and orchestrated with Kubernetes.

Mainflux is developed to manage a huge fleet of devices. Every device needs to be configured properly to communicate via Mainflux. In general, the process of configuration that aims to enable the functionning and use of a device is called bootstraping. There are two major ways to configure IoT devices. On one hand, we can configure a device in the place of production, i.e. in the factory. On the other, we can do it when the device is powered up, by connecting a device to the cloud and fetching a configuration from a configuration server. So, there are two major ways of device bootstrapping:

1. Factory bootstrap. It involves not only configuring a device on the place of production (factory), but also and, perhaps, more importantly, it implies the configuration hardcoding.
2. The bootstrap procedure. It implies a minimal necessary factory configuration hardcoding: a device is only supplied with a URL address of a bootstrap server and credentials needed to acces the server. Once it is powered up and online, the device makes a simple HTTP request to the bootstrap server providing credentials and asking for a configuration details. Once the device is authenticated by the server, it receieves, from the server, all the necessary configuration details. (For a more detailed explanation, please read this article.) From that point on, it is up to the device to decide what it will do with the received configuration. It might, for example, store the configuration in a .yaml or .toml file, parse the file, deserialize its content and set up a deamon program that implements the business logic of a device service. If a deamon is restarted, the device might check whether there is a configuration file on a disk and recreate a deamon program with exactly the same settings as before or it can, in the absence of a configuration file, repeat a bootstrap procedure, i.e. send a request to a bootstrap server in order to receive the new configuration data.

Mainflux bootstrap service

In this article, I will concentrate on the bootstrap procedure and show how it is carried out in Mainflux by means of its bootstrap service. Mainflux is a scalable and secure open source and patent-free IoT cloud platform written in Go that acts as a middleware for application and device messaging over various network protocols (i.e. HTTP, MQTT, WebSocket, CoAP). For more details, check out the official documentation. In the terminology of Mainflux — in its formal ontology — , device and application are represented as things. Things communicate over channels. Typically, a device thing sends sensor measurements, in SenML format, to application thing over a data channel and eventually receives back control and exec messages from application thing. In order to communicate, things have to be connected to channels. Once they are connected to channels they can listen for incoming messages and publish outgoing messages. Channels are, under the hood, implemented via NATS message broker.

A Mainflux enabled physical device needs to be represented as a thing connected to channels. So, in order to communicate by means of Mainflux, a device needs to know its corresponding Mainflux thing key (basically, thing’s token) as well as channels ID’s (based on NATS subjects) connected to that Mainflux thing. In order to facilitate device configuration, Mainflux has a bootstrapping service.

Before we go on, we need to make an important difference beetween bootstrapping and provisioning procedures. Provisioning refers to entities management while bootstrapping is related to entity configuration. In the terminology of Mainflux, provisioning refers to things and channels management, i.e. to things and channels creation, update and deletion (CRUD paradigm) as well as to things and channels connections management — a NATS subject PUB/SUB authorization. On the other hand, bootstrapping refers to a physical device or its software application configuration and setup.

For example, in order to connect a physical device, let’s say a gateway, to the Mainflux, we must first create a Mainflux thing that will represent a physical device,

and Mainflux channels used by the device to send and receive messages.

We must also connect a device thing to possibly more than one channels. Things and channels creation and connection belong to the provisionning aspect of the system implementation. We provision things and channels on the server, located on premise or in the cloud.

After that, we have to determine the physical device related information. We need three pieces of information: server URL, device id and key.

Finally, we need to communicate things and channels information to a physical device (gateway) that will use Mainflux to communicate with other devices and applications. The device needs to know how it is represented by the Mainflux in order to use it to send and receive messages. The role of a bootstrapping procedure is exactly to communicate this information from a server to a device, so a device — a gateway in our case — can use this information in order to set up a deamon program which implements a desired business logic. This means that a bootstrapping server needs to know both sides of the story: it needs to know Mainflux related data (thing credentials, channels connected to a thing) and physical device related data (device’s id and key) and has to establish and store a connection between the former and the latter.

Configure a gateway using Mainflux bootstrap service

We will now see, using an example, how to carry out a bootstrap procedure for a physical device with the help of the Mainflux bootstrap service.

Mainfluix provisioning

In order to use Mainflux, we first need to install Mainflux. Then, we need to create a user account, get user’s JSON Web Token (JWT) and use JWT to create things (devices and apps) and channels over which things will communicate. It is easy and takes little time to do it. You can find all the necessary instructions here. You can use the CLI of your OS terminal to quickly provision the system for testing. When I performed the step 3 of the instructions, i.e. when I ran the command for the rapid provisioning,

mainflux-cli provision test

I got the following result:

{
  "email": "distracted_mirzakhani@email.com",
  "password": "123"
}
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTcxODM2NTIsImlhdCI6MTU1NzE0NzY1MiwiaXNzIjoibWFpbmZsdXgiLCJzdWIiOiJkaXN0cmFjdGVkX21pcnpha2hhbmlAZW1haWwuY29tIn0.Q3WrhKSZWtgO6cG2ci5UizwDDkML4LqvRrKq88uDEN0"
[
  {
    "id": "6ef63b46-9262-4355-aca3-2c84eb5b5085",
    "key": "8171bba6-ad71-4a3e-b885-55c283736974",
    "name": "d0",
    "type": "device"
  },
  {
    "id": "1851374e-75e1-4b1d-9d0c-a74a5e7a5217",
    "key": "39d01484-1819-403a-8579-ab3197d05641",
    "name": "d1",
    "type": "app"
  }
]
[
  {
    "id": "77d9ac08-901c-40ca-a69a-5d4f9dcc7233",
    "name": "c0"
  },
  {
    "id": "a225801d-0a0e-4b7a-9860-619bf943fe54",
    "name": "c1"
  }
]

Your result will certainly be different, since the values are generated randomly by Mainflux. Briefly, we got Mainflux username and password, user’s JWT, an array of two things with names “d0” and “d1” — one representing a device and the other representing an application — and an array of two channels with names “c0” and “c1”.

I will use the given values to compose a configuration JSON that I will later post to the Mainflux bootstrap server. In order to have all the necessary information for the configuration JSON, we also have to determine the physical device’s id and key. The latter will be stored on the physical device itself and will be matched with the Mainflux thing supposed to represent the physical device. I have decided to store my physical device configuration details in the config.toml file (see below), stored on the device itself.

title = "Device Config"
[server]
URL = "http://localhost:8200/things/bootstrap/"
[thing]
external_ID = "00-AC-87-B4-86-13"
external_key= "00e0dcfa-6b46-11e9-a923-1681be663d3e"

As a summary, the otput of the “mainflux-cli provision test” gave us the Mainflux related data stored in the cloud or on premise. On the other hand, config.toml file, located on the physical device itself, stores the device related data. We need to connect these two pieces of information. Here is an example JSON configuration that relates these pieces and that we will post to the Mainflux bootstrap server:

{  
   "external_id":"00-AC-87-B4-86-13",
   "external_key":"00e0dcfa-6b46-11e9-a923-1681be663d3e",
   "thing_id":"6ef63b46-9262-4355-aca3-2c84eb5b5085",
   "channels":[  
      "77d9ac08-901c-40ca-a69a-5d4f9dcc7233",
      "a225801d-0a0e-4b7a-9860-619bf943fe54"
   ],
   "content":"gateway"
}

As I have said, we can freely choose the external_id and external_key. However, they must exactly match the ones stored on the physical device (see above). On the other hand, thing_id is an internal, Mainflux generated thing identification that I have read from the output of the “mainflux-cli provision test” command. The same goes for channels, a list of Mainflux generated channels ids. On the contrary, we can freely choose the content field according to the configuration needs of our device. I have put “gateway”, but you can put anything you need in any format you need. Once again, you will certainly get, when you execute “mainflux-cli provision test”, different values and be careful to use exactly these values in the thing_id and channels fields of your configuration JSON object.

Mainflux bootstrap server

Before we use it, we need to start a Mainflux bootstrap server. Currently, it is not launched as a part of a Mainflux Docker composition. In order to do it, we need to get a docker-compose.yml file here. As usual with Github hosted files, be careful to download a raw file version. I donwloaded the file with

wget https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml

Once you get the file, use it to create and launch a docker container with the bootstrap server (you have to be in the same directory as a docker-compose.yml file):

docker-compose -f docker-compose.yml up

We are now able to send an HTTP requests to the Mainflux bootstrap server. In this swagger file, you can find the complete specification of the requests you can send to a bootstrap server. To post a configuration of our device to the Mainflux bootstrap server, send a POST request to the Mainflux bootstrap service similar to this one:

curl -s -S -i -X POST -H "Authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTcxODM2NTIsImlhdCI6MTU1NzE0NzY1MiwiaXNzIjoibWFpbmZsdXgiLCJzdWIiOiJkaXN0cmFjdGVkX21pcnpha2hhbmlAZW1haWwuY29tIn0.Q3WrhKSZWtgO6cG2ci5UizwDDkML4LqvRrKq88uDEN0" -H "Content-Type: application/json" http://localhost:8200/things/configs -d '{  
   "external_id":"00-AC-87-B4-86-13",
   "external_key":"00e0dcfa-6b46-11e9-a923-1681be663d3e",
   "thing_id":"6ef63b46-9262-4355-aca3-2c84eb5b5085",
   "channels":[  
      "77d9ac08-901c-40ca-a69a-5d4f9dcc7233",
      "a225801d-0a0e-4b7a-9860-619bf943fe54"
   ],
   "content":"gateway"
}'

If all went well, you will get a

TTP/1.1 201 Created

response, together with the header related information,

Content-Type: application/json 
Location: /things/configs/6ef63b46-9262-4355-aca3-2c84eb5b5085 
Date: Mon, 06 May 2019 14:20:34 GMT 
Content-Length: 0

We are particularly interested in the

Location: /things/configs/6ef63b46-9262-4355-aca3-2c84eb5b5085

since it gives us an id of the configuration as it is stored on the bootstrap server. Please note the id you got from the server (yours will be different than mine). You need it in order to send some of the HTTP requests to the Mainflux bootstrap server (we won’t do it in this tutorial). NB: at the present time, config ID is the same as the Mainflux thing ID, and MF bootstrap server allows only one configuration per MF thing id.

Go language device configuration basic application

Our physical device needs to be equiped with an application that will be responsible for sending an HTTP POST request to the Mainflux bootstrap server once the device is powered up and has booted. In this guide, we will program only a basic application that gets the configuration from the bootstrap server, displays it and sends and receives few test messages. The idea is to use Mainflux credentials — that we obtained from a bootstrap server and that describe Mainflux thing as well as channels it is connected to — to send messages programmatically, i.e. via our basic app, on the behalf of the Mainflux thing in question.

The real world application would normally have to fetch a configuration from the server, do all the necessary device configuration and eventually listen to the server for any changes of the configuration. It should aslo be able to fetch again the configuration data from the server when the device is rebooted (for the maintenance, because of the power failure or for any other reason). Finally, a real world application should be capable of storing the configuration in a text file (.toml or .yaml, for example) and re-read the configuration data when the device is rebooted and can’t access the bootstrap server.

Our minimal app will only have two files. Both of these files have to be in the same directory. One file, config.toml, to store the configuration (see above):

title = "Device Config"
[server]
URL = "http://localhost:8200/things/bootstrap/"
[thing]
external_ID = "00-AC-87-B4-86-13"
external_key= "00e0dcfa-6b46-11e9-a923-1681be663d3e"

and the other file to read, from config.toml, the data needed to compose an HTTP post request to the Mainflux bootstrap service, main.go. You can find the code of the main.go file and of our entire basic application on https://github.com/darkodraskovic/mfbootstrap. I will just comment the parts specific to our present purpose, i.e. the parts that relate directly to the Mainflux bootstrap server. So, in order to read a .toml configuration file, we need to define a Go struct and store in it the data we read from config.toml:

type Config struct {
 URL         string
 ExternalID  string
 ExternalKey string
}
func loadConfig() (Config, error) {
 tomlConfig, err := toml.LoadFile("config.toml")
 if err != nil {
  fmt.Println("Error ", err.Error())
  return Config{}, err
 } else {
  return Config{
   URL:         tomlConfig.Get("server.URL").(string),
   ExternalID:  tomlConfig.Get("thing.external_ID").(string),
   ExternalKey: tomlConfig.Get("thing.external_key").(string),
  }, nil
 }
}

After that, we need to define the Go struct that will hold un unmarshalled JSON data that we will fetch from the Mainflux bootstrap server and send an HTTP GET request to fetch the data:

type MainfluxData struct {
 MainfluxID    string    `json:"mainflux_id"`
 MainfluxKey   string    `json:"mainflux_key"`
 MainfluxChans []Channel `json:"mainflux_channels"`
 Content       string    `json:"content"`
}
func getConfig(cfg Config) *MainfluxData {
 url := cfg.URL + cfg.ExternalID
 req, err := http.NewRequest("GET", url, nil)
 if err != nil {
  log.Fatal("Error reading request. ", err)
 }
 req.Header.Set("Authorization", cfg.ExternalKey)
 client := &http.Client{Timeout: time.Second * 10}
 resp, err := client.Do(req)
 if err != nil {
  log.Fatal("Error reading response. ", err)
 }
 defer resp.Body.Close()
 fmt.Println("response Status:", resp.Status)
 body, err := ioutil.ReadAll(resp.Body)
 if err != nil {
  log.Fatal("Error reading body. ", err)
 }
 var mfData MainfluxData
 json.Unmarshal([]byte(body), &mfData)
 return &mfData
}

Once we have fetched our data from the server in the form of JSON and unmarshalled it in the Go struct, we can connect to the MQTT broker to send and receive messages:

func connectToMQTTBroker(mqttURL, thingID, thingKey string) mqtt.Client {
 opts := mqtt.NewClientOptions()
 opts.AddBroker(mqttURL)
 opts.SetClientID("gateway")
 opts.SetUsername(thingID)
 opts.SetPassword(thingKey)
 opts.SetCleanSession(true)
 opts.SetDefaultPublishHandler(func(client mqtt.Client, msg mqtt.Message) {
  choke <- [3]string{msg.Topic(), string(msg.Payload())}
 })
 client := mqtt.NewClient(opts)
 if token := client.Connect(); token.Wait() && token.Error() != nil {
  log.Fatalf("Failed to connect to MQTT broker: %s", token.Error())
 }
 return client
}

All that is left is to send and receive some MQTT messages from our app. Please consult the Github repository to see how we do it and for the other code details. Once we have our app, we can simply execute, inside our app directory, the following command:

go run main.go

If all went well, you should get an output similar to the one that I got:

response Status: 200 OK 
{MainfluxID:6ef63b46-9262-4355-aca3-2c84eb5b5085 MainfluxKey:8171bba6-ad71-4a3e-b885-55c283736974 MainfluxChans:[{ID:a225801d-0a0e-4b7a-9860-619bf943fe54 Name:c1} {ID:77d9ac08-901c-40ca-a69a-5d4f9dcc7233 Name:c
0}] Content:gateway} 
---- doing publish ---- 
---- doing publish ---- 
---- doing publish ---- 
RECEIVED TOPIC: channels/a225801d-0a0e-4b7a-9860-619bf943fe54/messages MESSAGE: [{"bn":"some-base-name:","bt":1.276020076001e+09, "bu":"A","bver":5, "n":"voltage","u":"V","v":120.1}, {"n":"current","t":-5,"v":1
.2}, {"n":"current","t":-4,"v":1.3}] 
RECEIVED TOPIC: channels/a225801d-0a0e-4b7a-9860-619bf943fe54/messages MESSAGE: [{"bn":"some-base-name:","bt":1.276020076001e+09, "bu":"A","bver":5, "n":"voltage","u":"V","v":120.1}, {"n":"current","t":-5,"v":1
.2}, {"n":"current","t":-4,"v":1.3}] 
RECEIVED TOPIC: channels/a225801d-0a0e-4b7a-9860-619bf943fe54/messages MESSAGE: [{"bn":"some-base-name:","bt":1.276020076001e+09, "bu":"A","bver":5, "n":"voltage","u":"V","v":120.1}, {"n":"current","t":-5,"v":1
.2}, {"n":"current","t":-4,"v":1.3}]

We have fetched the configuration and got “response Status: 200 OK” and we displayed the unmarshalled configuration. Then our app sent and received 3 mqtt messages using the config information that we have fetched from the Mainflux bootstrap server.

As I have said, a real world application would rather — after parsing the server response body, i.e. after unmarshalling of server response JSON — do all the necessary device configuration accordingly and lauch a deamon process. It should probably also create another configuration file to store the config details such as mainflux_id, mainflux_key and mainflux_channels (see type MainfluxData struct above). Once again, we should make a difference between mainflux_id and external_id, as well as between mainflux_key and external_key. The mainflux_ values are related to the way that Mainflux as a platform internally represents a thing such as a physical device. On the other hand, external_ values are stored on the physical device itself — we stored them in a .toml file above — and are matched with the Mainflux representation of a thing by the Mainflux bootstrap service, which knows both pieces of information.

Conclusion

Mainflux is an incredibly flexible IoT platform thanks to the fact that it abstracts away from the implementation details of software applications and construction details of hardware devices. It offers, thus, a possibility to connect any kind of device to any kind of application and to ensure a seamless communication bridge between them. Mainflux bootstrap service, on the other hand, allows a device manufacturer to abstract away from the details of the device usage and configuration. All that is requeired from a device manufacturer is to supply three pieces of informations — server URL, device id and key — such as those we entered into our configuration file above (config.toml). Used together, Mainflux platform and Mainflux bootstrap service give performant, scalable, secure and flexible context for the development for full stack IoT solutions.

If you need any help with using Mainflux or you are just curious about Mainflux, you can always ask question on the gitter channel. The Mainflux documentation is a good source of information although it can be a bit outdated because of the fast rhythm of development — the current version of Mainflux is 0.8 and it is getting towards a 1.0 release in summer 2019. Finally, the “single source of truth” of Mainflux platform can be found on its Github repository.