Distributed Data Logger
Advanced MAM explorer for Industry 4.0 applications
Distributed Data Logger (DDL) was created with the purpose of extending and boosting the well-known Masked Authenticated Messaging (MAM) second IOTA layer, allowing a MAM channel reader to access additional features.
More specifically a MAM channel has these properties:
- it can be read only starting from a known root id (the unique message identification number) up to the last message in the channel;
- it can be read only sequentially, because the fetch function, given a root id, returns one message, a requested number of messages, or all messages in the channel starting from that root;
- there is no way to know in advance how many messages are stored in a MAM channel from a given root id, without reading all the channel up to the last messages.
These properties are not reducing MAM’s usefulness, but in some cases they are inconvenient.
For example both MAM writer and receiver, communicating on a channel for a long time, need to store the root id of the last message exchanged safely, in order to be able to start again from that point if the system needs to be restarted losing the state.
In doing this, they lose the opportunity to retrieve all the messages older than the last one they have managed.
To solve this issue, the first root id of the channel can be stored too, allowing the parties to start from the end or from the beginning to retrieve some data from the channel.
But what happens if the channel contains thousands of messages?
The transmitter is able to skip all the channel’s content and restart from the last root id, but the receiver, who needs to retrieve a specific message in a time frame, needs to process all the channel from the beginning with a lot of effort.
We found it really unpractical, and that’s why we started searching for a way to not only speed up the search, but also filter it: this is basically how our Distributed Data Logger came to life.
Features
Distributed Data Logger looks like generic public MAM decoders and can actually decode a generic MAM channel like any other. If the messages in the channel (the MAM message payload) have a specific format however, DDL is able to provide some useful additional features.
The UI is pretty simple and intuitive. Distributed Data Logger lets you choose a provider (otherwise a default one will be used) and then asks you for the entity root (first root of your MAM).
Once you copy/paste the entity root id (the root id of a MAM channel produced by an entity) you only need to press the Tab and the tool will automatically check if your MAM has the right characteristics to unlock the advanced features and filters.
If not, you’ll still be able to use the standard fetch which is similar to the one of the MAM decoder
How to enable the advanced features?
The secret behind the boost is a simple JSON structure.
Every message published on the MAM will contain a field “count”, which basically represents its position in the MAM. Besides, starting from the first message, a field “prevRoot10” and “root10” will be added every 10 messages and will respectively contain the root of the 10th previous message and the root of the 10th following message.
Every 100 messages a “root100” field will be added (together with the two described above). Same goes for every 1000 messages with a field “root1000” containing the 1000th following root.
Example:
Thanks to this approach and to specific algorithms it is possible to literally jump from one message to the other to get to the desired one in a considerably small amount of time.
In addition, the overload in term of message size is around 100 tryte every 10 messages, which is not a critical amount.
When messages contain the link to the other root in the channel, DDL is able to quickly find the last messages in the channel providing the channel dimension (in term of number of messages contained). Then it becomes simple to move backwards in the past messages or to ask DDL to retrieve messages from a specific sequence number.
Anyway the most interesting feature is the possibility to ask DDL to retrieve messages in a specific timeframe.
DDL is very fast in rolling such structured MAM channels thanks to a sophisticated implementation (dichotomic search for example).
Last but not least, DDL makes it possible to activate a “listen” feature to automatically retrieve messages from a channel in near real time
To try DDL, you can use this root id. The channel contains more than 7000 messages.
Root id: “APBFWNCJWDCPINZDOXDJDXXQVLZARSOXCGL9QHDWSOCOPOODKDYZRGVFMMK9VPZKQSYKHXZZJMPZOBHTV”
Use cases
DDL can be useful in many cases where data needs to be distributed from a source to one or more receivers that would otherwise need to navigate the channel.
A typical case is an environmental sensor that publishes data which needs to be verified only in some specific conditions.
Another interesting use case is common in industrial environments. Industrial equipments are normally controlled by PCs that collect events from sensors to activate an actuator: these PCs keep their work history on local log files, and in case of trouble engineers need to remotely access those logs to identify the issue. This leads to two common situations:
- remote devices logs are accessible via internet (hopefully through a VPN);
- remote devices are not accessible in any way from outside the local network due to security restrictions.
In the first case engineers are allowed to access the remote device and solve the issue, but at the same time the whole remote network is exposed to unauthorized operations by the authorized engineers or someone else with their access credentials.
The second case appears to be more secure, but after engineers reach the device locally, they are again able to access the whole local network.
Hopefully the engineers are honest, but the risk of a fraudulent action on a critical industrial network can not be underestimated.
The simple solution to both these risky cases is to let the remote device write its logs on a DDL: engineers can access the logs in any moment without the need to access the remote network directly.
Try the beta!
The public beta version of DDL can be found here.
Keep in mind that, as a beta, it will most likely have many and even severe bugs, any feedback is very welcome!
Also, we are actively working on it and making improvements, so expect downtimes or slowness.
Conclusion
DDL is a great tool to improve IOTA’s standard MAM implementation, implemented only by adding a JSON structured information in the message payload, which remains perfectly compatible with any MAM reader and with standard libraries.
Here you can find the sample code to write the messages in the JSON format described above.
We are obviously available to provide countless additional features, services and customization, just to name a few:
- customized DDL front-end;
- possibility to encrypt MAM messages with a side key to grant total privacy of the messages payload;
- adding an entity directory to allow to search the logs of many entities using an easier entity naming than the root id;
- fast IOTA node able to attach to tangle hundreds of MAM messages per second;
- private IOTA network for the users who need a totally controlled environment.
