Distributed Web with IOTA: a technical overview
IOTA is the distributed ledger for the Internet of Things. Here at Things Lab we try to find out solutions based on this technology in order to enable the rise of IoT devices with a secure infrastructure.
In this post I am not going to talk about how we could use IOTA to build the internet of Things but whether we can use IOTA to build a distributed Internet.
First we need to explain what we mean by distributed internet, then I am going to do a fast dive in IOTA transaction architecture and IRI node to show how we could use them to serve a distributed internet. Later I will share some code and solutions on how this could be managed by reporting some benchmarks and finally the drawbacks this approach would lead to.
Distributed Internet
In our minds the concept of distributed internet takes shape as a way to retrieve a web content not taking into account who serves it.
This concept applied to IOTA starts from the premise that we are able to serve a web page querying for a transaction. In particular, we could store a web page in a transactions bundle (I will specify the structure later), and serve it extracting the data from the bundle. We are talking about a transaction published to the IOTA network, hence a distributed piece of data. Starting from the assumption of the availability of this distributed data, and I am going to show how to store them, we need a distributed way to serve them.
We base our concept on available technologies. We don’t want to replace the HTTP protocol or anything like this. We should be able to use a simple open link web browser to ask for a domain name and receiving back a web page content.
Currently, to retrieve a transaction from the Tangle, we have to go through the IRI node, using the JSON RPC interface via the command line or using one of the API implementations. This operation implies the need of a web server able to listen for the request, where reasonably it should find the bundle hash, and retrieve the transaction through an HTTP request, extracting the content and sending it to the client specifying the Content-Type as text/html. This would work, but this is obviously not a distributed way to serve the web page.
Now let’s dive in the technical aspects to understand how this mechanism could be built.
IOTA Transactions
From the IOTA docs:
A transaction is a single instruction to credit IOTA tokens, debit IOTA tokens, or send a message. To transfer IOTA tokens, you need both input and outputs transactions, which are packaged together in a bundle.
Hence transactions are always packed in a bundle. I won’t paste here the entire page of the documentation (you can find the link in references below). What we care regarding transaction structure itself is that an input transaction contains a signatureMessageFragment, that is where data and the signature are stored. The signature length is based on the security level of the address and a security level higher or equal to 2 produces a signature too large to fit inside the signatureMessageFragment field. Hence commonly we will have more than an input transaction in a bundle, in consequence of the length of the signature.
Although is recommended a maximum of 30 input/output transactions, there are no limits about the number of transactions, so we can exploit the signatureMessageFragment to store data we are interested in. In our case we can store our web page in this field. Therefore if we want to retrieve and render the web page we have to extract this field from each input transaction, concatenate them and extract the data slice (remember that this will be stored as a string of trytes, but we have the utils function trytesToAscii to overcome this).
Take into consideration an easy webpage with a dozen of html tags and some dependencies. This web page after being minified a uglified will weight around 250kB, that means around 230 transactions, considering that an input transaction can contain a message of 1.5kB length. At first glance, one would question how long would it take to attach 250 transactions, or even execute the PoW for each of them.
Well, luckily we used a little FPGA hardware accelerator module using the Pidiver library (thanks to Thomas for supporting us, link in references) and we were able to attach the bundle to the Tangle in around 1 minute. Precise benchmarks in the last paragraph.
This is the code we used to accomplish it. As you can see is the same code of the Pidiver tutorial but I read the message from the file example.html:
IRI Node
As previously stated we would need a distributed way to serve the web page that without the fetching of a bundle through a web server able to retrieve, parse and return it.
One of the last features of IOTA are the IXI modules. IXI modules enable everyone to enhance the IRI’s exposed API adding custom commands. In particular it is possible to create a function (using Java o Javascript) and insert it in the node folder in order to make the node execute it when the specified endpoint is queried. We start from the assumption that IRI should be as fast as our web server to retrieve and return the transaction. In this way we could have a set of nodes able to receive a call passing a bundle hash and returning back the content of required transaction.
As you can guess this would be an important step to serve web contents in a distributed way, in particular if these functions were to be a default function in the IRI implementation. The last step we miss here is a distributed DNS able to redirect the request to one of the nodes that expose this function. Such services already exist and this functionality can be easily achieved using a failover or a load balancing strategy.
At this point we have traced the line to be able to realize what we have in mind, but now we face a big limitation in the IRI node implementation. Currently it is only possible to reach one of the endpoints exposed from the node through an HTTP POST request, specifying the command in the body of the request. Obviously this is a big limitation because web browsers execute a GET request when they need to retrieve a web page through a certain URI.
This limitation disrupts our plans and we plan to submit a pull request to the IRI node repository to enable this feature, e.g. expose some commands via a plain GET request, just based on the URL.
In this moment we are not able to overcome this limitation hence our tests are based on a centralized approach on the backend, using a classic client server architecture, but we are confident that this new features would produce similar results. Our tests are based on a NGINX web proxy backed by an Express NodeJs web server.
We registered the iotaweb.site domain in such a way to redirect the request to our web proxy. The temporary replacement to simulate a GET request directly to the node exploits a simple condition in the Nginx configuration file (line 20):
As you can see the request is redirected to out express web server where the request in handled like this:
You can see that the content is returned setting the Content-Type header to text/html to inform the browser that we are returning a web page and that he should render it. In the business logic the getTx method just fetches the bundle based on passed id and parses the result as previously explained:
In the last technical part of this article I am going to show you the results we obtained.
Technical Results
These are some results we obtained trying to build what explained so far. Consider that to load a web page on a single transaction (or across a group of them in a bundle) we need to have the entire page in that transaction to make it work. There is no way to import other files as we are used to do because the endpoint we created is only able to receive a get request containing a bundle hash and retrieve the bundle from the Tangle based on that hash. Thus if after we render the page we have a script tag trying to fetch a main.js file from the server, this won’t be possible. Therefore we need to use some instruments to be able to load everything we need inside the transaction. In order to do this first I used browserify to be able use npm module a create a minified js bundle. Then I used Inliner to assemble everything in a single html file. I think that currently this is the only way to load a web page because it is not possible to import other files from the server, but just from other resources as CDNs. Obviously load dependencies from a CDN increases the centralization of the solution.
I am starting from the writing operation described in Transaction section. With a file of 256,7kB, we needed exactly 235 transactions to store the file, and it took 01:38.57 minute with a medium hash rate of 15 MH/s. Therefore we are talking about reasonable time to store a web page in a decentralized way. If you consider the operation I explained before, e.g. minification and assembling in an automated process, the time increases just in a matter of seconds. The time grows more or less linearly with the number of transactions you have to write.
The second operation is instead the fetching of the web page. In this case with a direct curl request to the server would take less than 2 seconds to retrieve the transactions and return the data. Just a little bit more on the web browser (milliseconds), the time it takes to resolve the DNS and then send the response back. Try it yourself:
https://iotaweb.site/DIV9JKJSTZHLREEZPIPVLRIKAQNXOIBCKDUDEKCVKECLABQCXUFSNKEQXMKDIVYKSKUBXYEKEHNJAYPNDThis is an example of a web page fetched from the transaction with bundle id DIV9JKJSTZHLREEZPIPVLRIKAQNXOIBCKDUDEKCVKECLABQCXUFSNKEQXMKDIVYKSKUBXYEKEHNJAYPND (in this case the page is just the text Hello Tangle). The query based on findTransactionObjects you have seen in the snippet code of business logic retrieves all transactions based on the bundle hash, thereafter is reasonable to think that the time of this operation grows less than linearly with the the number of transaction in the bundle, because there is just one request to the Tangle to fetch a precise data structure, the bundle. I would have expected a more linear growth if we would have passed an array of transaction ids, where each transaction is not related to the others. This operation would have required the same search operation repeated many times based on how many ids we passed. I expect that the operation should be easier with related transactions in the bundle.
Conclusions
In this article we proposed a IOTA based approach to serve web pages in a distributed way. We will continue to experience this solution looking for optimized way to achieve this result. The first optimization could be a compression of the web page in storing phase. This could save a lot of space whether the browser is then able to automatically decompress and render the web page.
The solutions we proposed seems to be feasible for a set of applications where it is important to guarantee that a website contains some verified information or information that are not manipulated (read to our other articles for smart cards and tracking applications). As always we face some trade offs when it comes to distributed and decentralized solutions, the first is regarding the performances. In this case, based on a series of assumptions will be verified, we saw that it is possible to obtain feasible performances regards to the user experience. In applications where the features I previously mentioned are critical the user can wait some seconds to fetch the page. The deploy phase is not related with the user experience, and the cost of an FPGA hardware accelerator is something we are willing to pay the have a free permanently stored web page.
I didn’t talk about Qubic, the next big release of the IOTA foundation, but that feature could open the way to more distributed and complex applications based on this approach. We will keep to post some articles about our experiments.
Best Regards,
Patrick
