A Data Layer for Edge/IoT and Fog Computing

Introduction of modern, cloud-era clustered data layer “EdgeFS” to solve complex issues of Edge and Fog Computing

We’ve talked a lot lately about the ways of transforming cloud data-intensive workloads in terms of speed and efficiency. It can be achieved as long as dataset located fairly close to the application. By fairly close I meant networking latency.

This is a legacy, centralized design the problems of which in modern cloud-era we going to discuss in this article. Such legacy design assumes that all the data is somehow “cloud-born”, and somewhat magically appears in the cloud in the application hard-coded location.

This has worked when application operates with some few gigabytes of data and/or a few millions of records per day delivered. And to optimize further, we employ asynchronous data copy or some off-the-shelf persistent message queuing technique.

This has worked while cloud connecting network is “reasonably fast” thus an application can upload few megabytes within seconds. We will batch it and then copy it over in terms of to overcome latency problems. While not always possible, sometimes we could even modify our application and send data in batches using multithreaded parallel upload.

So, we are done, all works great, yeah?

Not even close. It reminds me wonders of FIDO-Net with batched packet messaging over ADSL modem…