Understanding the Technology Behind the iEx.ec Distributed Cloud
The key idea of iExec is to build a Blockchain-based fully distributed cloud infrastructure (see the previous posts introducing the project here).
To this end, iEx.ec combines two technologies : Blockchain and Desktop Grid Computing. Blockchain allows to build decentralized market place for computing resources. The principle of Desktop Grid (aka Volunteer Computing) is to collect the computing resources underutilized on the Internet to execute very large parallel applications at the fraction of the cost of a traditional supercomputer. You may already know or even participate to several applications that follow this computing paradigm : SETI@Home, Folding@home, disributed.net.
But do you know that in the past decade, many middleware have been developped that implement this model: BOINC, Condor, Ourgrid, XtremWeb-HEP; to name a few of them ?
Indeed, Desktop Grid middleware have several desired features to build a fully Distributed Cloud:
- resilience: if some nodes fail, the computation continues on other working nodes.
- efficiency: applications get excellent performances even if computing nodes are highly heterogeneous.
- ease of deployment: allows to use any nodes without specific configuration, even those located on the edge of the Internet.
Thus, contrary to what its name suggests, Desktop Grid is the perfect solution for assembling hybrid infrastructures, whose computing resources can range from classical HPC cluster, to the Cloud, and of course Internet PCs.
iEx.ec relies on Desktop Grid open-source software that have been developped by CNRS and INRIA to assemble loosely distributed computing resources for HPC and Big Data.
- XtremWeb-HEP is a mature, solid, and open-source Desktop Grid software that allows to use any kind of computing resources for executing compute-intensive or data-intensive applications. The architecture of XtremWeb-HEP is three roles: clients, which submit tasks and data, the dispatcher, which schedules the tasks to the computing resources, and workers, which executes the tasks and send the results back to the clients. Because of its simplicity, this approach has proven to be extremely scalable, reliable and secure.
- Nevertheless, XtremWeb-HEP provides many advanced features such as: fault-tolerance, multi-applications, multi-users, hybrid public/private infrastructure, deployment of virtual images, data management, security and accountability, overlay network, VM management, sandboxing, support for HPC standards, interoperability with eScience infrastructure, and many more.
- XtremWeb-HEP is developped and used in production at the French National Institute for Nuclear and Particle Physics. You can visit its homepage here at XtremWeb-HEP, or its page on GitHub.
In addition to XtremWeb-HEP, we have developed during the last decade a large portfolio of technologies that allows to assemble loosely distributed computing resources for HPC and Big Data: MPICH-V for parallel computing, BitDew for large scale data management, SpeQuloS for providing quality of service to application execution, the first implementation of MapReduce for Internet computing, and more. Many of our research results have been published in top scientific conferences and journals (>80 papers published).
Eventually, we have obtained a unique expertise in making the Desktop Grid technology running and available to various scientific communities as well as start-ups and innovative industries.
- From 2007 to 2012, with several European partners, we established the European Desktop Grid Infrastructure (EDGI). This considerable effort was supported by the European Union, which funded 4 FP7 projects (EDGeS, EDGI, DEGISCO, IDGF). The goal was to provide researchers and academics with an additional computing power coming from Desktop Grid infrastructures. EDGI has been a huge success. We connected a dozen of sites (Hungary, France, UK, Spain, Nederlands …) to the main European e-infrastructures, such as the one supporting the Large Hadron Collider in Switzerland. We gained a considerable experience in connecting Clouds and HPC systems to Desktop Grids as we succeeded in transparently executing millions of jobs on more than 200.000 nodes.
- By showing the applicability of the technology in many fields of science: high energy physics, biomedical research, mathematics, finance algorithm, material research, 3-D rendering, and more.
- Of course, we had also many collaborations with the industry (Total, Airbus, IFP, …). And more recently, we received a funding from the French National Research Industry in order to provide innovative SMEs access to low-cost, on-demand and secure HPC services. We conducted many interviews with SMEs from the bio-medical and eHealth sector to understand their needs and requirements and designed MVPs and PoCs. Somehow, the emergence of blockchain was the key enabler that eventualy lead to the iEx.ec project.
As you can understand now, we strongly believe that the knowledge and experience we gained by exploring, inventing and establishing Internet-wide distributed computing infrastructures is necessary to lead successfuly the iEx.ec project.
If you want to learn more Desktop Grid, I can recommand the Desktop Grid Computing book, co-edited with Christophe Cérin. Each chapter is written by the most prominent experts of the domain and presents the most advanced results in this area. The book covers many aspects, including practical chapters: security, applications, toolkits, software, algorithms, etc. If you want to know more about my research, feel free to have a look at my homepage, and why not in my Habilitation thesis.
