iEx.ec virtual machine management

To continue our series of articles introducing the technology behind the iEx.ec Blockchain-based Distributed Cloud, this article presents the iExec Virtual Machine management. One can refer to previous articles on medium.com.

Introduction

As described in a previous article, iExec belongs to the well known paradigm of global computing platforms that implement a three tier architecture where a set of centralized services are instantiated and reachable via Internet connection.

This series of articles aims to be as concise and didactic as possible, introducing paradigms step by step. So far, we have introduced technology in terms one may already be familiar with when discussing global computing: decentralized services, pull mode and peer to peer (P2P) communications.

But our technology go further by proposing what we call “provider’s sharing”. This article presents this so called “provider’s sharing” paradigm as well as its first use case: virtual machine management.

Architecture

The architecture shown in previous articles has not yet introduced all the tips and tricks to facilitate the understanding of our platform. The next figure digs more deeply into detail by introducing the provider’s sharing services and protocols. One can certainly retrieve what has already been introduced in previous articles where iEx.ec services were shown such as the scheduler, data repository, client and worker. On the other hand, one may certainly notice the “provider’s sharing” icon on the provided resource side -the “worker”. This does not break any global computing rules, but introduces a feature where provided resources can propose more than CPU and RAM “only”. With this provider’s sharing, any resource can propose some electronic assets it owns on its side.

Traditionally, global computing schedulers use “matchmaking” algorithms where decentralized resources characteristics, such as CPU, RAM, OS etc., are used to determine what to send and where. For example a job referring to an application with a Linux binary can only be sent to a resource running the Linux operating system.

With our provider’s sharing paradigm we propose to do more. A distributed resource can present any digital asset (an application, a library, a data, a GPU) and the scheduler is able to use this information to compute the matchmaking. This article focuses on provider’s application sharing that permits the management of virtual machines over a set of decentralized resources.

Security

This new paradigm -provider’s sharing- does not break security as introduced in previous articles in any manner therefore:

• provider’s sharings must be registered on iEx.ec services.
• authorizations and access rights apply to provider’s sharings (please refer to “iExec security” article).

Workflow

Provider’s sharing permits a new workflow, slightly but not entirely different from the one presented in a previous article. Differences are depicted in bold and underlined:

• The XWHEP services permit the administrator to manage registered applications.
• Client prepares data that is needed to successfully compute jobs. This data may be stored in the iEx.ec infrastructure or in any repository as soon as the data can be described by a URI and is accessible through the network. There is no limit on data size, but data exceeding a few hundred mega-bytes may take very long to download or upload. Thanks to provider’s sharing, the data metadata may define no content and specify that this data must be proposed by decentralized resources that own data content.
• An authorized Client registers the application on the iEx.ec infrastructure. Thanks to the provider’s sharing, the application metadata may define neither binary nor library, and specify that this application must be proposed by decentralized resources that own the application.
• A Client prepares jobs (units of work) containing the reference of a registered application, optional parameters, optional references to additional files.
• Finally, the Client submits the prepared jobs to the scheduler.
• Independently, workers contact the scheduler to get jobs suitable for their architecture and, eventually, their sharings.
• In response, the scheduler sends a suitable job description to a worker.
• For each file referenced by the job which is not present in the local cache of the worker, the worker fetches the file from a Data Repository or an external data repository. The worker don’t need to download and install any data, library or application it shares.
• As soon as a job has finished on worker side, the worker contacts the Result Collector to send the results.

Use case

The provider’s application sharing is our first use case. This permits virtual machine management. It necessitates registering a virtual machine as application, deployed resources sharing an hypervisor and finally submitting a job to launch the virtual machine. To do so, users must:

1. retrieve the registered virtual machine. Virtual machines must be registered as application on the scheduler.
2. submit a job to launch the virtual machine, eventually providing its SSH public key and requesting a port to connect to
3. the job is scheduled to a resource sharing an hypervisor
4. if a port was requested, the job automatically obtains a virtual socket the SSH daemon will listen to (please see “iEx.ec network virtualization”).
5. if a port was requested, the job is automatically connected to a local SmartSocket proxy in charge of interconnecting virtual address to local IP one, on worker side
6. if a port was requested, retrieve the job SmartSocket virtual address
7. if a port was requested, launch a SmartSocket proxy for the job, providing a local port, on user local machine where iEx.ec client is installed
8. the proxy connects to the job virtual address
9. the proxy listens to the local port
10. the proxy forwards all messages between local network and the job virtual address
11. the user can use ssh client as usual to connect to his running virtual machine, using its local secret key corresponding to the public key provided when submitting the job. The user must not try to connect directly to its running virtual machine, but to its local Smartsocket proxy

Conclusion

This article introduced the iEx.ec provider’s sharing that permits the management of Virtual Machines over a decentralized cloud. In next articles, we will discuss other provider’s sharings such as data sharing that permits data driven scheduling, or GPU sharing that allows GPU applications such as machine learning ones.

Media

You are kindly invited to:

• join our Slack: http://slack.iex.ec
• follow us on Twitter: https://twitter.com/iEx_ec
• interact with us on Reddit : https://www.reddit.com/r/iexec/

Resources

• iEx.ec Github
• XtremWeb-HEP Github
• XWHEP documentation
• Video introducing XWHEPI presented at CrowComputing 2014, in Almere, NL. https://vimeo.com/113122296