pymesos, doing mesos frameworks with python (IV)

Yeah, there’s a lot of debates about which is the right programming language for doing data science, python, R, java, ¿scala?…

And to be honest, it’s impossible to select one for all the tasks you have to solve in this new world. What SDKs are available in the different tools? is there a common language among all of them?… NO,

You will have to learn some of them at the end, and python maybe a good choice to begin with . At least if you are going to do science based on deep learning and in the case of rainforcement learning too. A lot of frameworks justifies: tensorflow, scikit, keras, theano… and RAY (do you remember?, it’s the new initiative from Berkeley to do deep reinforcement learning).

(warning: if you begin with python, sure you will have to learn c++, i’m afraid)

Let’s play with python then, and with Mesos, and so it comes pymesos, a python library to build Mesos frameworks. You can find it here:

In the last versions it uses the Mesos HTTP APIs for both Schedulers and Executors, so you have to worry only about the lifecycle of your tasks. MesosSchedulerDriver and MesosExecutorDriver are the classes where you can find all the stuff related to maintain an opened stream with Mesos Master (or agent in the case of Executors) and the low-level management of both Calls and Events.

At the same time you can find both interfaces Scheduler and Executor that define the methods to be implemented in your particular implementations.

The MinimalScheduler/MinimalExecutor is a simple example that lets you understand better all these things. It has implementations for both Scheduler and Executor (this is a bit confusing because both the Executor and the “Task” are implemented in the same object).

In the Scheduler you can find an example of how to build the right structure for the Task definition. It’s important to review this code deeply because it shows how to send info properly formatted to Mesos:

Minimal definition of tasks in pymesos

There is a main method where you define other two important structures, executor and framework:

Minimal Executor and Framework definitions

Finally the last piece of software that you have to look at is that related with the offers management where you say to Mesos how and how many tasks you want to launch.

In the next chapter I’ll explain what we believe is a “MinimalFramework”, some features:

  • Use default executor from Mesos.
  • Use universal containerizer with docker images, for both tasks and the scheduler itself.
  • Basic HA features for scheduler: persist state in redis (to begin to allign with Ray) for both scheduler id and tasks states.
  • Reconcile tasks when the scheduler is reregistered.
  • Use Task/Job definitions from tfmesos to build a more general framework.
  • API to receive from metronome new jobs to be executed with the framework. We use redis pubsub to queue these new jobs.