New way to deploy services

In SOA or microserivce world, deploying an application or a platform usually means to deploy a number of long-term running services. Each of them will typical listen on one or more TCP/UDP ports. Nowadays, a system may have dozens of related services running, and the deployment of each service may happen many times a day. How to deploy the services efficiently become much more important than ever.


The old way to support deployment is to build a rpm or deb package, and run yum or apt to install the packages. In order to support service start/stop/restart, usually the rpm/deb package include the init or systemd. This approach seems pretty straightforward, and works will for a single services. But it comes with the cost:

  • It is hard to make the init script 100% correct. The scripts involves many small pieces like defining bootstrap order, running application in background, saving the pid, gracefully stopping and restarting the service. Searching “init template” in StackOverflow, you might get different answers of how to implement that. Most of them are problematic. Systemd is much easier to use, but still needs extra work.
  • More important, dependencies might be complicated. One service might depend on other libraries. For example, it may require specific version of python/ruby which not what OS provided. One service may also depends on other services. To handle these dependencies can be challenging.

Too Big and Too Small

The fundamental limitation of rpm/deb and init/systemd is that it views the one OS as a single box. On the one hand, the single box is too big. It is hard to isolate different libraries and services in a single OS. On the other hand, the concept of this single box is too small. We are now working on distributed system, which involving maybe thousands of boxes. We really want a more high level concept of resource pool instead.

Marathon + Docker

The best solution at this point, in my opinion, is using combination of marathon and docker.

Docker is much smaller box. It resolves the dependency issue, by packaging the “environment” into a single image. Each docker image is an isolated container, with all dependencies packaged together. It can be run the top of any linux box in an efficient way. It can completely replace the tools like rpm or deb for service packaging.

Marathon, on the other hand, is the init/systemd in the distributed world. I provide some demos of it before. Compared with init scripts, it is much easy to use and less error-prone.

  • Developers are familiar with run application in a foreground mode, with log send to stdout, and use Ctrl+C to terminate the process. Using marathon, there is no need to change that. No extra script needed.
  • It is deadly easy to locate the service logs using mesos console. It is also not hard to collect the logs into a centralized view using tools like logstash.

Marathon is not just alternative, but provide many advanced features.

  • Advanced deployment. Marathon provide an easy way to describe the relationship among different services, and it will handle the services start/stop in a logic order. Moreover, you can configure minimumHealthCapacity when upgrading the services, to achieve zero-down time.
  • Resource management. User have a fine-grain control of resource allocation for each service. The services instance and resource allocation can be dynamically changed. For example, we can bright up more app instances when the traffic volume boots.
  • Resilience. Marathon has health check built-in. If services fails (e.g. a node crashes), it will be restart (e.g. on another node) if resources permits.

Since 0.7, marathon natively support docker. Combining these two tools can deploy the service much faster. There is an example of using marathon and docker to automate the deployment.

Further Discussion

Marathon + docker solution fits well to deploy stateless services in a large-scale. Most of these services may depends on external stateful parts, like MySQL, Redis, etc. There are many solution for deploying these parts and describe these dependencies, like Docker compose, DOCS, etc.

If you are not providing services, but the software, it still make sense to generate rpm and deb package so that users can easily install them on target machines.

Like what you read? Give Tongqing Qiu a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.