Notes on Apache Mesos Setup

Recently, I work on building a new data ingestion pipelines. I need to ingest data from kinesis and dump them on S3. Since I am familiar with flink and parquet, I decide to just use them. But before I can write and run some flink jobs, I need a cluster. I really like docker and I have lots of data framework related docker files like flink, spark, presto. So I am going to setup a mesos cluster which I can use it to play with my data frameworks :)

I don’t think I have much experience on setting up production ready mesos cluster. But I just want to take a note on what I went through. And hope this can help anyone wants to try it. Everything here

Enviroment

I am running all my services on top AWS.

instance type, m4.2xlarge

os: ubuntu 16.04

Install Mesos

# In case the host name is not the ip

cat <<EOF >> /etc/hosts
127.0.0.1 `hostname`
EOF

According to Mesosphere’s blog, we are going to install mesos and marathon. Make sure to install java 8 first. If you don’t install java 8 first, after you run sudo apt-get -y install mesos marathon chronos, it will pull java 9 automatically. Unfortunately, I have trouble to start mesos/zookeeper with java 9.

# Setup
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
CODENAME=$(lsb_release -cs)
# Add the repository
echo "deb http://repos.mesosphere.com/${DISTRO} ${CODENAME} main" | 
  sudo tee /etc/apt/sources.list.d/mesosphere.list
sudo apt-get -y update
# Install Java 8. Make sure it is java 8, not java 9
sudo apt-get -y install openjdk-8-jre
# Install packages
# this is the exact version I installed
# mesos=1.6.0-2.0.4 
# marathon=1.6.352 
# chronos=2.5.1-0.1.20171211074431.ubuntu1604
# you don't needto install chronos, but marathon is very useful
sudo apt-get -y install mesos marathon chronos

Configure Mesos slave

Now we want to configure the mesos slave. Run the following command under root

sudo su -
cat <<EOF > /etc/mesos-slave/containerizers
docker,mesos
EOF
cat <<EOF > /etc/mesos-slave/image_providers
docker
EOF
cat <<EOF > /etc/mesos-slave/isolation
filesystem/linux,docker/runtime
EOF
cat <<EOF > /etc/mesos-slave/executor_environment_variables
{
  "JAVA_HOME": "/usr/lib/jvm/java-8-openjdk-amd64"
}
EOF
#remove the default config. Assume the zookeeper ip is 10.10.10.10
cat <<EOF > /etc/mesos/zk
zk://10.10.10.10:2181/mesos
EOF
#remove the default config
cat <<EOF > /etc/default/mesos-slave
MASTER=`cat /etc/mesos/zk`
EOF
# By default, the ports offered by mesos are [31000-32000] 
echo "ports:[5500-60000]" | sudo tee /etc/mesos-slave/resources
# Configure the slave's ip
echo 10.10.10.14 | sudo tee /etc/mesos-slave/ip
cp /etc/mesos-slave/ip /etc/mesos-slave/hostname
echo manual | tee /etc/init/zookeeper.override
echo manual | tee /etc/init/mesos-master.override

Here we configured the ports resource. Because I hardcode some ports in my dockers. For example, presto’s coordinator, I configured 9090; flink’s blobserver, I configured 50100. So the default range is too small for me.

Now before we start the mesos’s slave, we want to install docker. According to the docker’s doc

sudo apt-get update
sudo apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get -y install docker-ce

One last thing, we need to modify the /etc/resolv.conf. We want our internal dns can be resolved by our mesos-dns. Put the your resolver’s ip at the first line of your resolv.conf. Somthing like this:

nameserver 10.10.10.11
# your old config below
nameserver 8.8.8.8

Configure Mesos DNS

Nothing fancy here, I just used a mesos dns docker image . I guess so far so good. Make sure security groups are correct, it actually takes me sometime to debug..