London skyline / loveoflondon.tumblr.com

Installing Kale Skyline

A more thorough installation

At my current job we’re discussing using Skyline to track down anomalies. It’s something that came up frequently in talks as the amount of metrics being collected and the amount of knowledge around each of them is increasing.

After checking the project’s github account I felt the installation instruction were not clear enough and there were a few particular occasions that I got stuck for a while, browsing other forums I noticed more people struggling with its installation so here is a detailed step by step reproducible description that anyone can follow to try out Skyline!

Get a box

First thing is to get a working barebones server installation. We use Debian for production servers so I picked Wheezy which is the latest stable release.

Not only that but I also found out that the previous version, Debian Squeeze, struggled with an easy installation of Redis and I’d rather do it all within regular Debian ports.

I also use Vagrant for all my local testing so I found out a helpful already made box here which will allow you to not only be able to bypass building a new Vagrant Debian Wheezy box but also use exactly the same initial setup as I did.

vagrant box add wheezy64 http://dl.dropbox.com/u/937870/VMs/wheezy64.box 
mkdir skyline
cd skyline
vagrant init wheezy64

Now you have a ready to use Wheezy box but to save a restart later on you should set up port forwarding on your local Vagrantfile which will bind to Skyline’s web application.

vim Vagrantfile

And change:

 config.vm.network :forwarded_port, guest: 1500, host: 1500

Boot up and login:

vagrant up
vagrant ssh

And after login in don't forget to update your box.

sudo apt-get update
sudo apt-get upgrade

Skyline requirements

Python

Once logged in to our new updated box we can start installing all requirements.

First things first we need pip to be able to add some of the requirements for python, as well as python-dev without which some of the packages used with pip will not work!

sudo apt-get install python-dev
sudo apt-get install python-pip 

After that we can clone the project’s repository

git clone https://github.com/etsy/skyline.git

Once we have the project we can change into the folder and start going through all the python dependencies.

cd skyline
sudo pip install -r requirements.txt
sudo apt-get install python-numpy python-scipy python-scikits.statsmodels
sudo pip install patsy msgpack_python 

Redis

Debian is not famous for having packages running on the latest version but for its stability. Unfortunately we do need the latest version which is 2.6 rather than the stable one which is 2.2.

But not all is bad, version 2.6 is available on the Wheezy backport which means it will be available in the next Debian stable version and was back ported to Wheezy.

You can do this easily by adding the backport URL for Wheezy and installing that specific version.

sudo vim /etc/apt/sources.list

Change.

deb http://ftp.ch.debian.org/debian/ wheezy-backports main

Update and install Redis-Server, don't forget to kill the process as it will be started with the default config right after the installation.

sudo apt-get update
sudo apt-get -t wheezy-backports install redis-server
sudo pkill redis

Configuration setup

With all dependencies installed you can now copy an example for the settings file. This will be the place to add your Graphite URL and other details but for now we only need to update the IP bound to the web application so the Vagrant host will display it.

cp src/settings.py.example src/settings.py

Edit file for bind.

vim src/settings.py

Change.

WEBAPP_IP = ‘0.0.0.0'

And one more thing. Add the log folders for all the applications.

sudo mkdir /var/log/skyline
sudo mkdir /var/run/skyline

And ensure /var/log/redis already exists

Running Skyline

Start all components

You are ready to start the application, just change into the bin directory and start all components.

cd bin
sudo redis-server redis.conf
sudo ./horizon.d start
sudo ./analyzer.d start
sudo ./webapp.d start

Ensure there are no errors in the logs, for example if you forgot to kill the Redis process earlier on you’d run into problems now!

Testing

cd ../utils
python seed_data.py

Should show you this output mentioning the installation was successful.

Connecting to Redis…
Loading data over UDP via Horizon…
Congratulations! The data made it in. The Horizon pipeline seems to be working.

Visualising it

You have now Skyline ready to go so just check the port you forwarded into your Vagrant host to use the web application!

http://localhost:1500

Skyline UI in Vagrant Host