To Play Around ELK in the Docker Playground

If you don’t know how to use Docker Playground, please read my previous log, “Setup a Docker Swarm in Docker Playground in X mins”

You only need docker-compose (default installed on each node) for ELK, you don’t need to install Docker Swarm. Create a new instance and then simply run the following commands to start an ELK working environment.

wget https://gist.githubusercontent.com/nomkhonwaan/41c352721aa2c327abb50825e2c90369/raw/a597ffd994c29bb0fef09bbe9a9e8d9509150af6/docker-compose.yml
docker-compose up
// or
docker-compose up -d

Click on the NodePort 5601 showed above.

then, you can see a kibana 5.5.2 home page with an elasticsearch running in the background.

Take a look at the k8s yaml file for ELK in this example, which we wget from https://gist.githubusercontent.com/nomkhonwaan/41c352721aa2c327abb50825e2c90369/raw/a597ffd994c29bb0fef09bbe9a9e8d9509150af6/docker-compose.yml and we can do some modification as folllows;

version: '2'
services:
  elasticsearch:
    image: elasticsearch:latest
    environment:
      ES_JAVA_OPTS: -Xms4g -Xmx4g
  logstash:
    image: logstash:latest
    volumes:
      - $PWD/config-dir:/config-dir
      - $PWD/logs:/logs
    depends_on:
      - elasticsearch
    links:
      - elasticsearch
    command: logstash -f /config-dir/logstash.conf
    ports:
      - 5000:5000
  kibana:
    image: kibana:latest
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    links:
      - elasticsearch

Because we have 8GB memory per node in the k8s or docker playground, so that, we can change ES_JAVA_OPTS: to -Xms4g -Xmx4g for the best practice.

And also add ports: 5000:5000 for logstash in order to stream syslog sample logs into logstash.

And we also know $PWD/config-der is the logstash config path we can use as the /etc/logstash/conf.d directory we can use as in the normal ELK installation.

So that, you can create a sample logstash config file in your $PWD/config-der/logstash.conf folder, such as logstash-syslog.conf as follow;

cd
mkdir -p $PWD/config-dir/logstash.conf

cat $PWD/config-dir/logstash.conf/logstash-syslog.conf

input {
  tcp {
    port => 5000
    type => syslog
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch { hosts => ["elasticsearch:9200"] }
  stdout { codec => rubydebug }
}

After you change logstash config file, you need to restart ELK by

docker-compose restart

It should re-run logstash with your config files under $PWD/config-dir/logstash.conf/ directory.

Now, you can try to stream some syslog to logstash on port 5000 from local or another instance (node IP: 10.0.9.4 for example). Such as

telnet 10.0.9.3 5000
// or
cat <file> | nc -w 1 -v localhost 5000

followed by the syslog samples bellow.

Dec 23 12:11:43 louis postfix/smtpd[31499]: connect from unknown[95.75.93.154]
Dec 23 14:42:56 louis named[16000]: client 199.48.164.7#64817: query (cache) 'amsterdamboothuren.com/MX/IN' denied
Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
Dec 22 18:28:06 louis rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="2253" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

Where, 10.0.9.3 is the node run ELK, and port 5000 is logstash input port.

and check the logs of logstash container on the ELK node (10,0.9.3) as follow

docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                    NAMES
69524ebd6ce8        logstash:latest        "/docker-entrypoin..."   10 minutes ago      Up 10 minutes       0.0.0.0:5000->5000/tcp   root_logstash_1
b54847eb3c9f        kibana:latest          "/docker-entrypoin..."   About an hour ago   Up 12 minutes       0.0.0.0:5601->5601/tcp   root_kibana_1
0930b302d5a7        elasticsearch:latest   "/docker-entrypoin..."   About an hour ago   Up 12 minutes       9200/tcp, 9300/tcp       root_elasticsearch_1

where 69524ebd6ce8 (695) is the container ID for logstash, so we can use docker logs -f 695 to see the parsed logs as follows;

docker logs 695

Now you can use kibana to see the logs too.

Further reading for Logstash Configuration you can refer to https://www.elastic.co/guide/en/logstash/current/config-examples.html with sample examples in this blog.