ELK 連接 MySQL

這次為了某些需求需要使用 Docker 建立一套 ELK ，但上網爬文的結果大多都是以 Docker Compose 的方式建立，礙於公司某些限制我沒辦法使用 Docker Compose ，所以將這次的建立流程記錄下來以防日後忘記。

Docker image download

這次使用的是 7.9.3 的版本，所以先將需要的 Image pull 下來。

$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.3
$ docker pull docker.elastic.co/logstash/logstash:7.9.3
$ docker pull docker.elastic.co/kibana/kibana:7.9.3

Elasticsearch

由於資安規定 Elasticsearch 必須要有認證機制，所以我們需要先把 Elasticsearch 的憑證生出來。

先建立一個產出憑證用的 Elasticsearch

$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es -d docker.elastic.co/elasticsearch/elasticsearch:7.9.3 #啟動elasticsearch並且設定為單一主機模式
$ docker exec -it es /bin/bash #進入該 Container
[root@25dee1848472 elasticsearch]# ./bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

# 輸出檔案位置和密碼皆可空白(直接按Enter)
Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 :

再生成cert: elastic-certificates.p12

# 輸出檔案位置和密碼皆可空白(直接按Enter)

[root@25dee1848472 elasticsearch]# ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

將憑證從container取出，並修改其檔案存取權限記住這點很重要，如果沒有修改權限的話到時 Elasticsearch 會因為無法存取憑證而起動失敗，雖然我也不知道為啥會這樣，但官網教學也是要修改權限。

在將原本的 container 刪除，並使用以下指令建立新的elasticsearch

$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es -d -v ~/桌面/Docker/ELK/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12 -v ~/桌面/Docker/ELK/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:7.9.3

container建立起來後進入container內設定密碼

$ docker exec -it es /bin/bash

生成密碼方式有兩種

1. auto

[root@cfeeab4bb0eb elasticsearch]# ./bin/elasticsearch-setup-passwords -h
Sets the passwords for reserved users

Commands
--------
auto - Uses randomly generated passwords
interactive - Uses passwords entered by a user

Non-option arguments:
command              

Option             Description        
------             -----------        
-E <KeyValuePair>  Configure a setting
-h, --help         Show help          
-s, --silent       Show minimal output
-v, --verbose      Show verbose output



[root@53d04efc3cb4 elasticsearch]# ./bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y


Changed password for user apm_system
PASSWORD apm_system = kve77x0XmSEWZZConMTt

Changed password for user kibana_system
PASSWORD kibana_system = pltXDkjzZrY1344uKavV

# kibana帳號是用來與elasticsearch連接用
Changed password for user kibana
PASSWORD kibana = pltXDkjzZrY1344uKavV

Changed password for user logstash_system
PASSWORD logstash_system = me8rtmiExvXtJBHV3D7e

Changed password for user beats_system
PASSWORD beats_system = ykcw0Q2bNYSqBbJiXAEb

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = SRuY16twbco0ANfGZTYV

# kibana網頁登入使用
Changed password for user elastic
PASSWORD elastic = x8Hbf2RTw0mMhG4xVQEC

2. interactive

[root@cfeeab4bb0eb elasticsearch]# ./bin/elasticsearch-setup-passwords interactive
--> y
--> y
--> 然後輸入想要的密碼

搞定 Elasticsearch 接著進行 Kibana，啟動方法也有兩種。

方法一

直接以 ContainerID 的方式連接 Elasticsearch

$ docker run --name kibana --link YOUR_ELASTICSEARCH_CONTAINER_NAME_OR_ID:elasticsearch -p 5601:5601 -d {docker-repo}:{version}

$ docker run --name kibana --link f3680b51fa0d:elasticsearch -p 5601:5601 -d kibana:7.9.3

方法二

透過 Config 的方式連接 Elasticsearch

$ docker run --name kibana -v ~/桌面/Docker/ELK/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml -p 5601:5601 -d {docker-repo}:{version

kibana.yml

# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://172.20.10.9:9200"  ]
xpack.monitoring.ui.container.elasticsearch.enabled: true

xpack.security.enabled: true
elasticsearch.username: "kibana"
elasticsearch.password: "pltXDkjzZrY1344uKavV"

Logstash

先去下載 Driver (mysql-connector-java-8.0.22.jar)在將 Driver 放在指定目錄

Logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://172.20.10.9:9200" ] #host IP
path.config: /usr/share/logstash/config/*.conf
path.logs: /usr/share/logstash/logs

連接資料庫的方式我們透過 Jdbc plugin 連接，由於我們的某些欄位在 MySql 裡面是以 Json 儲存，所以在 Logstash 需要先將資料拆解才能方便後續的統計應用。

logstash-mysql.conf

啟動 Logstash

$ docker run -d --name logstash -v ~/桌面/Docker/ELK/logstash/config:/usr/share/logstash/config -v ~/桌面/Docker/ELK/logstash/config/driver:/usr/share/logstash/driver -p 5044:5044 docker.elastic.co/logstash/logstash:7.9.3

最後就大功告成拉，可以先用 Elasticsearch head 確認資料有沒有順利進入Elasticsearch，然後由於 Elasticsearch 和 Logstash 內部都是採用 UTC 紀錄資料，所以從Elasticsearch head 看到的資料都會是 UTC 時間。