Real-time Logs from Multiple Sources
Finding issues on a proactive basis is always helpful to fix the source causing errors than getting to know about the problems from your customers.
Many tools and libraries exist to help us in getting better insights into our data and underlying issues. Graylog and Elasticsearch are two such awesome tools at our disposal which ate both open source, free and having great community around them.
The beginners still find it sometimes daunting to set up, configure and play around with log management and monitoring systems. The idea here is to highlight all the steps needed to make the logs monitoring easy to setup and use.
Step 1: Download
We are going to assume a Linux based system for the installation.
First of all, download the dependencies for the final setup
- Elasticsearch v6.6.0
- Mongodb v4.0.5
- Graylog 3.0.0-RC1
- Logstash v6.6.0
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.tar.gz
https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.5.tgz
https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-4.0.5.tgz (for Mac OSX)
https://downloads.graylog.org/releases/graylog/graylog-3.0.0-rc.1.tgz
https://artifacts.elastic.co/downloads/logstash/logstash-6.6.0.tar.gz
Download all above files inside a folder which will act as a base folder (in my case it is /home/ubuntu/apps for the installation and configuration.
We need to now untar/unzip all downloaded files. Assuming we are in our base folder, run below commands
tar -zxvf elasticsearch-6.6.0.tar.gz -C .
tar -zxvf mongodb-linux-x86_64–4.0.5.tgz -C .
tar -zxvf graylog-3.0.0-rc.1.tgz -C .
tar -zxvf logstash-6.6.0.tar.gz -C .
to create respective directory structure shown below.
ubuntu@server1:~/apps$ ls
elasticsearch-6.6.0 graylog-3.0.0-rc.1 logstash-6.6.0 mongodb-linux-x86_64-4.0.5Step 2: Installation and Configuration
Let us first install mongodb.
Create a file with name mongod.conf (any name can be used here) with content as follows
dbpath = /Users/pawan/databases/graylog
port = 27017
logpath = /tmp/mongograylog.log
auth = false
Our mongodb server is not secure by default (auth is set to false) and its fine for this setup as all applications will be running behind the firewall inside Google Compute Engine and will be accessible directly from the outside world.
We can now start mongodb server serving on port 27017 port by executing command
cd mongodb-linux-x86_64–4.0.5
./bin/mongod — config mongod.conf
How do we know mongodb server is started without errors? We can run command
lsof -i :27017
to see if mongodb server is up or not. If you see no output in the above command then mongodb may not be running. In that case, we need to see the log output /tmp/mongograylog.log to find the root cause of the issue.
The next part is to make sure elasticsearch is up and running.
Go to elasticsearch directory using cd elasticsearch-6.6.0 and open config file vim config/elasticsearch.yml and make sure below fields are set correctly.
cluster.name: cluster
node.name: lognode
path.data: /opt/data/elasticsearch
path.logs: /opt/logs/elasticsearch
network.host: 0.0.0.0
http.port: 9200
We can change all parameters as per our own directory structure.
Now, Elasticsearch can be started with command
./bin/elasticsearch -p /tmp/elasticsearch.pid
for running elasticsearch as a deamon,
./bin/elasticsearch -d -p /tmp/elasticsearch.pid
Again, we can make sure it is running by using lsof -i :9200
Logstash will be started next from the directory cd logstash-6.6.0
Before we start logstash we need to provide configuration file having inputsand outputsas per the logstash documentation.
input {
file {
path => “/tmp/*.log”
start_position => “beginning”
type => “logs”
sincedb_path => “/dev/null”
}
}filter {
}
output {
udp {
host => “127.0.0.1”
port => 5666
}
}
We are pulling all files ending with log extenstion from /tmp folder and sending the changes of such files to port 5666 (will be configured in graylog as listener for all such events).
Logstash can fetch log information from multiple sources and send the output to multiple destinations via pipelines. Pipelines can be configured via config/pipelines.yml file.
The setting for current setup is provided below
- pipeline.id: logs
path.config: “logs.config”
pipeline.workers: 4
- pipeline.id: syslog
path.config: “syslog.config”
pipeline.workers: 4
A total of 2 pipelines are used — one for any log available in /tmp folder and other directly from system log i.e. syslog from /var/log.
Start logstash via command
./bin/logstash — java-execution
If logstash throws No Message ([ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit) error then try using Java execution engine for logstash by using the command.
Now, finally, we need to run graylog which we are going to do from cd graylog-3.0.0-rc.1 folder.
Graylog configuration is located at /etc/graylog/server/server.conf so if it does not exists then we need to create it before making any further changes.
mkdir -p /etc/graylog/server
vim /etc/graylog/server/server.conf
Below only most relevant changes done to already existing graylog sample configuration file are shown.
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = Tbbri6RTGxY5Oyqs2dBLCkMCOYB6EEdyR5zolGc1EhVDxvpSrPy20YBOHQka3uziSMwqVPBlfFBbcl8uIHASJt8J8P6eTwp3
root_username = admin
root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
root_email = “pawan@example.com”
root_timezone = Asia/Kolkata
#root_timezone = UTC
bin_dir = bin
data_dir = /opt/data/graylog
plugin_dir = plugin
http_bind_address = 127.0.0.1:9000
http_publish_uri = http://127.0.0.1:9000/
http_external_uri = https://graylog-live.example.com/
http_enable_gzip = true
http_max_header_size = 8192
elasticsearch_hosts = http://0.0.0.0:9200
elasticsearch_connect_timeout = 20s
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
password_secret can be created by running pwgen -N 1 -s 96
Install pwgen via apt-get install pwgen.
The default root password is admin Do remember to change it using command echo -n admin | shasum -a 256 (replace admin with your favourite password string)
https://graylog-live.example.com/ is external URL to access graylog server running behind nginx server via proxy.
Assuming you already have nginx running, below config setting will direct traffic to internal graylog instance.
server {
listen 443;
ssl on;
ssl_certificate /usr/local/etc/nginx/ssl/rapidor.co/godaddy/certificate_bundle.crt;
ssl_certificate_key /usr/local/etc/nginx/ssl/rapidor.co/godaddy/certificate.key;
server_name graylog-live.example.com;
location / {
add_header ‘Access-Control-Allow-Origin’ ‘*’;
add_header ‘Access-Control-Allow-Credentials’ ‘true’;
add_header ‘Access-Control-Allow-Methods’ ‘GET, POST, OPTIONS, PUT, DELETE’;
add_header ‘Access-Control-Allow-Headers’ ‘X-Requested-With,Accept,Content-Type, Origin’;
proxy_read_timeout 36000s;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Graylog-Server-URL https://$server_name/;
proxy_pass http://127.0.0.1:9000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
}}
Start graylog using command
sudo ./bin/graylogctl start
tail -f log/graylog-server.log
Graylog server status can be seen by running command sudo ./bin/graylogctl status.
Step 3: Usage and Monitoring
Login with the username and password setup above for graylog from public facing URL: https://graylog-live.example.com/ or http://localhost:9000 (if running locally).
The next step is to setup inputs from System -> Inputs and create a new listener using RAW udp input binding on IP: 0.0.0.0 and port: 5666.
Go to Search and enjoy logs from multiple sources in a single window.
All configuration files are available at