Centralize Logs with Rsyslog and Logstash on debian linux securely using TCP.

In this series, we are going to be showing you how to use the Rsyslog utility that comes packaged with debian linux to centralize logs.

The configurations documented below have been tested and work out of the box. Others are using the elastic beats software suite, but I prefer rsyslog for centralising logs using TCP. Choose wisely!!

To configure Logstash server to receive data from syslog servers, edit /etc/rsyslog.conf on all rsyslog-clients and add the following configurations:

# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#################
#### MODULES ####
#################
# Prefix all the followed filed by these 4 lines
#$ModLoad imfile
#$InputFilePollInterval 10
#$PrivDropToGroup adm
#$WorkDirectory /var/rsyslog/logstash
module(load=”imuxsock”) # provides support for local system logging
module(load=”imklog”) # provides kernel logging support
#module(load=”immark”) # provides — MARK — message capability
#module(load=”omfwd”) #load the module to support multi forwarding
# provides UDP syslog reception
#module(load=”imudp”)
#input(type=”imudp” port=”514")
# provides TCP syslog reception
#module(load=”imtcp”)
#input(type=”imtcp” port=”514")
#Config file to add and enable tls for rsyslog client
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-cert.pem
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/rsyslog/logstash
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Some “catch-all” log files.
#
*.=debug;\
 auth,authpriv.none;\
 news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
 auth,authpriv.none;\
 cron,daemon.none;\
 mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
$ActionQueueType LinkedList 
$ActionQueueFileName elk_stackqueue #set queue file name
$ActionResumeRetryCount -1 #infinite retries on insert failure
$ActionQueueSaveOnShutdown on #save in-memory data if rsyslog shuts down
$ActionResumeInterval 10
*.* @@logstash_ip_address:10514 #primary logstash

Save and close the rsyslog configuration file.

Install rsyslog-gnutls package using

sudo apt-get install rsyslog-gnutls

Restart rsyslog by running:

sudo service rsyslog restart

Your rsyslog clients are now configured to forward messages to your central logstash instances.

The above configuration will also ensure the reliable delivery of logs to the central logstash server and in case of logstash downtime or failure, the rsyslog buffer on the client(forwarder) will store the logs in the specified buffer.

The input module for logstash should be configured to listen to port 10514 using the below logstash configuration:

input { tcp { host => “localhost” port => 10514 codec => “json_lines” type => “syslog” }}```

As a potential test to this setup and a test of the capabilities of the buffer, I have added a perl script to automate the sending of huge logs(about 4k logs per minute)

Be sure to install this perl module Log::Syslog::Fast(https://metacpan.org/pod/Log::Syslog::Fast)

You can also use the perl script to load test an ELK stack with plain text and why not json data. In the next posts, I will provide more insight on load testing an ELK stack.

#!/usr/bin/perl
use Log::Syslog::Fast ‘:all’;
use warnings;
#my $i=0;
for ( $i = 0; $i < 2000000; $i = $i + 1 ) { #neverending fun
my $logger = Log::Syslog::Fast->new(LOG_UNIX, “/dev/log”, “”, LOG_LOCAL0, LOG_INFO, “mymachine”, “logger”);
$logger->send(“rsyslog is normal in its functioning $i”, 2);
$i++;
#print “it works like charm”;
}

Hope this helps!!