Noob’s guide to Logstash

Ipinder Singh
Feb 26 · 4 min read
input {
file {
path => "/mnt/data/logs/daily.log"
start_position => "beginning"
sincedb_path => "/home/centos/logstash_sincedb/sincedb1"
}
}
filter {// transformations}output{
file {
path => "/tmp/output"
codec => json_lines
}
}

Pipeline

Logstash Pipeline

Input

input plugin list

Filter

filter plugin list

Output

output plugin list

Full Blown example:

input 
{
kafka
{
bootstrap_servers => "<kafka_broker_ip:6667>"
topic_id => "panw_logs"
reset_beginning => false
}
}

filter
{
mutate
{
strip => "message"
}
csv
{
separator => ","
}
}

output
{
if [column4] == "TRAFFIC"
{
kafka
{
codec => line {format => "%{message}"}
bootstrap_servers => "<kafka_broker_ip:6667>"
topic_id => "traffic_logs"
}
}
if [column4] == "THREAT"
{
kafka
{
codec => line {format => "%{message}"}
bootstrap_servers => "<kafka_broker_ip:6667>"
topic_id => "threat_logs"
}
}
}
Ipinder Singh

Written by

A Software Engineer, who loves to read. Starting my writing, still early-game!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade