how to parse nginx access log with fluentd and send it to elasticsearch

Behrooz hassanbeygi
Feb 2, 2019 · 1 min read

for the past week i was working on sending nginx custom json log to elasticsearch with replacing /hide sensitive data , one of the most painful problem with fluntd was problem in parsing json format , nginx was working correct as i expected but fluentd was not able to parsing some of access logs , i had tried many things but hell no , it was not working. in the end when i was going to give up i just tell myself to reading the parser code.

all the time i was thinking fluentd use yajl ( due to the popularity and maturity) but no fluentd use oj as json parser and all the problem was on oj.

this is how i send nginx log to elasticsearch

send nginx log to fluentd and elasticsearch

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade