Introducing Syslog to AWS Kinesis via Osquery

AirbnbEng
AirbnbEng
May 3, 2016 · 5 min read
Image for post
Image for post
Logs awaiting collection (Logs in Yyteri by kallerna, licensed under Creative Commons)

Simplify

We wanted an agent for OS X & Linux that supports:

  • IOC (indicator of compromise) intrusion detection
    (IPs, domains, ports, file names/paths/hashes, …)
  • State-based intrusion detection
    (shell history, /etc/hosts, NFS shares, firewall settings, …)
  • Syslog collection
  • Flexible remote logging

Since osquery is an open source project, we built syslog tables and contributed back to the community

Apple Syslog

Our first contribution is an osquery table that allows you to surface, collect and query OS X ASL syslog data without any additional configuration.

Image for post
Image for post
Image for post
Image for post

Linux Syslog

Our second contribution is an osquery table that allows you to surface, collect and query Linux syslog data.

Image for post
Image for post

Syslog Challenges

By default, OS X doesn’t send all of it’s logs to the Apple System Log (ASL). For example, here are two logs that are not sent to ASL that capture information about application and package installation:

Messages that match the query associated with a 'claim' action are not processed by the main ASL configuration file /etc/asl.conf.....
/etc/asl/com.apple.install:
? [= Facility install] store
/etc/asl/com.apple.commerce.asl:
? [= Facility com.apple.commerce] store

Amazon Kinesis Streams & Firehose

In addition to the syslog tables, we have released osquery plugins that allow for any query-results to be sent to Amazon Kinesis Streams & Kinesis Firehose. This feature is merged into osquery’s master branch and is expected to ship with osquery v1.7.4. These plugins use the AWS C++ SDK to avoid the need of deploying the Amazon Kinesis Agent.

Concluding Thoughts

This engineering effort was a labor of love by Zach Wasserman. We want to thank Teddy Reed and Mike Arpaia for their code reviews and help.

Image for post
Image for post

Check out all of our open source projects over at airbnb.io and follow us on Twitter: @AirbnbEng + @AirbnbData

Airbnb Engineering & Data Science

Creative engineers and data scientists building a world…

AirbnbEng

Written by

AirbnbEng

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io

Airbnb Engineering & Data Science

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io

AirbnbEng

Written by

AirbnbEng

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io

Airbnb Engineering & Data Science

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store