Beats on Raspberry Pi & a Side of ELK

Mixing Beats with Raspberry Pi and ELK sounds like a Martha Stewart recipe that went wrong. Probably something she would create with Snoop in effort to hide his veggies.

Is that ELK? src

Well I’m no Martha Stewart and I definitely don’t have the celeb status to hang with Snoop Dogg. — Whats up Snoop, if your reading this.

Nope, this is about Ealstic.co Beats data shippers and setting them up on Raspberry Pis.

Over the past six months I have been on a non stop exploration and learning journey. It began with Python and that led into mini projects for the house. All of the sudden I have five Raspberry Pi units and I am working through one project after another. As I expanded my tinkering I wanted to start capturing logs for all the devices in my home network for monitoring, fun, and learning. I work in InfoSec after-all, logs are sorta my thing.

Elastic became an obvious go to option for my home lab needs. I setup my ELK stack (Elasticsearch, Logstash, Kibana) quickly, and I even wrote a nice script I call easyELK to simplify the setup. I was on cruise control logging my firewall, Mac devices, Windows devices, and Ubuntu Servers in ELK. Then, I discovered Elastic doesn’t release and maintain Beats for arm architecture. You’re Killing me Smalls!

src

I searched for a workaround and spent hours going through numerous blogs, discussion threads, and videos, but none of them worked. They were either outdated or beyond my skillset. Many of them seemed to be very manual and left room for error. I was at a slight disadvantage because I only know Python which doesn’t require you to compile code. All the research I was doing led back to GOlang and compiling code. Did I mention this all started on a learning journey? Challenge accepted! I consolidated my notes and set out to create my own solution… easyBEATS

Hardware: Raspberry Pi 3b+, 4

Software: Raspbian, Beats 7.3.2

You want to get your ELK stack up and running first. If you don’t have this complete, check out my easyELK project mentioned earlier. Make sure that you have all your dashboards and index patterns loaded. easyBEATS wont handle that for you, but easyELK will.

I think that GOlang is great, except for setting GOPATH. It is a mind bender and a source of frustration for a noob. I kept mine with the default settings on my Raspberry Pi. $HOME/go is used through the entire script. The Go Workspace directories are created by easyBEATS.

mkdir $HOME/go 
mkdir $HOME/go/bin
mkdir $HOME/go/pkg
mkdir $HOME/go/src
mkdir $HOME/go/src/github.com
mkdir $HOME/go/src/github.com/elastic

There are some dependencies that need to be covered. I threw in a couple ‘if’ statements to check for them and reduce manual prep work.

if [ $(dpkg-query -W -f='${Status}' python-pip git 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
echo " ->Installing python-pip git..."
apt-get install python-pip git -y;
elif [ $(dpkg-query -W -f='${Status}' python-pip git 2>/dev/null | grep -c "ok installed") -eq 1 ];
then
echo " ->python-pip git is installed"
fi
echo "Checking for virtualenv..."
if python -c 'import pkgutil; exit(not pkgutil.find_loader("virtualenv"))';
then
echo " ->virtualenv is installed"
else
echo " ->Installing virtualenv..."
pip install virtualenv
fi
echo "Checking for Make..."
if [ $(dpkg-query -W -f='${Status}' make 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
echo " ->Installing make..."
apt-get install make -y;
elif [ $(dpkg-query -W -f='${Status}' make 2>/dev/null | grep -c "ok installed") -eq 1 ];
then
echo " ->Make is installed"
fi
echo "Checking for GCC..."
if [ $(dpkg-query -W -f='${Status}' gcc 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
echo " ->Installing GCC..."
apt-get install build-essential manpages-dev -y;
elif [ $(dpkg-query -W -f='${Status}' gcc 2>/dev/null | grep -c "ok installed") -eq 1 ];
then
echo " ->GCC is installed"
fi
echo "Checking for Go..."
if [ $(dpkg-query -W -f='${Status}' golang-go 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
echo " ->Installing Go..."
sudo apt-get install golang-go -y;
go version
echo " ->$(tput setaf 6) Go is installed$(tput sgr0)"
elif [ $(dpkg-query -W -f='${Status}' golang-go 2>/dev/null | grep -c "ok installed") -eq 1 ];
then
echo " ->Go is installed"
fi

Now it’s Go time. Yes pun intended. Time to use Go to git the source code and make the new arm versions of Filebeat, Packetbeat, Metricbeat, and Auditbeat.

Why do I have a variable BEAT_VERSION? Because it makes it much easier to manage the going forward. Each release Beats has a commit code. Once you know where to get the commit code, you can plug that into easyBEATS and you don’t have to update a whole bunch of links. The code is to the left of the release.

https://github.com/elastic/beats/releases

BEAT_VERSION=”5b046c5"
cd $HOME/go/src/github.com/elastic
go get github.com/elastic/beats | tee -a $log
cd beats
git fetch
git checkout $BEAT_VERSION | tee -a $log

I also learned that compiling can eat up a good amount of resources. I decided to temporarily enable and utilize swap space to ease the compiling process along.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

I also realized that it would be a good idea to check the version of the arm chipset. easyBEATS will run a check using lscpu to determine what arm version to build (arm or arm64). Well that was a fun learning experience too. Check out WikiChip for some assistance. Now that the architecture is set we download the source code and compile.

src

cd $HOME/go/src/github.com/elastic/beats/filebeat
if [ $(lscpu | grep -c “armv7”) -eq 1 ];
then
GOOS=linux GOARCH=arm go get;
make;
elif [ $(lscpu | grep -c “armv8”) -eq 1 ];
then
GOOS=linux GOARCH=arm64 go get;
make;
fi

Moving right along, literally, create the directories and move the files into place, enable the service, set the permissions, and have everything ready for final config. The last part requires manual work. Configure each of the Beats modules as needed and start the service.

sudo mkdir /usr/share/filebeat
sudo mkdir /usr/share/filebeat/bin
sudo mkdir /etc/filebeat
sudo mkdir /var/log/filebeat
sudo mkdir /var/lib/filebeat
sudo mv $HOME/go/src/github.com/elastic/beats/filebeat/filebeat /usr/share/filebeat/bin
sudo mv $HOME/go/src/github.com/elastic/beats/filebeat/module /usr/share/filebeat
sudo cp $HOME/go/src/github.com/elastic/beats/filebeat/filebeat.reference.yml /etc/filebeat
sudo mv $HOME/go/src/github.com/elastic/beats/filebeat/modules.d/ /etc/filebeat/
sudo cp $HOME/go/src/github.com/elastic/beats/filebeat/filebeat.yml /etc/filebeat
sudo cp $HOME/beats_arm/filebeat_files/fields.yml /etc/filebeat
sudo cp $HOME/beats_arm/filebeat_files/LICENSE.txt /usr/share/filebeat
sudo cp $HOME/beats_arm/filebeat_files/NOTICE.txt /usr/share/filebeat
sudo cp $HOME/beats_arm/filebeat_files/filebeat.service /lib/systemd/system
sudo chmod 700 /var/log/filebeat
sudo chmod 755 /etc/filebeat/
sudo chown -R root:root /etc/filebeat
sudo chown -R root:root /usr/share/filebeat/*
sudo /bin/systemctl daemon-reload
sudo systemctl enable filebeat
sudo swapoff -v /swapfile
sudo rm /swapfile

src

I learned a lot about Elastic products through this project. My bash scripting skills went up a couple notches and posting my project on GitHub was exciting. I didn’t calculate the number of hours spent on the project, but I estimate 60 hours learning, testing, and perfecting the script to make it repeatable. It was a lot of fun and I feel like I appreciate the Open Source spirit much more.

Two additional applications I also came across were pivpn and Pi-Hole. I originally setup Pi-hole on an Ubuntu server because I wanted to use elk-hole for logging. This gave me more reason to get Beats working on my Raspberry Pi. After creating easyBEATS, I was able to move Pi-Hole over to my Raspberry Pi and use elk-hole to ship the logs over to my ELK stack. Not sure how many birds that is, but one stone… . You get the point.

Hopefully you found this interesting and at minimum discovered that a solution to create Beats packages for Raspberry Pi does exist. You don’t need to work through the learning curve and fight all the frustration I had. If you want to use easyBEATS or contribute to the code visit my GitHub repo. Send me some ideas and I will be happy to try and help create another ‘easy’ solution with you.

Check these links out:

- easyBEATS
- easyELK
- elk-hole
- Pi-Hole
- pivpn
- Pi-Hole + OpenVPN