How to create elastic beat from your code snippet — part 2

This is continuation from the previous post.

Table of Contents

• Recap
• Versions
• Launching elastic stack locally
• Launching fileoccurencebeat locally
• Conclusions
• References

Recap

In the previous post, we managed to implement our own fileoccurancebeat. Now its time for some testing.

Versions

Launching elastic stack locally

We will use docker containers for this. Also, for this test, we just need Elasticsearch and Kibana, so EK stack is enough. Lets launch them:

curl https://raw.githubusercontent.com/cloudronics/fileoccurancebeat/master/elasticstack-docker.yaml >docker-compose.yaml && docker-compose up -d

This will get you running elasticsearch 7.1 and Kibana 7.1 running on ports 9200 and 5601 respectively.

Verify Elasticsearch

Lets verify if elasticsearch launched successfully and running.

$ curl localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Verify Kibana

Lets try to do the same for Kibana too.

$ curl -i localhost:5601/
HTTP/1.1 302 Found
location: /app/kibana
kbn-name: kibana
kbn-xpack-sig: f06b19db5de93fafdcbc1d7c1ef4aa3e
content-type: text/html; charset=utf-8
cache-control: no-cache
content-length: 0
connection: close
Date: Sun, 02 Jun 2019 06:25:27 GMT

Verify Kibana from browser

Now, open http://localhost:5601/ in browser.

Launching fileoccurencebeat locally

Now that we have our local E(L)K stack running, lets lauch our fileoccurencebeat from the previous post. As we are running on localhost and are using default ports, we do not have to configure anything.

Adding tmp root path

Lets add a custom rootpath and filename to the beat in fileoccurencebeat.yml. Increase the period to 20 seconds, add rootpath and filename. Replace:

fileoccurencebeat:
  # Defines how often an event is sent to the output
  period: 1s

with

fileoccurencebeat:
  # Defines how often an event is sent to the output
  period: 20s
  rootpath: /tmp/
  filename: abc

Start the fileoccurencebeat

$ ./fileoccurencebeat -e -d '*'
2019-06-02T11:37:49.574+0300 INFO instance/beat.go:606 Home path: [/home/go/src/github.com/cloudronics/fileoccurencebeat] Config path: [/home/go/src/github.com/cloudronics/fileoccurencebeat] Data path: [/home/go/src/github.com/cloudronics/fileoccurencebeat/data] Logs path: [/home/go/src/github.com/cloudronics/fileoccurencebeat/logs]
2019-06-02T11:37:49.575+0300 INFO instance/beat.go:614 Beat ID: ef1f96bf-1d6f-4059-a40c-0a76be79acfb
2019-06-02T11:37:49.577+0300 INFO [beat] instance/beat.go:902 Beat info {"system_info": {"beat": {"path": {"config": "/home/go/src/github.com/cloudronics/fileoccurencebeat", "data": "/home/go/src/github.com/cloudronics/fileoccurencebeat/data", "home": "/home/go/src/github.com/cloudronics/fileoccurencebeat", "logs": "/home/go/src/github.com/cloudronics/fileoccurencebeat/logs"}, "type": "fileoccurencebeat", "uuid": "ef1f96bf-1d6f-4059-a40c-0a76be79acfb"}}}

Leave it running on that terminal and use another terminal for testing.

Configure Kibana to receive beats

In Kibana, create index pattern of format fileoccurencebeat-8.0.0–yyyy.mm.dd–N. In my case, it was fileoccurencebeat-8.0.0–2019.06.02–000001. Select \@timestamp as the time filter field name. Open the discover tab to see the events arriving.

Create Kibana visuals for file occurences

If you are familiar with Kibana charts, this should be simpler.

1. Open Visualize -> Create a visualization -> Line charts -> Select Index (filebeatoccurence-). You will end up with an empty chart with Count.

2. Configure X-axis to Date Histogram and Y-Axis to Average of occurrences. Set the refresh rate to 1 second and set the time from 2/5 minutes — now. From below you can see that there is no occurrences of file abc under /tmp

Perform a simple test

Try adding files with name abc under /tmp and observe what happens

$ touch /tmp/abc
$ mkdir /tmp/xyz
$ touch /tmp/xyz/abc
$ mkdir /mno
$ touch /tmp/mno/abc

You can see from the above video that, upon creating a files with name abc under /tmp/, you see the graph starts to show the count of occurences in few seconds.

Conclusions and further reading

Now we managed to successfully test the beats generated in our local environment. Go ahead and publish your beat to elastic community beats.

References

1. Elastic community beats — https://www.elastic.co/guide/en/beats/libbeat/7.1/community-beats.html