How to create elastic beat from your code snippet

This article is based on the beats developer guide. Elastic supports only golang for beats at the time of writing. The full source code for this beat is available in cloudronics github repository

elastic.co

Elastic beats are shippers which sits on servers, containers etc. gathering data from the host and sending it to elasticsearch server. Read more about it here.

Table of Contents

• Versions
• Introduction
• Check for file availability on a server — snippet
• Convert the snippet to Elastic beat
• Conclusion and further reading
• References

Versions

versions of tools used

Introduction

Lets say you have some code which analyses your system and you would like to monitor the same in a time series fashion i.e. the outcome over a limited/unlimited duration. Here I am going to give an example of check for file availability on a server. This snippet notifies when a file appears on a server. This case is useful for monitoring queues, buffers etc.

Check for file availability on a server — snippet

Here is a example snippet which finds and prints the occurrence of a fileabc under /home/go directory.

Convert the snippet to Elastic beat

Lets first name the beat: fileoccurancebeat.

1. Export env variables

export USER=user1
export GOPATH=/home/${USER}/go

2. Create directories and clone the elastic beat repository

mkdir -p ${GOPATH}/src/github.com/elastic
git clone https://github.com/elastic/beats ${GOPATH}/src/github.com/elastic/beats
git checkout 7.2

3. Generate the beat

mkdir ${GOPATH}/src/github.com/${USER}
cd ${GOPATH}/src/github.com/${USER}
python $GOPATH/src/github.com/elastic/beats/script/generate.py
Beat Name [Examplebeat]: Fileoccurancebeat
Your Github Name [your-github-name]: cloudronics
Beat Path [github.com/cloudronics/fileoccurancebeat]:
Firstname Lastname: Cloudronics Oy

4. Set it up

cd ${GOPATH}/src/github.com/cloudronics/fileoccurancebeat
make setup

5. Now we have a boilerplate setup to start with. The following files needed to be modified:

• _meta/beat.yaml — The template file from which the configuration is generated
• config/config.go — Struct defining the above config
• beater/fileoccurancebeat.go — Actual implementation resides here

6. Include our snippet as a function fileOccurences and add it to the event

// Checks for a given file under a given path recurssively
func fileOccurences(rootPath string, fileName string) int64 {
  var count int64 = 0
  err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
    if err != nil {
      logp.Error(err)
      return err
    }
    if info.Name() == fileName {
      count++
    }
    return nil
  })
  if err != nil {
    logp.Error(err)
  return 0
  }
  return count
}

7. An option to provide the root directory and the file name from config needs to be included. Add them to config.go and beat.yaml template and run the build with mage (yes not make):

mage build

8. Now try to launch fileoccurencebeat locally (Ctl+c to stop):

./fileoccurencebeat -e -d "*"

Conclusion and further reading

Now we have a working beat, in the next article we can test the beat metrics, plot graphs etc. with an elastic stack running locally.

References

1. Beats developer guide from Elastic — https://www.elastic.co/guide/en/beats/devguide/7.2/index.html
2. What are elastic beats — https://www.elastic.co/products/beats
3. Github repository — https://github.com/cloudronics/fileoccurancebeat