Elasticbeanstalk + ELK Stack
Mobile guy: “Hey, your API is responding I’m missing the ‘sort’ parameter, but I’m sending it!”
API guy: “Give me 5 minutes. I need to ssh to one of our servers and find the corresponding log”
When you only have one server responding to the requests, this is a manageable situation . However, when the number of servers increases you can’t simply have one ssh session tailing each of the logs. This is when Elasticsearch, Logstash and Kibana come into action.
Elasticsearch is a document-oriented, high availability database that is well known for its full-text search. Logstash is a data pipeline that helps you process logs and other event data from a variety of systems. It can connect to a variety of sources and can stream data at scale to a central analytics system. Finally , Kibana is a tool that sits on top of Elasticsearch and allows users to query and build dashboards incredibly fast.
For those of you who are not familiar with these three, take a look at these links:
Getting our hands dirty
1) Add Logstash to yum
The first thing we’ll need to set up is Logstash in the servers. Elasticbeanstalk installs packages using yum. If we want to add a new package to it, we need to explicitly do so by creating a file in /etc/yum.repos.d/. To tell Elasticbeanstalk we want to create this file, we must add this file to .ebextensions within your project’s root folder:
2) Tell Elasticbeanstalk to install logstash
Even though we added the package to yum, this does not install it. To do so, we must add the following file to .ebextensions within your project’s root folder:
This file does two key actions:
- Creates these directories: /opt/elasticbeanstalk/hooks/appdeploy/post and /opt/elasticbeanstalk/hooks/restartappserver/post. Scripts placed in these folders will be executed when they should. For example, those placed under /appdeploy/post will be executed after the deploy has succeeded
- Install logstash.
We created two different types of files. The file titled start_logstash.sh restarts logstash each time there is a deploy or the instance has been restarted. The file titled logstash_geoip.sh downloads a database that associates IPs with geolocations. This will be used in the next step when we build Logstash’s filter.
3) Change Rails logs
If we try to plug Logstash with Rails as it is, it won’t be able to parse logs and, therefore, we will not be able to take full advantage of the Elasticsearch + Kibana. In order to do so, I added logstasher gem. This will generate logs in another format, allowing Elasticsearch to parse them.
4) Add Logstash configuration file
So far so good. We need to tell Logstash what it is going to do. We’ll define one input, one filter and one output.
This file defines the following:
- Input: Rails server’s logs
- Filter: Using
- Output: Send log to elasticsearch
If we’ve done everything correctly, we should now be able to add a new index to Elasticsearch!
5) Go ahead and build your own Dashboards!
Kibana is incredibly flexible at the time of building dashboards. Go ahead and try to build as many as you can. You will find that you can extract more that than you imagined from your logs.
You can also visit the Discover tab and explore your logs one by one. This can be useful to trace specific requests.
6) Build some amazing tools querying Elasticsearch
If Kibana’s dashboards don’t suit your needs, you can take it one step further build alarms based on your server’s logs that periodically query Elasticsearch to get metrics. For example, average response time, number of requests that returned 5XX or 4XX, average database time, etc.
Conclusions
Configuring the whole stack helped our team to better understand what was going on with our servers. Continuous monitoring can gives you a heads up on abnormal situations before the error reaches a critical status. By looking at our server’s logs we can better understand two things:
- Where our platform needs fixing to increase speed.
- How our users are using the app.
