Our growth makes tracking everything a challenge — our intern had the solution

At Dataprovider.com we don’t just collect data, we also have a number of services that benefit our clients. We need to keep an eye on these services at all times to see if they are running smoothly. That is why our talented intern Eelco has hand-crafted the latest Status Board. Here’s how he did it, while also having a fun time:

“I’m Eelco Blaauw, 27 years old and I’m currently working on my Bachelor Software Engineer at Stenden university. When my internship started during the summer, I figured this was the reason everyone at Dataprovider.com was so laid back. But nothing was further from the truth, as at the time of writing this article it is mid-December and everyone is even more laid back than during the summer. I found the people working at Dataprovider.com very open and helpful. This has contributed to the wonderful and educational time I experienced during my internship. The icing on the cake for me, was the fact that Dataprovider.com provides the lunch. This makes it very easy to connect with co-workers and discuss topics that occur outside the scope of the company.

Dataprovider.com is growing rapidly. Because of this, keeping track of all services and their current status becomes more and more difficult as time progresses. The problem has already been tackled years ago by none other than Dataprovider.com’s CTO Christian Branbergen. However, the Status Board he developed is considered ancient for today’s needs. Simply said, it’s not up to the task of providing the information that is needed by employees today.

Image from Tirza van Dijk — via Unsplash

My assignment was to create a brand-spanking new Status Board using the (by employees) preferred PHP micro-framework called Silex. This micro-framework is a minimalistic version of Symfony. One of my biggest challenges, was making sure I worked according to the guidelines for programming that are accustom to all Dataprovider.com employees. One of the goals was to make the product as straightforward to maintain and update as possible. Therefore staying close to how the rest of the developers operate is crucial. Let’s take a short dive into the solution I came up with for monitoring the services provided by Dataprovider.com:

Every server that is running a service generates logs. What are logs you say? Logs are text files that contain messages, and a message could for example state that an error has occurred. So why not catch all these logs and store them in one location? For this, a new server has been installed and configured with a Logstash and Elasticsearch solution. Logstash is used for filtering and storing all incoming logs. To be able to find logs stored, Elasticsearch is used. On all the servers that require monitoring Filebeat is installed. Basically, Filebeat sends logs from the server to the previously mentioned Logstash server. So now we were already storing logs in one location and we were able to find specific logs if needed. All that was then left, was to develop the front-end. This was relatively easy since the front-end simply needs to display all the logs. Per server the front-end displays the following categories: ‘error’, ‘warning’ and ‘total’. Logs categorized as ‘error’ are logs that have a high impact on the service, logs with the ‘warning’ category have a lower priority. The last category ‘total’ contains all logs provided by that specific server.

My internship was a wonderful journey and I’ve learned a great deal. The awesome people working at Dataprovider.com provided me with great moments and humor is definitely not in short supply. You even have the ability to influence what music is being played in the office. All things combined, Dataprovider.com is a great company and has a lot of potential for any IT-student to thrive in.

The entire internship led to Datapovider.com offering me a part time function. Which I obviously accepted because who wouldn’t want to work at such an awesome company.

The Dataprovider.com team was happy to hear that Eelco loved his time with us as an intern, and we liked having him around just as much. That’s why we are very excited that Eelco has decided to stay with us as a part-time Software Engineer in the future.