Image for post
Image for post

Making Big Data Work for You

Marcel Domke
May 2, 2017 · 5 min read

Optimize your workflow by using DataTables instead of Excel sheets

Why we do what we do

Perhaps you are familiar with Excel sheets (.csv, .xls) and how unwieldy they become when the dataset grows to a certain point. We’ve decided to optimize our processes by using HTML tables to show our users the revalued data instantly in our application. Here, I will explain how we optimized our code by increasing MySQL performance, handling a huge http response, and made our users happier by using fancy HTML table plugins instead of boring Excel reports.

That is huge!

Let’s say we have 500 000 products which are live and every day we collect the information we need for these.

Boring DUH!

Maybe you wonder now, why should I continue reading? There are several ways to deal with, like utilizing ajax and pagination, then everything is fine. But maybe I can still show you some new stuff that may change your point of view. Sometimes it is necessary to display all the data, because you need to calculate or interact with it.

Join to Death

First of all, you should always optimize your queries. As I already mentioned, we are using MySQL and the first task was making the query quick and efficient, therefore we analyzed the query and reduced the typical query blockers.

  • INNODB (more less better for writing operations)
  • MEMORY (if you can run the TABLE in memory)
Image for post
Image for post
Short feature overview of the 5 engines

On Page Load is so 2010

In the beginning, when I started programming, I often heard that a good developer has to use ajax requests to be top notch in the web community. What if I tell you now, that this is not always the best way to deal with a huge response? We started of course with ajax requests. Fetching from our DB was less than a second, however the fetching by the ajax request was unbelievably slow (60 seconds or more). We analyzed the response and found out that the generated and human readable JSON response was the issue. It was simply too huge and needed a lot of time to render (>10 MB).

DataTables for the win

This was the first day we used DataTables (http://www.datatables.net). DataTables is a highly flexible jQuery Javascript library, based upon the foundations of progressive enhancement, and gave us the opportunity to add advanced interaction controls to our HTML tables.

Image for post
Image for post

Conclusion

We experienced a positive response from our users after integrating DataTables (or any kind of web views) to display their aggregated data. The biggest wins were of course the speed, instant search, and sorting filters. Users found themselves with new opportunities to analyze the data more efficiently. On the developer side, it’s now much easier to extend the table, write tests, or identify bugs.

ABOUT YOU TECH blog

Want to know more ABOUT TECH?

Marcel Domke

Written by

Head of Backbone Panel Development / Cloud-Services Cloud-Panel at ABOUT YOU GmbH Employed since 01.06.2015

ABOUT YOU TECH blog

Do you want to know more ABOUT TECH? What we're developing? Which technologies we're using? Have a look at corporate.aboutyou.de/de/jobs. We successfully launched the 1st eCommerce infrastructure for large scale & innovative retail - ABOUT YOU CLOUD see https://cloud.aboutyou.com

Marcel Domke

Written by

Head of Backbone Panel Development / Cloud-Services Cloud-Panel at ABOUT YOU GmbH Employed since 01.06.2015

ABOUT YOU TECH blog

Do you want to know more ABOUT TECH? What we're developing? Which technologies we're using? Have a look at corporate.aboutyou.de/de/jobs. We successfully launched the 1st eCommerce infrastructure for large scale & innovative retail - ABOUT YOU CLOUD see https://cloud.aboutyou.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store