Big Brother by Undefined Variables — eMAG Hackathon 2017 Best Hack Award
eMAG Hackathon is an annual event for software developers where teams consisting of eMAG employees take on the challenge to develop their projects during 24 hours competition. This year’s theme was “Future Hacks”, as a follow-up of previous year’s edition. Teams were expected to implement innovative, business valuable ideas in only 24 hours and present them to the jury and the other 15 competing teams the next day. You can read more about the event here.
The event took place on 9th and 10th of September at eMAG headquarters in Bucharest, Romania. We arrived a day before as we needed to get there from Warsaw, Poland — where we work on daily basis at eMAG Software Development Hub. Our team, called “Undefined Variables”, consisted of 4 people: two Software Developers (including Team Lead — myself), one Quality Assurance Engineer and one Scrum Product Owner, which was a pretty optimal set, because we were able to cover all areas required in a regular software project. Our project “Big Brother” won the grand prize called “Best Hack Award” — which was awarded to the team that had gathered the most points in all evaluation categories.
What was it all about
Our idea was to develop an application which would be able to recognize customers in a showroom and track their activities in order to provide real time user behavior analysis to answer the following questions:
- in which products are customers interested more?
- how much time do they spend in front of which products (what implies that a person is interested in a particular item)?
- which areas of the showroom are the most attractive to customers?
It would be possible to analyze recordings from existing video files as well as process live video stream by connecting to a store surveillance camera. We called our project “Big Brother” as a reference to Orwell’s novel “Nineteen Eighty-Four”. The project idea itself was a result of many hours of brainstorming before we finally came up with its final shape.
Before everything started, when projects were registered, every team was expected to assign their project to one of several available categories. We had reported ours as “showroom customer experience improvement”, which seemed a natural choice as it was all about showrooms and dealing with customers behaviors inside. For someone responsible for placing products on the shelves or arranging shop’s space to make customers reach particular products easily such data may be very valuable as nowadays the only data those people may use are products sales data and their intuition. And when sales data can be associated with the data describing customer attention to particular goods there may be many more ratios calculated.
Technology
For the core logic of our project we decided to use OpenCV library (http://opencv.org/) which provides rich set of image and video processing features to a developer. We also utilized Python to develop logic and a demo of our app. We also considered Java and C++ as there were several bindings but we eventually decided for Python so that we wouldn’t spend much time on coding — the fact was there was little time and many problems to solve.
Since we needed some video recordings with customers and products we used a GoPro camera mounted on the ceiling with a selfie stick. There was an empty conference room nearby (the office was pretty empty as it was Saturday) so we turned it into a fake showroom with some random objects we found in the office space acting as products and of course ourselves in the role of showroom customers.
For data storage we chose MySQL (just to demonstrate that there is some raw result of video processing) available for future analysis, but anything similar (e.g. MongoDB) would be suitable here.
How we achieved our goal
We organized our work as a regular agile team. We created some initial diagram with an overview of all our ideas, we did story mapping to split the whole big picture into smaller parts and assign priorities. That gave us possibility to easily cut off features with lower business value after realizing that we don’t have enough time and leave only those more relevant. We created a Kanban board on a nearby door and started working on features. We agreed on having a project status review every 2 hours to check whether we are on track and to solve any impediments quickly and also to adapt the scope to the time we had left.
As the outcome of our work we wanted to detect customer movements and have them described as the coordinates in time function. The goal was also to have product visits counted: by “visits” we meant people approaching to product “hot zones” which had been defined after placing products on tables and before customers started visiting the fake showroom. To achieve this we created an algorithm which had to do the following:
- for each frame detect the object contours by applying several transformations (color reduction, Gaussian blur, threshold)
- subtract initial frame with only products visible from currently processed frame
- calculate difference of 2 neighboring frames (including initial frame subtraction)
- differentiate objects by assigning identifiers and tracking movement directions
Some early results of our work can be found on the next picture
When we reached the point where we agreed that the results finally meet our expectations and all the must-have features are ready (as well as some should-haves) we recorded the demo to a video. Here you can see it exactly it was shown to the jury:
What can be observed on the demo are product hot zones indicated by red rectangles, customers movements detected indicated by green surrounding rectangles (customers are marked with green dots at the center of green rectangles). Entering product hot zone for at least 3 seconds results with a visit counted for a product. There are product visits counters displayed at the top right corner of the video.
Presentation time
Each team was expected to prepare a 3-minute presentation of their project. We found it important to have not too many slides in order to save time to show also 45 seconds video as a demonstration of the app. We practiced the presentation several times to fit in the time limit. Couple slides needed eventually to be removed to make it happen. Our main goal of the presentation was to emphasize the business benefits of implementing this project and also easiness of implementation (it’s enough to process existing recordings or use surveillance cameras which every store already has).
Then the presentations time came! The jury drew presentations order before and we were assigned number 11 out of 16. The less time there was left to our turn, the more stressful it was, but finally we did it! The presentation was done and there was finally some time to rest while waiting for the results to be announced. We were really surprised when we heard “[…] and the Best Hack prize goes to Undefined Variables!” Nobody from our team expected that to happen. Neither did I. But after couple moments of this “I can’t believe it” state I felt awesome to be part of the team rewarded for the past 24 hours of this technological struggle.
Summary
To summarize, taking part in the Hackathon was a great experience for me and my team and of course winning the grand prize made this experience much more meaningful and important as my personal achievement. It was a fantastic experience and I can for sure recommend its future edition to every developer.
