Foosballytics — real time computer vision

Breaks are extremely important for every employee’s work efficiency. Not only do they give your eyes time to rest, but also keep your mind focused for longer. There are different opinions on how this time should be spent. Some like to keep it simple and limit themselves to a chitchat by a coffee machine. Others prefer to grab some fresh air and go for a smoke. In Tooploox’s office in Wrocław, you can also find a group of people who really know what is good for their health. And that, of course, is sport. Unfortunately, playing a whole football match during a break would be pretty difficult even if we had enough space for a pitch in the office. That’s where foosball comes in.

Even though some dare to disagree with us, we do consider foosball a real sport. And its fans, like any other real sport fans, deserve the right to a full, professional sport experience — with score counting, live broadcast, goal replays and even some professional commentator. Last year, during the first internal hackathon, we decided to fight for this right. A multidisciplinary team of developers, designers and data scientists united in the mission of moving our foosball table into the digital world. That’s how the Foosballytics system was born.

The whole system has two major purposes. Firstly, we want the system to automatically track the score. Secondly, we want to improve the player’s experience by providing them with statistics and highlights of played matches.

To track the score, the system is equipped with a camera that detects each scored goal. This feature makes the game more enjoyable, as the participants don’t have to worry about moving the cubes — they can simply focus on controlling the ball.

Tracking the score is virtually tracking the ball location. It highly improves the player’s experience, as the goals are counted automatically and each time someone scores, a proper sound signal is triggered. Therefore, choosing the right camera was crucial for the system to be reliable. We had a few requirements for it:

• Sufficient framerate to detect fast ball movement
• Decent color fidelity
• Connection between camera and workstation which is fast enough to handle the real-time tracking
• OS X support — we wanted to make it accessible for everyone at Tooploox

Initially, we considered a GoPro camera but due to the wireless transfer delay, we gave up this idea. Admittedly, there are many cheap webcams available on the market, unfortunately most of them can grab the video with the maximum of 30 frames per second. It is good enough for human’s eye perception, but not sufficient to track the ball, which may accelerate rapidly and move with high speeds — “Expert players have been known to move balls at speeds up to 56 km/h (15,6m/s)”!. It means the ball on our table (which is 1.2 meters long) needs 0.07s to get from one goal post to another, while travelling with the max speed. With 30 fps we would be able to detect it on 2 frames.

We needed something better to build a reliable solution. We established we were not able to process more than 56 fps when using our computer. Therefore, we couldn’t go for high speed cameras. We have considered many different devices. Finally, we have discovered that in the last quarter of 2016 Logitech has released a new webcam, which was capable of recording video at 60 frames per second in 720p format. Taking into account that we didn’t know any professional players, we decided this frame rate was good enough to be used in our project.

We needed something better to build a reliable solution. We established we were not able to process more than 56 fps when using our computer. Therefore, we couldn’t go for high speed cameras. We have considered many different devices. Finally, we have discovered that in the last quarter of 2016 Logitech has released a new webcam, which was capable of recording video at 60 frames per second in 720p format. Taking into account that we didn’t know any professional players, we decided this frame rate was good enough to be used in our project.

The next step involved assembling a construction that would hold our camera. For our internal use, mounting the camera on the ceiling of our basement where the table is placed would be good enough. However, since we were considering the possibility of going out of the basement and showing Foosballytics to the world, we decided to create something movable. After some time spent on shopping in DYI store and working in our basement, the construction was ready to use. The only hardware issue that remained was the lighting equipment — we wanted static and pretty bright conditions, thus we bought a perspex, and we added a LED strip. This made our stadium look much more like Wembley!

The core of the system was the software. We wanted it to detect the x and y coordinates of the ball and detect each scored goal. In order to achieve it, we used OpenCV library to perform color based segmentation for pitch, ball, and goalposts. There are many object detection algorithms, however they require meaningful computational power. We decided to keep it as simple as possible to provide algorithm that would operate efficiently on any standard personal computer.

After several steps of image processing involving a few of morphological operations that helped us to extract the objects of interest and remove noise, we were able to apply color based segmentation that involved HSV model calibrated automatically before the start of the game. We were operating in HSV color space, which is better than the standard RGB model due to robustness to lighting conditions. In this color space, we built a model of the pitch, the goal posts and the ball.

The first step was to detect the pitch and resize the captured frame to exactly fit the area of playing a game. Secondly, we needed to look for goalposts. When we were assembling the construction, we indicated the goal line with yellow tape. It was a pretty good idea, but it turned out that due to the official foosball rules (Did you know that such document exists? It was a total surprise for me.) we cannot make any alterations to the surface of foosball table as it may affect the playing characteristics.

After some adjustments, we realized that the camera catches perfectly the dark area behind the goal line, which was good enough to extract goalposts by dark color! To minimize noise in the system, we calibrated the pitch and goalposts by averaging the detections from multiple frames. The ball was also detected by its color, but there was a problem with score detection. Every goal was counted when the coordinates of the ball were inside the area of goalposts calibrated earlier. At Tooploox we have a lot of colleagues that are capable of striking very fast shots (hello, Paweł!), and sometimes we were not able to track and read coordinates of the ball, even with the camera holding 60 frames per second.

There was a pretty easy solution for score detection for that kind of cases — even a fast ball leaves a blurred trace on the captured image, so if the ball was not detected in the pitch or goalposts, we looked into colour histogram of goalposts area for any changes — if there was an anomaly of Hue value corresponding to yellow color, we counted it as a score. Those settings gave us pretty good results with very high accuracy — just in case we specified that our app should have a feature to adjust score, but fortunately, it was rarely used.

In the meantime, our Android team was working on the app for showing the game results. The app users users could select teams before the game. It also shown when all the goals occurred, and at the end of the match the replays of goals and heat map were displayed in order to analyze it, and learn from previous mistakes ;) The connection between machine running computer vision software and the mobile application was our only concern. At the beginning of hackathon we decided to use buttons connected to a Particle device for goals counting, but with the possibilities of computer vision we decided to drop this idea. There was a Particle Cloud that we could use for communication.

We managed to meet the deadline for preparing a demo version, so we could use the table as a part of our booth during the Boiling Frog conference in Wrocław. Every attendee had a chance to play foosball with live view for the audience, and recall all the game highlights.

We have a good base to work on in the future. We have a lot of ideas about potential Foosballytics’ features, including a database with each player performance, detection of goal scorer, live commentary and many more. I’m sure that we laid groundwork for future development.

Check our facebook page for more photos and videos!

Graphics by Ania Langiewicz

Originally published at Tooploox.