Cut Highlights From Volleyball Game With Ball Tracking and Computer Vision
Recently I started to play with ball tracking in volleyball and this experiment promised interesting results.
There are multiple ways to go further and I stuck for a while, looking for insights.
I wanted something useful, ready to apply in the real world. Ball tracking is fun but does not worth a lot as is.
Then I got an idea to use ball tracking to “filter” raw volleyball video for the most interesting moments.
Usually, volleyball video is very sparse, often contains warmups, breakouts, disputes, etc, but viewers are looking for great hits, incredible defense, or killing serves.
People do a lot of work to cut highlights and the job is time-consuming and diligent. Professionals hire editors for that, but most amateurs don't have such ability, the only exceptions are very enthusiastic people but they are pretty rare.
So automation of highlight cuts would help players to enjoy short rally clips instead of viewing the whole boring game.
The logic is to track the ball using the algo from the mentioned article. To be honest, the algo was modified many times to reduce false-positive detections. Then I am measuring a pause between ball detections — if it is more than the specified threshold — game over.
So far the detections are not 100% accurate, it is not easy to find a correct threshold. Experimentally I found the best results are when a rally starts 1 second before the ball detection and ends 3 seconds after.
Sometimes it happens to be too long, but better to see one more second than suddenly stop at the most interesting moment.
There is a short demo — a few rallies detected by the ball tracker in a real game (from Moscow Volleyball League):
The first processed game, one hour long, had 28 points (15:13), but the detector found 197 clips with the moving ball. The total time of the combined clips is 28 minutes of playing.
Most of false clips are just ball transitions, like that:
Technically everything is all right with this clip — there is a ball, the ball is moving, with parabolic trajectory. CV itself cannot filter out that, need to apply some other logic.
The good news — real rallies recognized correctly, but we still need a way to pick them from the garbage. One simple approach could be to use duration — but then we will lose all the aces — they are very quick and short.
Applied to some beach volleyball from Youtube:
Implementation details
The ball tracking algo is written with Python + OpenCV, so I kept going with this stack.
Also, I wanted to view results easily and with convenience, so spent some time building a reasonable UI. Django turned out to be a great choice in terms of supporting both UI and backend.
The main service (aka video-processor, aka vproc) was doing ball tracking, rallies cutting, clips generating. The working with frames turned out to be heavy, so I found a way to make it parallel. Then the 3rd service appeared — it generated rally clips with FFmpeg. I called it Clipper, according to its job. Also, I used Celery to provide communication between the services.
Initially, I made everything local but later switched to Docker to get rid of the dependency mess. So far there were multiple docker instances, I set up a Kubernetes cluster with microk8s, and this decision made the deployment much easier.
Having UI allowed viewing all the clips:
