Quickdraw detection via video: climbing gym analytics for climbing gym owners.

Left image: examples of augmentation for object 1 (quickdraw with rope), left second image: example of a target quickdraw with rope (object 1), third left image: example of a target quickdraw without rope (object 0), fourth left image: examples of augmentations for object 0 (quickdraw without a rope).

In this article, I would like to describe a system for quickdraw detection in a climbing wall. Sport climbing is an extreme sport and linked with a high risk for sportsmen. Therefore, the safety is important for those who would like to build their own high wall climbing gym: we consider sport climbing scenario where an athlete climbs with a rope and use quickdraws during the ascent. Unfortunately, there is a lack of a system which could automatically track climber’s activities: some research is done towards solving safety system via sensors placed into the harness[1], but they didn’t consider analysing quickdraws to track the activities of a climber.

By exploiting video data, we would like to describe how computer vision and machine learning could be used to detect the objects of quickdraws in an artificial climbing wall and further, could be processed to detect the activities of a climber.

Object detection is a powerful tool to detect objects from images: many convolutional neural networks are developed for this purpose. By applying transfer learning, one can solve the problem of object detection: for example, a system could detect a car or a pedestrian in a video. We are focusing in a system, which detects quickdraws automatically. Quickdraw is a special device in an artificial climbing wall for lead climbing: the example of it is shown in figure 1:

Figure 1. Example of a quickdraw in an artificial climbing wall.

Furthermore, it is an additional source of information to detect the climbing activities in lead climbing: if the quickdraw has a rope inside, that indicates that someone occupied the route and route can not be used now. The example of a rope inside a quickdraw is illustrated in figure 2:

Figure 2. Example of a quickdraw which is clipped with a rope in an artificial climbing wall.

We created an object detection problem for quickdraws detection, where convolutional neural network (CNN) detects 2 objects of a quickdraw in the climbing wall: ‘with a rope’ and ‘without a rope’. For this purpose, we collected the data for two types of objects, where object 0 is represented by images of quickdraws witout rope on the wall, and object 1 — by images of quickdraws with clipped rope inside. We consider a scenario where the system tracks only the target quickdraw and the target one is outlined from the others by the coloured yellow tape (see the differences in figures 3 and 4).

Figure 3. Object 1. ‘Target quickdraw’ outlined physically on the wall by yellow tag.

Figure 4. Object 0. Quickdraw without yellow tag.

To test the system, we created two datasets: as a training data, we choose the images generated from video of 5 climbs of one person, and as a testing data, we used images, generated from videos of 5 climbs of another person. The route for the test set and train set was the same, the difference was on camera location (we filmed them from different locations). The target quickdraw has the same outliner (yellow tape). We manually annotated images on all frames for test and train set in a following way: for each frame, we generated points object location and object id. Then, we applied augmentations on the test set for every image: we slightly shifted, rotated, changed the colors and brightness of the initial data. In that way, we generated 24000 images of object 1 (quickdraw with rope) and 12000 of object 0 (quickdraw without rope). We applied transfer learning on keras-retinanet to detect target quickdraw with or without rope, see example how to apply keras-retinanet for object detection in my article. After transfer learning, we tested the model on test set.

Test set has 5 climbs. For each of the climb, we plot the trajectory of a person who climb (gray dotted line in figure 5), so that it is more clear when the activity of rope pulling happens.

Example of person detection in video for trajectory estimation of a climber (gray dotted line in Figure 5)

Red horizontal line shows prediction of keras-retinanet after transfer learning: it shows the change of prediction from object 1 to object 0 (or from object 0 to object 1). We can see, that when the person starts or finishes to climb (gray dotted line goes up or down on the top of figure 5), there are several red horizontal lines, and it indicated uncertainty of the predictions.

Figure 5. Keras-retinanet predictions for test set.

In order to improve prediction of the network, we applied approximation method for predictions: the object is predicted to be object 0 only if the next 90% of the next frames generated from 3 seconds video contain predictions of object 0. In this way, we removed the uncertainty of a network (bottom figure in figure 5).

We can see that the proposed system could detect the target quickdraw quite accurate.

As a summary, we can say that the system can be used for the next purposes:

1. detect quickdraws on a wall automatically;
2. detect target quickdraw on a wall automatically;
3. detect whether the route is occupied by climber.

Read more about the topic in the article: https://dl.acm.org/doi/10.1145/3422844.3423058

Resources.

[1] Andrea Tonoli, Renato Galluzzi, Enrico C Zenerino, and Diego Boero. Fall identification in rock climbing using wearable device.Proceedings of the Institution of Mechanical Engineers, Part P: Journal of Sports Engineering and Technology, 230(3):171–179, 2016.