Autonomous Car Chasing: From thesis to a conference paper
It was a year ago (summer 2019) when I was deciding what topic to choose for my bachelor thesis. At the time I was finishing an internship at the European Space Agency where I worked on mineralogical machine learning research¹. However, I also knew that at my university there is a lot of research with self-driving cars in collaboration with Toyota. While mineralogical research that could be possibly used in future planetary missions is very important, self-driving cars are not only exciting, but also tangible and presentable. So I contacted my former research supervisor from my university whether he had any research ideas in this area. Among many proposed, I was most excited by Autonomous Car Chasing.
Why Car Chasing?
When you look at autonomous cars in real-life, you will see that they are most often used in situations when the driving is too boring for a driver. They are used on a highway in a traffic jam or in a convoy driving at a near-constant speed. However, regular traffic forces the drivers to react to many unexpected situations and perform dynamic maneuvers. The proposed car chasing scenario, in which an autonomous car chases a non-cooperative vehicle that actively tries to drive away, is so extreme that an autonomous vehicle with car chasing capabilities can surely handle regular traffic.
New challenging dataset
The first task was to create an autonomous car chasing system and test it in CARLA. To perform the experiments we first collected a new challenging publicly available CARLA Car Chasing Dataset collected by manually driving the chased car. We created a baseline algorithm, which detected the pursued car and then basically drove straight after it. This approach had a huge flaw. It wasn’t considering the surrounding environment while driving. It was only focused on the pursued car. Therefore, this system crashed a lot. We realized our system needed an understanding of the surrounding environment. We considered adding semantic segmentation neural network, but we already had a neural network detector. We needed to analyze as many frames per second as possible and also had a very limited computational power (embedded system on a RC car). We needed a fast and effective solution.
Dual-task neural network
During inference, an image is passed through the network just once. The network provides an object detection as well as the semantic segmentation outputs. While the training is slightly slower than training a single-task neural network, the extra cost during inference of the proposed architecture is negligible. The segmentation output provides a semantic map of the input image consisting of 10x10 cells of two classes: a drivable surface and a background, see the image below.
The neural network shares the same backbone for both tasks — a 53 layer feature extractor called Darknet-53⁴. Attached to the feature extractor are two sets of layers — one that gives the output for the object detection and the other that gives an output for the image segmentation. The architecture of the neural network is depicted in the image below. The network is trained by alternating optimization — in every second batch, the network is optimized only for detection, while the segmentation is optimized in the remaining batches. The neural network uses different loss functions depending on the batch².
Experiment results
First, we tested the system using a difficult subset of the CARLA Car Chasing Dataset. We observed, that the full algorithm performed significantly better than other versions of the system that did not use coarse semantic segmentation. It achieved almost 10 percentage points higher drive completion on average than the next best-evaluated version. We also showed that the dual-task neural network system is more resistant to detector miss rate (when the detector failed to detect the pursued car).
Then, we performed several live tests under different weather and lighting conditions. The system was tested on an empty roundabout as well as in a residential area as depicted in the image below. The autonomous system followed the other car smoothly without jerky movements. For the most part, it was able to successfully chase the other vehicle. It was maintaining the desired distance when the pursued car was driving in a straight line. If the chased car stopped, so did the autonomous system. A limitation of the system comes from its current reactive nature, which in certain rides affected the ability to make a U-turn on a narrow road.
Conference
The results of the novel proposed approach were clear and before the bachelor thesis was even finished, we knew there was a potential for publication. After finishing polishing the thesis, we started its transformation into a research article. A multiple-page section in the thesis easily turned into a couple of lines in the publication. Unlike a thesis that mainly serves as proof of your understanding of the topic, we turned it into something that could be read by other people. We decided to publish to a workshop associated with the third most prestigious conference on computer vision called European Conference on Computer Vision (ECCV). The main advantage of a workshop is its short processing time. In two weeks, we knew our paper was accepted.
Discussion
This paper isn’t the first (nor second) paper I have published. I am also not flying to the other side of the world to present it like with my first publication (FG 2018 in China) because the conference is only online/virtual due to COVID-19. That said, I am very proud of the paper. The project is very exciting and hopefully, other researchers (or students at FIT CTU) will attempt to improve on the method.
I believe that my motivation for the topic was the main reason for the success of the thesis. The thesis topic was exciting and also purposeful and I wish you to find a topic that you can get excited about. I hope there can be a collaboration between Center for Machine Perception (CMP) at FEL CTU and FIT CTU. If you are interested in autonomous driving or computer vision, CMP is a place to be. I can also definitely recommend my supervisor Jan Cech, who discussed the thesis progress with me on a regular weekly basis.
Conclusion
We have developed a system, capable of autonomously chasing another vehicle, using the novel dual-task network that concurrently detects objects and predicts coarse semantic segmentation. The proposed system was extensively tested in CARLA simulator using a new challenging publicly available (at our GitHub) chasing dataset and on a real sub-scale vehicle platform (video available on youtube).
Despite the simplicity of the proposed system, it shows robust chasing capabilities by using only information from a single RGB camera. One of the system limitations is its reactive nature. We believe that the system could improve by using a more sophisticated trajectory planning algorithm that would include predictive modeling of the chased car.
We hope to inspire other researchers or students looking for a thesis topic to try and improve our methods. We believe that testing an autonomous driving system under extreme conditions is an important step to creating public trust in self-driving cars. Ultimately, extreme challenges like the DARPA challenge, or our autonomous car chasing is what will drive innovation forward.
References
[1] P. Jahoda, I. Drozdovskiy, F. Sauro, L. Turchi, S. Payler, L. Bessone. Machine Learning for recognition of minerals from multispectral data. Pre-print, arxiv:2005.14324. 2020
[2] P. Jahoda, J. Cech, J. Matas. Autonomous Car Chasing. Proceedings of the European Conference on Computer Vision (ECCV). 2020
[3] P. Jahoda. Autonomous Car Chasing. Bachelor thesis, Czech Technical University in Prague. 2020
[4] J. Redmon, A. Farhadi. YOLOv3: An Incremental Improvement. Pre-print, arxiv:1804.02767. 2018
