DIYRobocars — Move it: Story Of Self-driving Hackathon #1
Move-it Hackathon has successfully ended with the hacking of Honda Civic and a “raw” chassis under the joint efforts of 20+ global engineers.
It was indeed fun and productive five-days with real hands-on work and some staying-up-late at night. Learn, communicate, inspire and hack, a lot of gains and moments during the process. Everyone was so dedicated and focused during the hackathon. Some outcomes started to reward our hard work. It’s a meaningful thing to contribute together to the engineering challenge in self-driving.
Here is a blog from one of the participants Adam Shan. It contains some details of what was going on during the Five Days, Enjoy
Adam Shan,
Deep Learning & Computer Vision, Lanzhou University
Nano Degree of Udacity, Core Member of Move-it Community
March 17th, 2018
My records of Move-it Hackathon
The outcomes of Hackathon and how to achieve gesture recognition in Five Days
My name is Adam. I’m studying for a master degree and majoring in computer vision at Lanzhou University. Meanwhile, I’m a student of self-driving car engineer project provided by Udacity. I feel excited to participate in this Hackathon, coding with more than 20 self-driving engineers overseas to remodel a self-driving car.
I got to know about the activity in the community of Udacity. The host advocate to build an affordable development platform for self-driving car engineers, which attracted me a lot. My groupmates and I are all fanatics and contributors of open source software and community. We are engaged to solve some common issues about autonomous vehicles and try to benefit more developers and startups through the event.
Before this event, we discussed some urgent issues in the self-driving car development for individual developers on slack and summarized three goals as follows:
CAN-BUS Hack;
implement an on-site test self-driving car with basic tracking and obstacle avoidance capabilities;
Traffic police gesture detection and recognition.
Overall, obtaining a car that can be controlled is primarily needed. However, automotive companies do not open their CAN-BUS interface to individual developers and small teams. Open-source CAN control projects (such as OSCC) only support several vehicles. It’s a common problem at present. In this hackathon, the Hacking object is a prototype car — we called Coffee-Car and a Civic, provided by the organizers. After that, we are supposed to get the entire top-level software stack including algorithms for sensing, positioning, decision making, controlling, etc. The most important goal is to make the testing car self-driving. There are two relatively mature open source solutions available for us, that is:
Apollo
Autoware
Both of the open source frameworks are capable of implementing self-diving car in a short time. Personally, I am more familiar with Autoware (and we use the Autoware as the top-level software stack in the hackathon).
The last goal was proposed because Beijing recently proposed a new rule for self-driving road tests (“Content and Methods for Beijing Automated Driving Road Test Capability Evaluation (Trial)”) that clearly stated that road test vehicles should have the ability to understand traffic command gestures. At present, the traffic command gesture recognition does not yet have a open source solution. It is obviously necessary for the autonomous vehicle to understand the eight commanding gestures of the traffic police in China.
After clarifying the goal of hackathon, I made some preliminary preparations for these issues. My major is computer vision. Obviously I can contribute to the traffic police gesture recognition project. Besides, I have read the source code of Autoware and have rich experience in ROS development. Before I went to Guiyang, I was full of confidence.
The Hackathon goes the same as I expected — everyone is so awesome! A group of engineers from different countries, covering all directions in the area of self-driving car, completed the task subdivision in less than a few hours at the first meeting. The tasks were divided into groups and the whole process was very liberal and democratic. All people are roughly divided into three groups: CAN-BUS hack group, Autoware group, traffic gesture recognition group. Among them, the CAN control group needs to hack into the Coffee-Car and the Honda Civic CAN-BUS, control the throttle, brake and steering of the vehicle, and finally use the twist message from the Autoware group to control the vehicle. The Autoware group needs to complete the integration and testing of the entire system, including the driver of various sensors (in this Hackathon we use the VLP-32c LiDAR and cameras), collecting and plotting the 3D map of the test sites, drawing waypoint, calibrating the camera and Lidar, testing the system and so on. The gesture recognition group needs to collect traffic police gesture dataset, train deep learning model and wirte gesture recognition ROS nodes. I am a member of the gesture recognition group.
Through some previous studies, we have found that there is no open source traffic police gesture recognition solution, the only several papers did not open its datasets and code. There are roughly two ideas to detect traffic police gestures based on vision: The first is to perform pose estimation on the traffic police to obtain skeleton information, and then input the sequence skeleton information into a model, and then classify the sequence gesture. The second is to directly skip pose estimation and input the sequence of motion images into a model to recognize the gesture. Regardless of which approach is adopted, there are two important prerequisites: first, we have to distinguish the traffic police from ordinary pedestrians; second, we need to collect gesture datasets. Therefore, the first task is to complete the collection and preparation of relevant data sets.
For traffic police identification, we wrote a python script to capture 10,000 images containing traffic police keywords in the network and used YOLO2 to detect human in the images, so that all the people in the picture were cut off and saved separately. Through the manual annotation of our own and local university volunteers, we have obtained a dataset for traffic police identification, including more than 10,000 images of Chinese traffic police and more than 10,000 images of pedestrians. Through these data, we trained a convolutional neural network to identify traffic police with pedestrians.
Since that there is no relatively standard public traffic gesture dataset, we have to collect it ourselves. We used six cameras to photograph more than 180 groups of traffic police gestures from more than 20 volunteers in different directions, and also used two laser radars to record the point cloud data. We used the green screen as the background when collecting data so that the user can fill the background to do data augmentation. We manually tagged the point-in-time files and used the YOLO2 to extract all the traffic gestures from the video at the rate of 30FPS and save them as images. Eventually, we get a sequence image data set of 1700000 traffic gesture images.
After getting the data set, we started the training of the deep learning model. We finally chose to use image instead of the skeleton to train the model for that pose estimation is a computationally intensive task. The deep learning method can directly use the original image as input to create an abstract representation by the neural network itself. Due to the limited activity time, we failed to successfully train the sequence model, but we used Inception-v4 directly for feature extraction and gesture recognition for single-frame images.
The last step is to write a ROS node to deploy the model into Autoware. We subscribe the output topic of the pedestrian detection module in Autoware (still YOLO2) and publish the detection results to the /police_gesture topic.
Finally, together with the other teams, we completed all the set goals and some additional goals to make the car self-driving! Thanks to the organizer Pixmoving and Udacity. I am very proud to have participated in this hackathon. Such offline Hackathon gave us a good chance to control a real car, know more about the self-driving technology stack, and get acquaintance with more self-driving car engineers. I wish that there will be more similar activities in the future.
Hack with love!
