Tello Drone Face Follower
The full 45 Second flying video can be found on YouTube.
The script is on my Github Repo in the tello_face_tracking.py script.
Tello Drone
The Tello drone is manufactured by Ryze Robotics and they have a few different models. These drones are meant to be easy for beginners to fly and there are Android and iOS applications available. But this drone will also stream 720p video is also meant to be programmed, making it a fun platform for computer vision applications.
I decided on the Tello Drone because it is meant to be programmed and has the ability to send streaming video. Perfect!
Inspiration
My favorite site for Computer Vision tutorials and how to apply Deep Learning to Computer Vision is PyImageSearch by Adrian Rosebrock. If you are interested in computer vision and deeplearning I would encourage you to check out his site and the numerous free articles. But also check out his books and courses — they are excellent. I am not affiliated with the site — I just think the resources are great.
The PyImageSearch site has a series of books called, RaspberryPI for Computer Vision and in the Hobbyist Bundle — there is a project that uses a PID controller to control the Pan/Tilt of a camera to follow a face. I did not have the Pan/Tilt device, but I did have a drone. I was interested in taking the same concepts and apply them to a drone.
Assumptions
- Right now the script is setup to assume there is only one face in the field of view to track. You would have to do more processing if there were multiple faces.
- At some point I would like to move this to the RaspberryPi and I would like to use Haar Cascades for the low processing overhead for Face detection.
Tello Python API
Essentially the Tello protocol is a UDP based protocol for sending instructions to the drone. You can find details of the SDK here. DJI also provides a sample Python script to interact with the drone. You can find that script here.
I decided use the DJITelloPy Python package by Damià Fuentes Escoté and so far it has been very reliable for me. I would recommend starting with this package first before diving into the deeper UDP protocol.
Here is a simple example script to see what the API interface looks like:
Application Architecture
You can find all of the code for the project on my GitHub Repo. Look for the file called, tello_face_tracking.py. Right now that repo is a sandbox of files as I learn more about programming the Tello drone.
Much like the PyImageSearch RPi4CV project — I decided to break up the different parts of the application into different processes.
There is the main script which starts all of the sub-processes.
The ‘Tello Flying Process’ handles all of the communication with the Tello drone and receives the video stream. This process will use the PID controllers to determine which direction ( Up/Down and Left/Right ) the drone should move to keep the center of the face rectangle in the center of the video frame. The Tello Flying Process will adorn the video stream with a face box, some information about X,Y error or difference from the centers, and an arrow showing the direction of movement. The process will then use Pipes to send the video frame to the other two processes.
The Video Writer process will read the adorned video frame from the Pipe, and write the them to an mp4 file for later offline viewing.
The Video Display Process will read the adorned video frame and display those frames using OpenCV so you can have real-time feedback and see what is going on.
Tello Flying Process
Below is the function that represents the Flying Process. I wanted to highlight just a few lines:
Line 28 — Create a Tello object to interface with your drone.
Line 30 — Connect to the Tello Drone
Line 36 — Have the drone take off and hover
Line 46 — Read a video frame
Line 119 — Sends the flight command after the function figures out which direction to go to follow the face.
The Haar Cascade used to detect a face, is not the most accurate. One area that I am still researching is how to keep the false positives down. I want to make sure it does not falsely detect a ‘face’ and jump to another location. On line 65 I am attempting to measure if the face detect jumped too far from the previous to keep that jitter down. Feel free to leave comments on how to tune this better.
Video Writer Process
As you can see there is not much to this process. It will use a Python multiprocessing Pipe to read frames from the Tello Flying Processing and use OpenCV VideoWriter to write an mp4 file. This will record the video for offline viewing. Note that the video frame is adorned with the Face rectangle, center dots, direction arrow, etc.
Show Video Process
This process receives frames from the Tello Flying Process and displays them in an OpenCV window so the video can be seen in real-time as the drone flys.
This process also helps to terminate all of the processes. If the ‘q’ key is present, it will set the exit_event which will begin the termination process.
Main Process
The main script setups up the communication Pipes, creates and starts the Processes and waits for them to terminate.
Again see my Github Repo and in particular the tello_face_tracking.py script for the complete implementation.
Exiting the main program
Exiting the main program can be done by either pressing ‘q’ on the OpenCV window. This will set the exit Event which will cause the processes to terminate. You can also ctrl-c which will also cause the processes to terminate. Terminating will send the land command to the Tello.
Getting the script to terminate the processes and make sure the drone lands when it exists was a little tricky. If anyone has suggestions for improvements please leave a comment.
To Infinity and beyond..
As Buzz Lightyear would say.
I still have some work I want to do on this project
- Tune the PID controllers and Speed to follow faster
- Add having the drone move forward and back
- Move to the RaspberryPI and add a GUI to be more demo friendly
The Tello is a great drone for indoor use and as a platform to hack on or teach kids to code. How exciting would it be for a kid to learn to write a script that flew their own drone.
I hope you have been inspired to pick up a Tello drone and start coding it to do something fun.
