Technology for defense

How to build the Eyes of an Autopilot for FPV Combat Drone

10 min readJan 16, 2025

Learn how to create a Video-module for an FPV combat drone capable of capturing video streams from an analog FPV camera, utilizing them for computer vision tasks, and transmitting the imagery to the Flight Controller for further processing (OSD) and video transmission via VTX.

The original photo was taken from the Bloomberg article

This article is a natural continuation of the R&D I have been conducting for over a year, focused on developing an Autopilot for FPV Combat Drones. You can read the earlier articles on this topic, such as How to Build an FPV Combat Drone to Defend Your Country and FPV autonomous flight with MAVLink and Raspberry Pi. Part II.

When I started exploring ways to notify the FPV operator about the autopilot’s current state and flight parameters, I realized there were no available software solutions within flight controllers on Betaflight firmware. This led me to consider a hardware-software solution.

As I evaluated different options, two key factors stood out: First, it was important to maintain compatibility with Betaflight firmware and the popular FPV drone configuration without altering them. Second, the solution needed to have the capability to modify the FPV video feed and the OSD (On-Screen Display) to show the target recognized by the computer vision model, along with other autopilot states and flight parameters.

In this article, I am sharing what I discovered along the way.

Schema

The initial idea was to create a middleman between the FPV camera and the flight controller (FC) to overlay additional controls and messages on the FPV screen while also processing the image for AI-driven computer vision tasks. For this role, I chose a Raspberry Pi along with two dongles: an AV2USB dongle (to retrieve digital MJPEG imagery from the FPV camera) and an HDMI2AV dongle (to transmit the altered imagery to the FC in PAL/NTSC analog format).

To implement a middleman between the FPV camera and the flight controller, I decided to support commonly used interfaces, specifically the JST 3-pin cable. In this setup, a JST-cable connects the AV2USB dongle to the FPV camera, while another JST-cable links the flight controller to the HDMI2AV dongle. Both dongles are connected to the companion computer (Raspberry Pi 3 in our case), enabling it to retrieve/transmit the imagery from FPV camera with video overlay specific to CV tasks.

As shown in the picture above, I soldered the red (5V) wire from the USB cable to the 5V pad on the dongle, along with the red wire from the JST cable. This setup allows the USB cable to simultaneously power both the dongle and the FPV camera.

The ground (GND) wire (black) and the video signal wire (yellow) from the JST cable are soldered to the AV and G pads on the dongle. Additionally, the ground (GND) wire (gray) from the USB cable is also connected to the G pad on the dongle. The USB blue wire is soldered to the DP (Data+) pad, while the USB white wire is soldered to the DM (Data–) pad on the dongle.

Proper soldering ensures the reliable operation of the board.

**4-pole 3.5mm AV2PAL-cable** with **CVBS signal wire** soldered to 3-pin JST Connector

To connect the HDMI2AV dongle to the FC, I used the CVBS wire (the yellow wire from the PAL output) and connected it to a JST cable, which is connected directly to the FC board using the same 3-pin JST-wire dedicated for FPV camera connection. Refer to the wiring diagram above for details.

Only two wires from the CVBS-cable are connected to the 3-pin JST connector. The yellow wire is the signal wire (CAM) and connects to the yellow wire on the other side. The black wire is the ground (GND) and connects to the corresponding ground on the other side. The red wire (5V) remains unconnected, as it is typically used to power the FPV camera. However, in our setup, we power the FPV camera differently.

Let’s proceed further with the software part.

Imports

To implement the video-overlay feature, we will need to import just two packages that will be used throughout the project: cv2 for computer vision tasks, such as capturing frames from the FPV camera, and numpy for performing array operations and data manipulation.

import cv2
import numpy as np

If you haven’t installed them yet, now is the time to do so.

Definitions and State

Sourced from the appropriate files, definitions.py and autopilot.py in the repository Autopilot "BEE" with target following for FPV Combat Drone (Simulator version), this section provides the settings and current state of the autopilot that should be displayed on the FPV screen.

# definitions
video_source = 0
video_full = 1
video_message_limit = 60
camera_width = 720
camera_height = 480
camera_fps = 1

# state
state = {
    'bee_state' : 'OFF', # OFF, READY, ...
    'rssi':0,
    'rssi_msg':'Strong signal',
    'frame': {},
    'video_msg': '[Manual control is ON]',
    'video_msg_countdown':0
}

Definitions — settings for Video-overlay module:

video_source = 0, this refers to the camera connected through the USB port on the RPi, as no camera is installed by default this is very first camera with index 0.
video_full = 1, means the video overlay will be displayed in full-screen mode. Set to 0 for debugging.
video_message_limit = 60, limits the display of video_msg on the FPV screen to 60 frames.
camera_width and camera_height, set resolution to 720x480 pixels.
camera_fps = 1, captures 1 frame per second using OpenCV (cv2).

Autopilot’s State (current status):

rssi_msg, can be set to Strong signal or No signal. Based on this message, the RC label and circle symbol will be displayed on the FPV screen in either red or green, indicating the signal strength.
video_msg, can contain any message to be displayed on the FPV screen until the limited number of frames, counted in video_msg_countdown.
frame, holds the current frame, which can be processed by the computer vision model for autonomous FPV drone operation.

The current status and all field states are updated during the autopilot operation by other components of the application.

Video overlay

The core of this subsystem is the video overlay functions, which add additional components and messages about the autopilot’s status, such as RC signal strength, custom message, scaled target, and a cross at the center for targeting, all displayed on the FPV screen (FPV Googles or FPV Display).

Look at the code and its description under the listing.

# Autopilot's overlay
def draw_rc_auto_status(frame):
    color_green = (0, 255, 0)
    color_red = (0, 0, 255)
    
    color = color_green if state['rssi_msg'] == 'Strong signal' else color_red
    cv2.circle(frame, (50, 50), 7, color, -1)
    
    font = cv2.FONT_HERSHEY_SIMPLEX
    cv2.putText(frame, "RC", (65, 55), font, 0.5, color, 2)

    if state['bee_state'] == 'OFF':
        cv2.putText(frame, 'MANUAL', (110, 55), font, 0.5, color_red, 1)
    else:
        cv2.putText(frame, 'AUTO', (110, 55), font, 0.5, color_green, 2)

def draw_dotted_line(frame, start, end, color, thickness, gap):
    x1, y1 = start
    x2, y2 = end
    length = int(np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2))
    for i in range(0, length, gap * 2):
        start_x = int(x1 + (x2 - x1) * i / length)
        start_y = int(y1 + (y2 - y1) * i / length)
        end_x = int(x1 + (x2 - x1) * (i + gap) / length)
        end_y = int(y1 + (y2 - y1) * (i + gap) / length)
        cv2.line(frame, (start_x, start_y), (end_x, end_y), color, thickness)

def draw_cross_target(frame):
    color_white = (255, 255, 255)
    height, width, _ = frame.shape
    center_x, center_y = width // 2, height // 2

    draw_dotted_line(frame, (center_x - 50, center_y), 
                     (center_x + 50, center_y), color_white, 2, 5)

    draw_dotted_line(frame, (center_x, center_y - 50), 
                     (center_x, center_y + 50), color_white, 2, 5)

def draw_scaled_target(frame):
    color_white = (255, 255, 255)
    rect_size = 50
    height, width, _ = frame.shape
    center_x, center_y = width // 2, height // 2
    top_left_x = center_x - rect_size // 2
    top_left_y = center_y - rect_size // 2

    center_region = frame[top_left_y:top_left_y + rect_size, 
                          top_left_x:top_left_x + rect_size]
    scaled_region = cv2.resize(center_region, (rect_size * 2, rect_size * 2), 
                               interpolation=cv2.INTER_LINEAR)

    overlay_x_start = width - rect_size * 2 - 20
    overlay_y_start = 20
    frame[overlay_y_start:overlay_y_start + rect_size * 2, 
          overlay_x_start:overlay_x_start + rect_size * 2] = scaled_region

    cv2.rectangle(frame, (overlay_x_start, overlay_y_start),
                (overlay_x_start + rect_size * 2, overlay_y_start + rect_size * 2), color_white, 1)

def draw_video_message(frame):
    font = cv2.FONT_HERSHEY_SIMPLEX
    color_white = (256, 256, 256)
    
    if state['video_msg'] != '':
        cv2.putText(frame, state['video_msg'], (43, 80), font, 0.5, color_white, 1)
        
        countdown = int(state['video_msg_countdown'])
        if countdown < video_message_limit:
            state['video_msg_countdown'] = countdown + 1
        else:
            state['video_msg'] = ''
            state['video_msg_countdown'] = 0

Function draw_rc_auto_status(frame) displays the RC signal strength and autopilot mode on the video overlay. It uses a green or red circle to indicate whether the signal is strong or weak. It also shows the flight mode: MANUAL in red if the autopilot is off, and AUTO in green if it is active.

The function draw_dotted_line(frame, start, end, color, thickness, gap) draws a dotted line between two points on the frame. It calculates line segments based on the specified gap and thickness, drawing small line segments to create a dotted effect.

Another function, draw_cross_target(frame), draws a crosshair at the center of the video frame. Using draw_dotted_line, it creates a pair of horizontal and vertical dotted lines intersecting at the center. This visual aid helps track the central point of the FPV screen (target direction) during FPV drone operation.

The draw_scaled_target(frame) function extracts and enlarges a central region of the FPV video feed, displaying it as an inset in the top-right corner. It captures a square section, scales it up, and overlays it onto the frame. This provides a more detailed view of the target.

The draw_video_message(frame) function is responsible for displaying text messages on the video overlay. If a message exists in the system state under the video_msg field, it is shown below the RC signal strength. This includes a countdown timer to control the message's duration and clears it once the timer exceeds a preset limit (default: 60). This message can display any custom text retrieved by the video-overlay feature for the FPV goggles screen. It’s useful for displaying various stages of the Autopilot's operation.

Run point

The main() function implements all necessary operations for the Video-overlay feature, handling the initialization and execution of other functions to draw additional controls and messages on the FPV screen.

# Main function
def main():
    cap = cv2.VideoCapture(video_source)

    if not cap.isOpened():
        print("Error: Could not access the camera.")
        exit()

    cap.set(cv2.CAP_PROP_FRAME_WIDTH, camera_width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, camera_height)
    cap.set(cv2.CAP_PROP_FPS, camera_fps)

    if video_full:
        cv2.namedWindow("BEE", cv2.WND_PROP_FULLSCREEN)
        cv2.setWindowProperty("BEE", cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)

    print("Press 'q' to exit.")

    while True:
        ret, frame = cap.read()

        if not ret:
            print("Error: Could not read frame.")
            break

        # Save current frame to state for
        # Computer Vision tasks
        state['frame'] = frame

        draw_rc_auto_status(frame)
        draw_scaled_target(frame)
        draw_cross_target(frame)
        draw_video_message(frame)

        cv2.imshow('BEE', frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    cap.release()
    cv2.destroyAllWindows()

The main() function initializes a video capture device (e.g., an FPV camera which we connected with before) using OpenCV, configuring its resolution and frame rate. It optionally sets the display window to full-screen mode if video_full is enabled (set to 1). The function enters a loop to continuously read video frames, updating a shared state dictionary with the current frame for computer vision tasks.

It consequentially applies several Video-overlay functions (draw_rc_auto_status, draw_scaled_target, draw_cross_target, draw_video_message) to display controls and messages on the FPV screen.

The main() function, as implemented above, is suitable for experimental purposes only. For integration into Autopilot, this code would be moved to a separate thread and managed by the main Autopilot process. However, that is a topic for another story.

Launch and usage

Once all the wiring and soldering are completed, I powered up the FPV drone along with the Companion Computer installed on it. I then ran a Python script to verify that the video module is functioning correctly, successfully transmitting the FPV video feed from the FPV-camera to the flight controller and subsequently to the FPV Goggles via VTX.

**FPV display** with **Video-overlay** and OSD-screen on it

As shown on the photo above, everything, including the OSD, video feed, and video overlay, are displayed on the FPV screen correctly. The default PAL resolution of 720x480 pixels results in a less-than-ideal image quality. However, this level of resolution is quite common for FPV drones with analog FPV camera on board.

UNCLASSIFIED_20250109_OP_TI_FPV_VIDEO_OVERLAY_v1.0

Sometimes, I like to add a touch of mystery to my articles, and this one is not exception. For a creative twist, I have used NATO-style military notation to highlight the public nature of these files. Enjoy the ride!

Full script shared as a gist on GitHub — Video overlay for FPV drone with Computer Vision-based Autopilot.
Repository where this Video-module can be used and would be helpful — Autopilot for FPV Combat Drone on Betaflight (Empty version).

Drop your thoughts in the comments about what this NATO-code (above) might contain. Intriguing, isn’t it? Consider it as a challenge!

Support Ukrainian army

It is the third year of Russia’s full-scale invasion of Ukraine — a challenging and arduous fight for independence and the preservation of lives. As many of you have heard from international news, this struggle remains heavy.

Many of us continue to support Ukrainian defenders through financial contributions, donations, and by providing essential components. I encourage you to do the same to stand against this global evil.

Use this article as one of the sources to help you build your own autopilot system for an FPV Combat Drone and supply your finished devices to a trusted contact within the Ukrainian Armed Forces.

Together, we can make a difference.