Winging it —evaluating Gen Z’s need for AI-assisted flight for drone search and rescue missions
We invited Gen Z drone pilots to flex their l33t gaming skills as part of our tech showcase booth at Brainhack 2023 earlier this year. Pilots were tasked to fly around an urban estate and count targets (red cars), as a proxy search and rescue mission. Gen Z slayed it by showing their innate aerobatic skills outperformed AI-assisted semi-autopilot flight. Yet, our semi-autopilot mode helped to bring out the best in our worst pilots. This article was adapted from work done by DSTA interns Tan Hsien Rong, Wes Lee, and Julienne Goh — a fine squad mentored by gamers and engineers from the Air Systems Programme Centre and Digital Hub.
Introduction
Bottom line up-front: The objective of this game-based research was to understand whether AI-assisted semi-autopilot was required to improve the task performance of Gen Z pilots for drone search and rescue operations.
Games are an interesting way to train-up and investigate psychomotor performance and aptitude. Being avid gamers ourselves, we sought to throw in a dash of fun for our Autonomous Drone Tech Showcase booth for Brainhack 2023. We designed two game scenarios for intrepid, curious participants, who were challenged on their ability to conduct a search and rescue drone mission using Microsoft Airsim. Everyone brought their A-game. No gamer egos were spared.
TLDR: Two thirds of Gen Z pilots outperformed on performance measures for both tasks on drone flight control, and target counting tasks with their innate skills, sans assistance. AI-assistance for semi-autopilot flight did help improve poorest performing pilots.
Motivation & Approach
Gen Z’s are curious creatures that are perpetually glued to their screens. When they’re not texting or infinite-scrolling, they are probably gaming. The open-sourced Microsoft Airsim was a natural choice of simulator environment to brew and manifest ideas we had in mind to further understand human-robot interaction, and effects of cognitive load, while keeping the experience fun. t
We especially liked that Airsim offered an immersive experience over primitive 8-bit simulators which might only catch the fancy of the Roblox/Minecraft crowd. This was the only open source software with good baseline capabilities, with out-of-the-box interfaces for drone research, while capable enough for us to design and deploy additional features to understand how our budding drone pilots measured up. You might also be interested to check out past Airsim work by others using Reinforcement Learning to train drones to fly.
Afternote: In Oct 2023, some of you might have caught Microsoft would be pulling the plug on our beloved open source simulator Airsim. 🤯 You may be be thrilled to check out Colloseum, the successor to Airsim.
Methodology and Materials
We used a low-rise urban neighborhood in Airsim as the mission area to test our Gen Z gamers.
Instead of a live mini-map/radar plot to help the pilots localize themselves in the mission area, we wanted to keep visual aids as rudimentary as possible to simulate the scrappiness associated with search and rescue pre-mission briefs during disaster relief. To this end, we provided a static printed visual aid to indicate their start location (green arrow in image below) relative to the mission area.
Participants were tasked to fly along the axis within the geofence (demarcated by outer perimeter of image below). If you were designing a more complex scenario for future research, and needed ideas on search patterns, check out our intern Chun Jia’s research on optimal search patterns.
The game was set up with two participatory scenarios with different degrees of flight assistance, each requiring the pilot to fly through the estate’s roads while counting a fixed number (22) of randomly placed red cars (as a proxy target of interest) among other vehicles in the area. Each scenario started with a drone on ground, with mission timer starting upon take-off. Task was considered complete when pilot completed the flight route and landed the drone. To prevent task familiarity from creeping in, competitive entrants were only allowed one attempt.
Scenario 1 — Manual Flight: Participants used an Xbox controller to fly a simulated quadcopter through a virtual urban landscape to locate and count red cars.
Scenario 2 — Semi-autopiloted ‘click-to-fly’ flight: This station featured a click-to-fly system, reducing the operator’s cognitive load and allowing them to focus on the search task. The drone was human-operated via a first-person view. The operator was required to click an unobstructed space in the scene to command the drone to fly to. Clicking onto obstacles without corrective action would results in a crashed drone. This was implemented with visual-inertial simultaneous localization & mapping (VI-SLAM). Jump to the end of article for a short technical primer.
Motivation for this feature — Click-to-fly on an overhead map (i.e. designating waypoints) is a pretty common feature in games and Command & Control systems. However overhead/satellite maps may no longer be accurate especially after disasters. A stop gap was required prior to the attainment of safe, fully autonomous flight. This motivated the search for additional methods of pilot assistance may be required to navigate regions unstructured obstacles (e.g. collapsed structures). This VI-SLAM based click-to-fly is just one of the many possibilities.
Scenario 3- Fully Autonomous Flight (non participatory): After their flights, participants observed a pre-recorded video snippet of an autonomous drone flight equipped YOLOv7 onboard for target detection to better appreciate how technology can be applied to augment search and rescue missions. As of Nov 2023, YOLOv8 is the latest version of the acclaimed real-time object detection and image segmentation model. The results for the fully autonomous flight is classified 😜 (join us to find out)
We ran pre-event trials on DSTA Engineers as a pilot to uncover latent needs to improve in-game user experience. Hsien Rong and Julienne also developed utils to automate measurement of participants’ skills, as well as UI features for drone health feedback (drone health status, flight modes). Automating aspects of game analytics allowed us to keep the experience fun. Imagine having to go through a post-flight 15 minute research QnA — very party-pooping 😐
Features added (for consideration if you’re running similar games for performance analysis):
- An automatic stopwatch measuring elapsed missions time for task commencement and completion (you’d have to pre-determine what are the start/end triggers)
- A text prompt for participants to report their target count post mission)
- Automatic logging of game performance data into Google Sheets for further crunching of data, with autosave.
- Drone health status and flight modes (to tell us if the game hung unexpectedly, is non-responsive due to software errors, or if the drone is simply hovering and waiting for commands)
Gamer demographics
Following a data clean-up from visitors at the event, we had 52 Gen Z participants aged 13–24 (mean age 18.4 years; SD = 3.499; 30 males, 20 females, 2 gender unspecified) who chose to complete both stations competitively and submit their target count. Only 7 out of 52 participants reported having past experience in operating a drone. For post-game analysis, we omitted participants who (i) did not fall within Gen Z (aged 11–26); (ii) casual try-outs or participants who elected not to complete. The fastest drone pilot who completed all tasks and with the most accurate red car target count (closest to ground truth) was awarded a hobbyist drone.
Results
At the start of each mission (for the unassisted and assisted scenarios), 22 red cars were randomly spawned throughout the map among other cars, with placement confined to roads and front patio of the houses within the virtual estate. Our avid gamers were scored on their ability to accurately count targets and the time they took to complete the mission. Mission timer started upon take-off, and ends when drone lands (or crashes).
Drone Ace — our overall top drone pilot was a male student aged 17, who reported sighting 21 out of 22 targets, in both unassisted and assisted flights. Surprisingly, he clocked a faster mission completion time of 2:16:46 mins with manual flight (scenario 1) over the ‘click to fly’ flight (scenario 2) of 5:06:18.
Performance measure 1 — flight time
TLDR — ‘click to fly’ feature did not significantly improve flight times, but helped to reduce outlier effects on worst pilot
a. Unassisted flight (n = 52) mission times ranged from a blistering fast 1:04:10 minutes to a maximum of 16:14:22 minutes. An interesting observation was that the fastest pilot for unassisted flight fared poorer on target accuracy during this scenario (13 out of 23 targets); but scored full marks on target count during assisted flight at a slightly longer, 3:20:25 minutes. Flight time across all pilots averaged 3:34:274, with 1SD = 2:25:54.
b. AI-assisted ‘click-to-fly’ flight (n = 47) mission times ranged from 2:21:14minutes to 21 minutes. Flight time across all pilots averaged 3:25:01 minutes, with 1SD 1:38:26.
A 5-second improvement with AI-assisted flight isn’t much to shout about. Let’s peel the layers of the onion further!
Different strokes for different folks — do we need software nannies to fly well afterall?
None of our competitive Gen-Z pilots crashed during manual flight.
6 out of 52 pilots crashed during the second run with AI-assistance ‘click-to-fly’ feature switched on, leaving us with 46 data points for comparison.
Only 18 out of 46 pilots (or 39%) clocked faster runs with the AI-assisted semi-autopilot feature, which meant our AI-assisted click-to-fly feature only helped improve flight performance for approx. one-third of the group. This also meant that two-third of the Gen-Zs who showed up had l33t baseline skills and didn’t require any software nannying to get the job done (semi autopilot assistance made it worse). Interestingly, the ‘click to fly’ feature did not improve target counting accuracy, but helped to reduce outlier effects on worst pilot.
Performance measure 2 — target counting accuracy
We observed that the assisted, click-to-fly flight mode did not contribute to a performance improvement. We attribute this to the constraint brought about by the tighter coupling between field of view and platform forward-facing orientation (i.e. it was not possible to quick-glance at a direction different from flight path without introducing jerky and hence inefficient flight dynamics). From our own experience (i.e. COD4 Prestige. Kidding, we also do serious autonomy developments), drones achieved better coverage if programmed/operated to yaw mid flight to allow wider sweeps if there was only one target searching sensor onboard (e.g. like a forward-facing, non-gimballed camera).
A second observation stumped us. Three out of the best five pilots by target counting accuracy achieved better flight times with the click-to-fly feature and no significant difference (+/- 2 count) in target counting performance.
If you grew up on a healthy diet of Call of Duty and Battlefield, then it is not surprising that manual flight is preferred to execute fun maneuvers like circular strafing and scanning to cover a larger area with fewer moves. While aggressive flight maneuvers (extreme yaw mid-flight, hard flares for directional changes) appear fancy, it does have merit in helping a sensor with narrow field-of-view scan a larger area.
Our untested hypothesis from the second observation was good pilots may have adapted quickly to this new mode of mouse-based flight controls (given its similarity to games like War Thunder). Bringing this back to the science & tech, check out this paper on transfer learning of pilot skills.
Discussion
This study sought to explore the impact of AI-assistance on the flight performance of a cohort of Gen-Z drone pilots within a simulated environment. By incorporating both assisted and unassisted flight scenarios in a controlled virtual estate and evaluating performance based on flight time and target counting accuracy, we aimed to discern the potential enhancements attributable to semi-automated flight assistance.
The results indicate a nuanced relationship between AI-assistance and pilot performance. Notably, the introduction of the ‘click to fly’ feature did not uniformly enhance the flight times across the participant group. In fact, the observed data suggest that the AI-assisted flights were not faster on average. The top pilot exemplified this trend by completing the mission more rapidly without AI assistance. This suggests that, for certain individuals, manual control of the drone provides an advantage in speed, possibly due to a more intimate familiarity with the controls or a better strategy in navigating the map.
Interestingly, while the AI-assistance did not significantly impact the average flight times, it did have a notable effect on minimizing the variance among the poorer performers. This finding suggests that AI-assistance might be more beneficial in standardizing performance across a diverse group of operators, reducing the gap between the least and most skilled pilots.
When considering the frequency of crashes, the data revealed that manual flights resulted in a complete absence of crashes, whereas the ‘click to fly’ mode saw a small but significant number of crashes. This could imply that the AI feature, while intended to simplify the operation, may introduce unfamiliar dynamics to the pilots, leading to errors.
Regarding target counting accuracy, the results were counterintuitive; the AI-assistance did not improve the pilots’ ability to count targets. However, a subset of pilots did achieve better flight times with the AI-assistance without compromising their target counting accuracy. This subset could represent individuals who are able to adapt quickly to new technological aids and integrate them effectively into their existing skillset.
The differential impact of AI-assistance on the cohort might also reflect varying degrees of video game experience among the participants. This was omitted from our pre-flight checks given the large variance in video games genres and associated variance in meta-skills (our untested hypothesis is that we learn different skills playing first-person shooters vs puzzle bobble). The familiarity with manual control in gaming could translate to better manual flight performance, as seen in the top performer. This echoes findings in the literature on the transfer of skills from gaming to real-world tasks, suggesting that video game experience could enhance certain cognitive and motor control capabilities relevant to single drone operation.
The study’s insights into the heterogeneity of the impact of AI-assistance underscore the importance of personalized approaches to the implementation of such technologies. While AI-assistance has the potential to democratize skill levels among a group of operators, there is a significant subset for whom traditional, manual controls yield better performance. This dichotomy raises questions about the necessity and design of what we like to deem as ‘software nannies’ for single platform control, or AI-assist features, and suggests that future research should consider individual differences when assessing the efficacy of AI integration into skill-based tasks. On that note, automated flight might be required for operations that involve a single operator overseeing multiple drones performing operations given s/he would not have the cognitive capacity to execute low level controls simultaneously across a larger number of drones faced with each different obstacles in their vicinity.
Future Work
Future studies could expand on the sample size and demographic diversity. Additionally, integrating more complex decision-making and precision tasks (e.g. deliver a first aid kit) for the human operators could provide deeper insights into the performance measures of human-robot cooperation, and shed light on what might be a good reallocation of tasks between man and machine.
What’s next
Tinkering and exploration should not halt at the edge of simulation. The next chapter beckons the bold, the inventive, and the spirited.
This is where you come in.
If you’ve followed us to this para, chances are there’s a gamer and avant-garde test pilot in you waiting for the right opportunity to manifest your ideas, stress test them, and ship it for missions that matter. Come as you are — we don’t offer just a project, we offer a launchpad for your aspirations to take flight 🚀.
— — -
For our fellow geeks — a special technical section looking under the hood for the click-to-fly feature implemented in Microsoft Airsim:
A short primer on Visual-Inertial SLAM
The implementation of a click-to-fly feature for AI-assisted drone flight within a Airsim used Visual Inertial Simultaneous Localization and Mapping (VI-SLAM), which combines visual data from cameras and inertial measurements from the simulated drone’s accelerometers and gyroscopes for translation into navigation (and associated low-level control) commands for drone navigation. The aim is to simplify the drone flight process, allowing for intuitive point-and-click destination selection based on the visual scene an operators looks at, while relying on onboard collision avoidance keep out of harm’s way.
Check out Korea Advanced Institute of Science & Technology (KAIST)’s implementation of Visual-Inertial SLAM to visualize this capability:
Visual-Inertial SLAM Fundamentals
VI-SLAM consists of two primary components: the visual and the inertial measurement units (IMUs). The visual component processes frames from onboard cameras to identify unique features in the environment, tracking their positions across multiple frames to establish a sense of movement and depth. The inertial component uses IMUs to capture the drone’s acceleration and rotational changes, providing data on the drone’s dynamics that complement the visual information.
For implementing the click-to-fly feature on your own, you could have a deeper dive in this interesting paper here. You’d need to consider:
- Sensors: High-resolution cameras and IMUs.
- Processing Unit: A robust onboard computer capable of running SLAM algorithms in real-time.
- Control System: A flight control module that translates SLAM outputs into actuator commands.
- User Interface: A GUI for the operator to input click-to-fly commands.
Broad implementation of Click-to-Fly Feature
(there are alternative ways to do this)
The implementation involves several steps:
- Map Initialization: Upon starting the simulation, the VI-SLAM system creates an initial map of the environment using the drone’s sensors.
- Point Selection: The operator selects a destination point within the simulator’s GUI.
- Path Planning: The system calculates an optimal flight path to the selected point, considering the existing map and any obstacles.
- Flight Execution: The drone autonomously flies to the destination, with VI-SLAM continuously updating the map and drone’s position.
- Obstacle Avoidance: If an obstacle is detected, the system recalculates the path in real-time to avoid collisions.
SLAM Algorithm Details
The core of the VI-SLAM system is the SLAM algorithm, which can be broken down into:
- Feature Extraction and Matching: Identifying key points in visual data and matching them across successive frames.
- Pose Estimation: Determining the drone’s position and orientation by comparing feature points to the known map.
- Map Updating: Adjusting the map with new data from the sensors to account for new features or changes in the environment.
- Loop Closure Detection: Identifying previously visited areas to correct any drift in the map and drone’s estimated trajectory.
Integration with Flight Control
The VI-SLAM outputs are fed into the flight control system, which uses a feedback loop to adjust the drone’s actuators. This integration ensures that the drone follows the computed path and responds dynamically to any changes in its environment (which is picked up by onboard sensors).
Simulator Testing:
The click-to-fly feature is extensively tested in a simulator. If you have the good fortune living near uncluttered, less-regulated airspace, do ensure you test your feature well before live flights. This allows for:
- Safety: Ensuring the system can handle various scenarios without risk (e.g. check that onboard collision avoidance works fine and can handle what you expect the drone to face). Most drones struggle with detecting and avoiding overhead powerlines, for example. In this game scenario we abstracted away the effect of power lines on drones.
- Debugging: Identifying and fixing any issues in a controlled environment.
- Performance Tuning: Optimizing the system for different types of drones and flight conditions. In our case for the purpose of experimentation, we had to hand-tune the controller so drone attitude was less jerky during fully autonomous flight, click-to-fly modes.
