Is there a Way to Transform a Toy Drone into a Racer?
The following story is about an experiment halfway between a game and a technology scouting activity that Emanuele Di Saverio and I did during our spare time at frog, and that eventually ended being the main attraction at frog’s Salone del Mobile party in 2016. It involves designing a game, hacking some drones, and transforming a single-user toy into a multiplayer game platform.
The background
I bought my first Parrot drone on Amazon. I’ve always been attracted to high tech toys, but also concerned about their price. So when I found one at a bargain on Black Friday — and after having watched tons of videos on YouTube - I decided to buy it to play with my kids. My first reaction was “wow!”. Parrots are really fun to play with. At least for the first ten minutes.
While I have no problems admitting that they are extremely sophisticated and that the experience of driving them using a mobile phone is fantastic, I also found no purpose or challenge in driving one: to increase its longevity, I needed to invent something.
The concept
My struggle with the toy drone became sort of an obsession and my one and only conversation topic on coffee and lunch breaks until my cry for help finally reached Emanuele Di Saverio, who accepted the challenge and started helping me find a way to transform the drone. We agreed on two key pillars:
- A game isn’t a real game if you cannot share the fun
- Some competition is needed to spice things up
Before starting the actual hack of the drone, we tried to come up with a multi-player game concept based on Parrots, and to check its feasibility. The first idea was building a digital-physical game based on the same mechanism of World of Tanks: many drones compete in an arena, and each one tries to destroy the others while avoiding being destroyed. In our arena, the tanks from the video game would be physical drones, while the weapons would be totally digital: a sort of augmented reality war.
The second idea came from a classic game that features a lot of competition, speed, a dash of good old violence and iconic characters: Super Mario Kart. Why not replace the open arena with a race track and the goal of destroying the opponents with being faster than them, maybe using a few tricks to gain a small advantage? The idea was good enough for us: it was time to check if it could actually become real, and to get some funding to develop it.
The validation of the concept
Parrot drones are shipped with an API that allows developers to build custom applications on top of them. The API for the mini-drones allows you to completely take control of the machines and their small camera. Building a POC on top of that has been relatively easy for Emanuele. The big problem was in networking: if you want to build a multiplayer game you need all the clients to be able to talk to each other and to an orchestrator server that dictates the rules. This seemed impossible based on how Parrot mini-drones are distributed: once you switch them on, each of them creates its own wi-fi access point, and the controller (the smartphone) needs to connect to this access point in order to be able to actually control the drone. If a smartphone is connected to a wi-fi network, and this network is not able to access the internet, then the drone will never be able to send and receive signals to/from the game controller server.
This problem seemed a blocker, and also Parrot says that it’s not possible to connect a mini-drone to a router. But then Emanuele found a way to use telnet to connect to a mini-drone and, once in, edit network configuration. Mini-drones run a very stripped-out light version of Linux, so editing network configuration is not hard at all.
We were at a good point, the only thing missing was how to make a drone able to recognise its peers by using the on-board camera. We tried different solutions - shape recognition, colour recognition, etc. - but in the end, considering the poor quality of the drone’s camera, the most reliable solution proved to be the usage of fiducial markers, huge markers (changes in light conditions proved to be the first enemy in colour and shape-based detection methods, and are the reason why we ended up having large markers).
At this stage we were able to control more than one drone, we were able to make them recognise each other using the fiducial marker, we had a server able to receive messages from the drones and relay important game messages to the others. We were even able to shoot one drone, and to make it spin when hit.
Even if it was still at an early stage prototype-level, the POC proved that the idea could be implemented. It was time to get some funding. The following is one of the test drives we held internally at frog to pitch the idea. As you can see, markers are glued to the top of the drones.
Once we had secured some help from the studio, the real adventure began.
The development
The overall architecture of the game is quite simple: a server gets and sends messages from/to the drones. The server doesn’t know if a drone has been hit by one of its peers until one of the drones claims the shot. In this sense, the game is not really hack-proof, since hits are claimed by the clients, but this was the only feasible way. Since we were developing the clients, we were sure that no one would have ever try to trick the system. The development of the Android controller App took a lot of time in terms of optimising performances. In the following video, shot during our speech at Droidcon 2017 in Turin, Emanuele explains some technicalities and highlights all the technical solutions he found to work around all the problems.
While Emanuele was fighting with the development, Federico Bressan, industrial designer, and Elena Marengoni, visual designer, joined the team and started helping to build the real experience: the design of the drones, the positioning of the markers, and the construction of the race track.
We designed and 3d printed a “cover” for the drones with a dual goal: housing the fiducial markers in a more fashionable way, and making them look more aggressive. Also, during our tests we realised that the lines we had drawn on the floor to highlight the shape of the track weren’t visible enough, and we decided to build some physical barriers around the track.
Once everything was ready, we started performing some tests, to fine-tune the game options.
During these tests we took some important decisions. For instance, we limited the maximum speed of the drones to make them easier to drive and lower the difference between skilled and unskilled players. We also added some obstacles along the track and a “bonus cube”: each of the drones’ sides had a different marker, and depending on which one was hit, the hitter drone would receive a bonus (like maximum speed increased by 10% for a limited amount of time), a malus (maximum speed decreased), or an advantage (maximum speed of all other drones decreased).
We were almost there. Once we started testing with more than two drones, we understood that we needed a more powerful wi-fi access point (we ended up buying a top of class 5Ghz AP), and continued improving the game mechanics and the performances until the very last minute.
The tecnology
To build the game we used different tools: the controller App is Android based, developed using OpenCV, ARToolkit, Kotlin. More details on the path we followed to reach this setup are in the slides of our speeches, and in the video. The server is a simple NodeJs server that uses Socket.io to talk with all the clients.
As mentioned above, the drones’ covers have been designed and 3d-printed in-house, while the track delimiters have been made out of recycled and recyclable cardboard.
The entire code-base (both the client and the server) is available for free on frog’s Github.
The event
After a few months during which we dedicated all our bench time and sometimes overtime to this project, we finally got our chance when frog Milan decided to feature our game as the main event of the Salone del Mobile frog party in April 2016. During that single night, over one hundred people played and had fun with us.
The post-event
During the following few months, we published an article on our journey on Design Mind and we talked about our experience both at CodeMotion Milan 2016 and Droidcon Italy 2017.
The outcome
This has been a really exciting and instructive project for a few different reasons: one, we’ve been able to create something from scratch, and to prove that it was working; two, we managed to create a real multi-disciplinary team (technology, visual design, industrial design) that worked together very closely. Three, we had a lot of fun.
So here’s what I have learnt from this project:
- If you want to pitch something, a prototype is better than a keynote;
- You don’t need a perfect prototype, showing the experience is enough;
- Telling the story of the product is important, especially when you try to get funding;
- I really like open technologies, things that you can open, hack, tinker. Being able to create something that is “yours” out of a commercial product is great.
My name is Simone Lippolis, after spending almost ten years as Design Technologist at frog, I am now with Cisco, as a Data Visualization Expert. This article is part of my online portfolio that you can access at: simonelippolis.com.
