Truly Bringing Toys to Life

We always wanted our toys to actually fight


One of the most appealing aspects of physical toys is that they just work. There is no boot-up, friends list, matchmaking, or timeouts. You grab them and start playing. They become multiplayer as soon another person walks into a room.

At Fantasmo, we have been experimenting with a digital toy application that replicates the ease of usability of physical toys. In the process, we created Star Wars Arena, an Augmented Reality toys-to-life game. (Disclaimer: We are not affliated with Disney or Star Wars. We just happen to enjoy games that have lightsabers and blasters.)

Star Wars Arena Trailer, a toys-to-life game demo by Fantasmo

The toys were off-the-shelf figures from Hasbro’s Mighty Muggs line. They do not have wireless chips or any other hardware. The mobile devices used were off-the-shelf Apple products (iPhone 6, iPhone 6+, and a iPad Air 2). Everything that happens in the game (e.g., detecting the toys, bringing the toys to life, and connecting the players) happens without any setup by the players.

Star Wars Arena demonstrates a concept we call organic usability where the interaction between the physical and the digital as well as the interaction between users is natural and intuitive. When a player looks at a toy, the toy springs to life. If a player doesn’t have a character to control, they are assigned the next character they interact with. If additional players join, they are automatically connected to the ongoing game. The level of simplicity is suitable for preschoolers.

How we did it

The streamlined user experience was created using a combination of computer vision, ad hoc networking, and distributed control.

The toys are detected by looking at them through the camera of the mobile device. We leveraged Qualcomm’s Vuforia SDK to perform the object recognition. A predefined target is created from a 3D point cloud and a series of 2D images. The target may be included in any application which wants to recognize the object. When a toy is recognized in the game, an event is triggered to spawn the corresponding character in the appropriate location.

The players are connected through ad hoc networking automatically when the app is launched. Ad hoc networks are peer-to-peer which means no infrastructure (e.g., wireless router or access point) is required. Since the game was built for iOS devices, Apple’s multipeer framework was used which supports peer-to-peer communications over Wifi and Bluetooth (Android’s analog is called Wifi Direct). The name of the network is specified as the app bundle ID, and the hostnames are set to the vendor device IDs. Both of these IDs are globally unique, so there will never be a duplicate network or node.

All decisions about the network and the application are made through distributed control. Nodes will take actions without a centralized server by observing the network and polling peers. A node will spawn the characters individually as the toys are recognized. If a node is not already controlling a character when a character is spawned, it will poll the network to determine which characters are available. If that character is available, the node will take control. A node in control of a character will update the network with the movements and actions of its character. If a node is unable to find an available character, it will still receive updates from other nodes so that it can spectate.

Why toys-to-life?

Toys-to-life gaming encompasses games in which physical toys and accessories are bridged into the digital realm. Activision’s Skylanders, Disney Infinity, Nintendo’s Amiibo toy line, and Lego’s soon-to-be-released Lego Dimensions are the prime examples. The category has proven to be immensely popular by generating $2 Billion in sales since the release of Skylanders in 2011.

The current generation of toys-to-life figures have NFC tags embedded into the base of the figure. The toys are brought into the game when they are scanned by a portal, a physical platform with an NFC scanner. While these systems have worked well, they have two inherent drawbacks. First, each toy must have a wireless chip, so a toy has to be specifically manufactured to be compatible with the intended game. Second, the consoles must have a wireless reader either built into the console or sold separately.

Our demo shows that is possible to create a toys-to-life game without either of those restrictions. For the producers, production is cheaper since wireless chips don’t have to be embedded. Additionally, all toys, past and present, can become part of digital experiences. For the consumers, it is not necessary to buy and operate peripherals which saves money and improves the user experience.

Where we go from here

This demo was an early experiment with organic usability. It provided insights into how the user experience is impacted when you move beyond the screen and interact with the physical world. The ultimate goal is to gain to an understanding of how humans can more naturally interact with their environment and each other using digital devices. It is wrong to think that the traditional ways of interacting (e.g., buttons, menus) are going to be suitable. By leveraging the increasingly sophisticated sensors, cameras, and communications of mobile devices, methods can be devised that are more intuitive and transparent to the user.

At Fantasmo, we know that “augmenting reality” is just the beginning. The best experiences will not just augment, they will actually integrate the real world into the digital. The real world objects will actually inform and change the way the virtual content is displayed and behaves.

At Fantasmo, we’re always exploring new ways to create magic in the real world. If you’re interested in working with us (perhaps to bring your toys to life), email us at hello@fantasmo.io or sign up here for email updates.