BeyondVision | Spatial Computing comes to the entire Apple ecosystem
Just like vision often eclipses our other senses, Apple’s Vision Pro stole the lights during WWDC. But glaring signs announce a much wider ambition for Apple’s spatial computing platform. The revolution is staged outside of the augmented reality headset. From the iPhone and Watch to the less conspicuous TV and Airpods, Apple has been embedding spatial awareness capabilities in its devices for years. But what is the purpose of this? And what does it reveal about Apple’s spatial computing strategy?
After Tim Cook’s presentation, something felt amiss. Surely it could not be just that. Spatially aware technologies are nothing unique to the Vision Pro, but rather something that Apple has been including in its devices since at least 3 years. If you’ve ever wondered why Apple would include cutting-edge spatial chips in stationary devices like its HomePod speaker, you are not alone. But with its latest announcement, everything became crystal clear.
The success of spatial computing is intertwined with its availability on all of Apple’s devices, and there are signs that this has been a deliberate strategy for the company. Are you ready to discover what happens when it all comes together? Let’s unravel the mystery of Apple’s spatial computing strategy!
The future is multimodal and gradual
Spatial computing promises to enrich our world with digital information and interactions that are seamlessly integrated with our physical environment. However, this vision cannot be fully realized if we rely on headsets as the sole interface for spatial computing. Wearing a headset all day long is neither practical nor desirable for most users. Putting on a headset for a brief interaction involves intentionality which generates friction. We need a more flexible and natural way to interact with spatial computing, one that adapts to different levels of immersion and comfort. This is where gradual interaction comes in.
Gradual interaction is the idea that users can access spatial computing through various devices, depending on their needs and preferences. A headset provides the most immersive and realistic experience, but it is not your only option. Given the choice, you surely would opt for less intrusive and more portable devices, such as iPhones or Apple Watches, to access specific information or actions that are spatially aware.
For example, pointing your iPhone at a plant displays its watering history and soil moisture level. Similarly, your watch could know when you gesture at your TV and adapt the volume. Reach for your door knob and your watch will remind you to take an umbrella as it is forecasted to rain in the evening. Walk past your mail box and your Airpods would chime in to let you know that your package delivery has arrived. Those are the promises of multimodal spatial computing.
This is achieved by a common platform that syncs all your spatial experiences — iCloud — allowing you to switch between devices without losing context nor continuity. Gradual interaction enables users to reap the benefits of spatial computing in a more convenient and accessible way, while still preserving the possibility of full immersion when desired.
Glaring signs to an imminent transformation
A long-term investment
Why would Apple include advanced spatial chips in a watch or a speaker? This can only make sense in the light of a grander purpose.
Apple has a secret weapon for spatial computing: the U1 chip. This chip is a hidden gem that has been quietly added to every iPhone, Watch, and HomePod since 2020. It is a chip that can enable amazing spatial interactions using ultrawideband radio, a technology that precisely measures the distance and direction between devices. It is a chip that has no clear purpose yet, but that holds the keys to unlocking the full potential of spatial computing.
Let’s take a look at a few examples:
Your iPhone is a magic wand
HandOff already allows you to tap your HomePod with your iPhone to transfer any music currently playing. This proximity interaction, handled by U1, might soon turn into a long range one, and affect many more “dumb” devices than you would expect. With the U1 chip, your iPhone and HomePod can locate each other. So you might be able to achieve the same HandOff functionality without moving from your couch. Just point your phone in the direction of your speaker.
To avoid unsolicited interactions, the last missing ingredient is a simple trigger, which could well come in the form of the rumoured Action Button. This would deserve its own article.
Thinking more systematically, nothing stops your iPhone from controlling other devices that are not equipped with a U1 chip. For instance, if you have previously mapped your space with ARKit, it is sufficient to know the iPhone’s position within that space to guess what you are pointing it at.
You may be able to dim the lights, or trigger specific shortcuts by pointing your iPhone in the direction of a bookshelf (pop-up your reading list), or a coffee machine. Could you imagine any long-range spatial shortcuts that would be useful for you?
But wait! ARKit also has a map of the outside world.
To me, a future where you point your iPhone at a restaurant front to quickly access their menu is unavoidable.
Heck, you might even get a Mapstr spatial widget to let you know that your friends have tried and loved the croissant, or that the oyster made them sick. That is surely not something within the reach of a cumbersome AR headset… for now.
The Apple TV is a mirror portal in your living room
One of the most underrated announcements at WWDC23 is the arrival of Continuity Camera on Apple TV. It states the following :
Developers can take advantage of Continuity Camera APIs on Apple TV 4K to integrate the iPhone or iPad camera and microphone into their tvOS app, and create new shared and immersive entertainment experiences for the living room.
It is easy to focus on its applications for video calls. But the real impact is when this camera is used to show you an enhanced mirror image of your world. One of the most exciting applications is within the Fitness app. The app can track your 3D body posture in real time while you work out, giving you detailed feedback on how to improve your form and the number of repetitions you performed. It may even be able to adapt the pace of your workout to follow your rhythm. Ubisoft’s Just Dance might also make a welcome comeback to hands-free body control using Continuity Camera on Apple TV. And this is only scratching the surface of its potential.
These few examples illustrate how spatial computing is a platform that is ready to be activated once the use cases are there. And Apple is ahead of the curve.
When technology pulls a Houdini, “it just works”.
The Vision Pro’s best trick resides in its eye tracking capabilities, predicting the user’s intentions, and reacting to what the user is looking at. The magic achieved with eye tracking could be distilled into other devices. Although it might be diluted in the process, the convenience of the interaction might still make it an enticing user interface. With head tracking on your AirPods, Siri can whisper in your ear relevant information based on what you are looking at or where you are going. Approach a subway entrance and it will announce the next departures. Spatialized chimes might subtly guide you within the underground corridors without having to look at a screen.
The Vision Pro is just the tip of the iceberg. As Apple continues to develop a spatial computing ecosystem with gradual and multimodal interactions, countless new use cases will emerge. Technology will recede into the background of our lives, allowing us to focus on what truly matters — each other and the world around us. This might finally deliver one of technology’s wildest dreams, a user interface so intuitive that it feels like magic imbued in your world.
Indeed, our brains are inherently multimodal, filling in missing sensory information through associations with other senses. Similarly, multimodal spatial computing can compensate for the absence of immersive visual feedback, creating natural and intuitive user interfaces based on our other senses. Once other devices join the dance, haptic rumbles and chimes might make virtual experiences spring into existence, feeling tangible even in the absence of visual feedback. No longer will you need to scroll through long lists of apps — instead, you can simply point to an associated object to trigger an action and feel its consequences. You don’t want to wait until you wear an AR headset to know that your plants need watering. Instead, your watch might rumble in the vicinity of a neglected plant’s sensor to alert about its imminent desiccation.
The Vision Pro is a diamond in the rough. And a spatial computing ecosystem with gradual and multimodal interactions would open the flood gates to many more use cases. The flow might end up polishing the gem and revealing its true sheen.
The AR headset’s possibilities are not bounded by its hardware, but by its social acceptance. A gradual and multimodal approach could take spatial computing to the streets.
Conclusion
We came to expect a slowdown in technological breakthroughs, but what is coming might well catch us off guard. Apple endowed the Vision pro with an external display facing the world, surely a metaphor of what’s to come. The lights are bleeding out of it and into the rest of the Apple ecosystem.
When the devices recede in the background and technology permeates our physical objects, the world edges more towards magic. The genie is already out of the bottle, and Disney’s partnership with Apple might grant it its first wish.
It is now clear that the Vision Pro is only the spark that sets everything ablaze; but once that the dust has settled, it will become clearer what all of these changes mean for Apple’s ecosystem of devices.
We are on the verge of a new era, where technology becomes magic, and magic becomes reality. Are you ready for it?
This is the second chapter in a series exploring the history and future of spatial computing. If you liked this story, check out my previous one:
Part 1: Spatial Computing, a revolution 16 years in the making.
In the final chapter, we will examine spatial computing’s societal impact and how we can steer it to help technology deliver on its unmet promises.
