The Apple Vision Pro has a killer app, and it's neglected
"Holy shit, Alicia Keys!", my friend Jessica exclaimed.
Ever since receiving my Apple Vision Pro on launch date this past February, I put it on as many faces as possible. Friends, family, neighbors, men and women, young and less young, techies and luddites. Taking a page off the UX research playbook, I set aside my own excitement for the gadget and quietly observed how different people responded to it.
My name is Anthony Maës, and I've worked in the Augmented Reality (AR) space for the best part of 15 years, most recently driving AR engineering for Pokémon Go. For as long as I can remember, I've anticipated the release of Apple's first AR headset as the watershed moment that would catapult this new medium into the mainstream, the same way Tesla's Model 3 (another thing I worked on) ushered the era of electric vehicles for the common folk.
Fast forward to 2024, we are now living in this future time where the Apple Vision Pro features prominently in Apple Stores, geniuses eagerly waiting to give you a tour. And yet, in spite of a cautiously positive reception, it seems like the long awaited disruption is all but a storm in a teacup.
Three months in, the doom and gloom discourse has set in. New Vision Pro app releases have plummeted since the February release, rumors about shrinking production targets are rattling the developer community, and Tim Cook's nod to early adopters in this week's keynote feels forced and uninspired.
How did we get here?
Apple is a famously risk-averse company when it comes to product design, often perfecting ideas developed outside its glass walls. The iPhone, iPad, Apple Watch all took multiple iterations to become the market leaders they are today. Apple's first generation products are always beautifully designed, fabulously expensive, but severely lacking in features: who remembers the original iPhone had no apps, no 3G, no copy-and-paste?
Marques Brownlee noted in his review of the Vision Pro that people have been willing to overlook those early pain points when they found the device otherwise desirable and compelling. AR, now known as spatial computing, has historically struggled to overcome the hump of novelty. So then, what is the Vision Pro's killer app? I think I have the answer.
Compared to Meta's Quest 3, the experience of demoing a Vision Pro is very clunky. Apple's visionOS doesn’t support multiple accounts on a same device, so everyone except me must enter a guest mode, and go through an eye calibration flow that takes a few minutes, every time. The eye-to-lens fit has no room for prescription glasses; wearers in most cases must take them off and put contacts on, or squint a lot. Apple sells $99 prescription magnetic optical inserts, something somehow unnecessary with any competing headset.
On the plus side, it's easy to monitor and guide guests through the system. AirPlay can cast the Vision Pro's displays to a TV or a Macbook screen with no performance drop. This way, I can coach them to look with their eyes and click fingers (a user experience that's less intuitive than you'd think).
Once they're finally set up, and their excitement has waned somewhat, they find that most applications are iPad-style rectangular windows floating in space. At this time, every single one of those is worse than its screen-bound counterpart. Despite Apple marketing promoting productivity apps, again just like the iPad, input limitations makes the Vision Pro primarily a media consumption device.
My go-to first demo is Encounter Dinosaurs, a surprisingly compelling interactive 3D animation built in partnership with Jon Favreau (Star Wars: The Mandalorian) and Unity. The app opens a portal to prehistoric times, and encourages you to walk closer, setting you up for the thrill of meeting and interacting with a full size, albeit virtual, T-Rex.
"This is amazing, imagine all the other things they could show you with portals like this", my neighbor John said.
And this is precisely the problem: producing immersive 3D, with high resolution models, textures, and animations, is prohibitively expensive. The experience itself is only a couple minutes long. Its limited user interaction gives a bit of life to the animatronics (a butterfly lands on your hand), which wows reliably, but falls short of enabling repeat usage.
It is, by definition, novelty, and it's hard to imagine the kind of economies of scale that would enable a content treadmill for which audiences would keep coming back. For all its merits, this is not the killer app.
Further down the demo script, which I perfected over time, I found another experience resonated even more: immersive videos.
Apple distinguishes between two kinds of 3D videos on the Vision Pro: spatial and immersive. Both are stereoscopic, meaning each frame has two images seen from a couple inches apart, creating an illusion of depth when projected in front of each eye. The main difference is size and quality: spatial videos are small and low quality but can be easily produced with the Vision Pro itself or an iPhone 15 Pro. Immersive videos, on the other hand, are designed to be viewed in full 180 degrees immersion at full resolution, looking better than reality as seen with the camera pass through.
Immersive videos are, at the moment, few and far between. Apple TV debuted multiple shows: adventure and animal documentaries, music videos, 3D animation (more dinos!), and sports. At time of writing, Apple offers less than an hour of aggregate content, with only 10 minutes of new episodes released since the February launch. However, in the words of my mom Blandine, "now I get it".
"I never watch sports, but I would watch more soccer like this", my friend Ben mused while playing a beautifully produced highlight reel of the 2023 MLS cup. Fellow founder and office buddy Emily echoed a similar sentiment: "I'm secretly a big UFC fan, but as a small Asian woman, I feel too out of place to go see a fight on my own. It'd be amazing to experience it, though, from the comfort of my home!".
In Silicon Valley parlance, this right there is signal.
Immersive video is the killer app.
By teleporting you inside the video with crystal clear 8K+ quality, while still focusing your attention in front of you, immersive videos break down the fourth wall. They are magnitudes more visceral and intimate than an IMAX movie, conveying emotions that could never cross a flat screen.
This is the reason I decided to invest in this space. I witnessed immersive video appeal to everyone who tried my Vision Pro, and inspire a desire for the device… if only Apple could overcome the content drought.
Why is there so little content?
Immersive videos are not a new thing. Veterans of Virtual Reality (VR) call them VR180 videos. Thousands of such videos are already available on Youtube, Vimeo, or the dedicated platform DeoVR. The big difference, though, is that their quality is not on par with Apple's offerings. Not only their resolution was designed for the smaller display resolution of Meta's Quest lineup at best, their production value is also a lot lower, from capture to editing to contents. On top of that, most of these platforms are absent from the AppStore, leaving users to fiddle with advanced WebXR settings in Safari or buggy paid third party apps to access the content.
On the Quest side of things, Meta curates a walled garden of immersive video with Quest TV, and while some of the content there is really high quality, it shares the front page with old, underwhelming content, resulting in a poor viewer experience overall.
Vision Pro sets a new bar, and creates a chicken-and-egg problem: consumers need a critical mass and frequent release cadence of quality immersive videos to justify such an expensive purchase. At the same time, professional creators need a large enough audience to hope for a return on investment.
Even for established studios, making immersive videos is still an experimental affair. Apple released no behind-the-scenes technical details on their production process. Their exotic video encoding is jealously proprietary; and even the nature of Apple's custom camera hardware is subject of speculation. Canon, who sells a popular $5000 VR camera kit designed for Meta Quest consumption, flatly admitted that there's currently no camera on the market to meet the specs of the Vision Pro.
To make things worse, unlike Meta, Apple is not funding third party efforts to bootstrap content to their platform, leaving developers and filmmakers to front their own experiments. At most, Apple might commission content via their TV or Music arms, and occasionally quietly acquire startups in the space. In venture capital, VR has come to be a bad word, and only timid but laudable initiatives have propped up to support new entrepreneurs.
So it isn't surprising that, in spite of the huge potential, only a handful non-Apple productions launched on the Vision Pro.
Indie New Zealand videographer James Hustler recently released Explore POV, an app he built to showcase his own splendid videos. However, each video requires a download ahead of time, typically amounting to one gigabyte per minute of footage.
VR video veteran Mike Swanson built an open-source player capable of reading local immersive video files, because visionOS didn't ship with one.
Needless to say, this community of immersive video pioneers is anxiously anticipating this year’s WWDC (Apple's developer conference), expecting announcements for new tools and APIs.
As for myself, after transitioning roles in another Vision Pro startup, I'm excited to take part in the early days of immersive video. The user response I've witnessed is just too strong, refuting the ambient skepticism.
In future posts, I'll explore the production pipeline, from camera hardware to streaming formats, highlighting current bottlenecks and idiosyncrasies.
In the meantime, if you care to learn more, I warmly recommend the following sources:
- Hugh Hou is a VR video making influencer, whose channel is a treasure trove of tutorials and technical walkthroughs of VR video production hardware and software.
- Mike Swanson is an expert VR video engineer, who maintains the aforementioned open-source player and frequently shares findings.
- r/AppleImmersiveVideo is a subreddit for professionals and enthusiasts discussing the tech and its slow drip of content.
And if you have an Apple Store near you, go try it out for yourself. I would love to hear what you think!
