Looking beyond the iPhone with AR Pioneer, Patrick O’Shaughnessey
How software is remaking the augmented reality landscape
- Inspired in 2009 by the GE Windmill AR campaign, Patrick set out to build early AR experiences using a Webcam and hasn’t stopped since.
- His business, Patched Reality, is reaching its 10 year anniversary and continues to push the barriers of the new technologies emerging, currently using spoken word to interact with the real world.
- “Today’s AR advancements are similar to the push geolocation gave mobile development ushering a new era of creative opportunities.”
- “6D provided the playground — now we can brainstorm what would be interesting to make use of that playground.”
Patrick has been in the augmented reality scene before it was even really a scene. Going back to the late 2000s, augmented reality was always on his radar, from testing out some of the earliest experiments to dabbling in each of the newest technologies that promised a next-generation augmented reality experience.
Some of augmented reality’s flashes of excitement came in the form of tricks with a webcam and a marker. Windmills would pop up on pages. Pieces of paper would become a kind of control mechanism for a cube on a screen. Microsoft’s motion-tracking Kinect for the Xbox came and went, but throughout the past decade, augmented reality became more sophisticated to the point that we can now see a different kind of world through our phones.
Using the 3D mesh generated by 6D, developers can build world-scale apps where assets are persistent, responsive to occlusion and synced between multiple users more efficiently. Developers no longer need to know what a model is up front (like a pint of Ben & Jerry’s, or a bottle of Pepsi, in his case) to have something to hide behind. This, Patrick says, is also still scratching the surface. People that have not been in AR for as long as developers like Patrick will not necessarily appreciate how hard this problem is to tackle. If a cat runs behind a sofa, you naturally expect the cat will be behind the sofa.
So, really, Patrick has seen it all. He’s at the forefront of game-changing technology that will chart some of the next great augmented reality experiences. Windmills aren’t popping up on pieces of paper as a kind of trick anymore — instead, AR experiences are going to be part of our everyday lives.
Perhaps more importantly, developers like Patrick are starting to look beyond just the iPhone. As companies are still trying to create augmented reality headsets, it’s clear AR is moving beyond the experimental phase to one where hardware works around the software. Patrick took some time to sit down with us and talk a little bit about his experience and where he thinks things are going.
Below is a Q&A our team conducted with Patrick to get his thoughts on the evolution of AR over the last decade and where he thinks its going next.
Let’s start off with webcams in the early 2000s.
Some of the first projects I did were using ARToolkit. I did a project for A&E’s The Five Lives of Criss Angel — you would hold a marker up to the webcam, and you would see yourself and a little maze. It was a game that you had to lead an icon that represents Criss Angel levitating through this maze. Instead of using gravity, you were going against gravity.
There was also a little game, for Jim Henson’s Dinosaur Train on the PBS Kids website where kids could point the marker at their webcam and see a little dinosaur egg. After a little while, a small dinosaur would come out.
Then the iPhone got into the hands of millions of people.
Right. Eventually, the iPhone came along and it had a built-in camera. Companies were building SDKs that would allow the iPhone to look at the world and sense natural-feature pictures. I did a collection game for the Museum of Natural History in New York. It was part of their exhibit, Beyond Planet Earth. It included different types of space exploration stations — some for the moon and objects in our solar system, one for the Mars rover, and one for each of the different spacecraft used for manned flight. Visitors would walk around the exhibit, point their phones at different markers and see a 3D model and animation tied to the marker floating in space.
What was it like navigating these different phases of augmented reality development?
It was quite a few pivots. I started with black and white markers on webcams, then we went to phones, then natural feature markers like pictures and magazines on phones. We did the first cylindrical tracking back for Pepsi Max Israel for one of their campaigns using a company called ObviousEngine, the first SDK to do cylindrical tracking.
Eventually, we did a little bit of stuff with the Point Cloud SDK from 13th Lab, they ultimately got acquired by Oculus. Meanwhile, toolkits like Vuforia were also adding 3D object tracking, so we could track things like toys or shoes. The next leap after that was ARKit, ARCore, much more sophisticated versions of SLAM tracking than PointCloud.io’s implementation.
What was the a-ha moment that drew you to AR?
The a-ha moment with me for AR was in early 2009. I was working at a startup called The Electric Sheep Company doing virtual worlds development. I think the first thing I saw with the flash-based AR was the GE windmill ad campaign. You would hold a piece of paper up to the webcam and a little windmill would show up in 3D with hills and the Golden Gate Bridge behind it. GE was using AR to promote its green energy plant.
I thought — That’s in Flash. I could certainly learn Flash and everyone’s got a webcam. This is actually doable. People can have augmented reality in their homes. It’s a crude form of it, but I think you can probably get some business doing this.
It so happened that the startup I was at had started laying people off. I was like, alright, I have to do something with my life, and I’m really into this AR stuff, so I’m going to form my business around it.
I had to see if I can make a go of with this AR stuff. It started with a prototype. I got a marker and a 3D model of a watch and a prototype where you could hold a hand up to a webcam and see your watch on your wrist. Ori Inbar said I think I can probably sell that to one of my former employers. We did a small app for SAP for their trade show, where you held your paper up to a webcam and it used a Rubik’s cube PaperVision 3D animation. I put the bug in the ear of some former coworkers about augmented reality and I became the tech arm of this small boutique creative agency and did a whole bunch of projects for big companies.
It’s been non-stop AR and VR ever since.
Could you compare this moment for developers in AR to one in the past where they got access to new game-changing technology?
Geolocation is a good one. Being able to access GPS on the phone, for instance, opened up the possibility that you could make location-based games — before you weren’t able to do that.
Having access to the gyroscope meant you could stabilize your content much better. If you’re not just talking about the occlusion piece, but also the localization piece and being able to locate you in space. 6D’s technology has the promise to locate you in the area much more accurately than you can with GPS, or even enhanced GPS. This brings you down to the centimeter level and also you can bring it indoors and get great tracking indoors.
If you think about the multiplayer implications capability that’s unlocked, to be able to both position content in the space but also position all the people in the space pretty accurately, there are a lot of opportunities.
Where does the hardware go from here?
It’s still a ways off before eyewear becomes ubiquitous, but maybe not as far off as people think.
Ultimately, I think it’s going to be eyewear. I think that’s where the industry ends up. I want to continue evolving our technology capabilities to align with the trajectory of the industry. We’ve already done small things with internal enterprise clients that wanted to see how HoloLens could be used for instance. There’s also a whole graveyard of eyewear that I’ve done prototypes on for clients like Google Glass, ODG, the Epson Moverio, and a whole host of VR headsets and the like.
How did you first get going with 6D’s platform?
I had to actually buy a phone to do it because it didn’t initially support the iPhone 7, which I had at the time. My first experience once I got the phone and booted up the SDK, it was a little laggy at first. At first, the meshing worked great out of the box — they had a built-in demo they had created with the little ball-pit where you could put balls out into the space, and they would bounce off your furniture. I work in my attic, so I could get things to bounce off my railings and the stairs and off my desk. But, the main thing was 6D was releasing a new version of the SDK every week. With the very next update, some of the lag had been taken care of and it was much faster.
This wasn’t an individual effort. While I developed the code, there were two other collaborators involved. It was more of a team effort from the brainstorming side. This wasn’t the only pitch we did. It was the one we got selected. There were 6 or 7 that we all collectively brainstormed on and submitted and talked about. As I was developing it, we had daily conversations.
Wade Tinney: Wade is a game industry veteran. He and Josh Welber had a game studio together in New York. I think we probably met about fifteen years ago, or something around that time frame. We met first just as people who were interested in the game industry. I was working at a startup doing virtual worlds. We even talked about collaborating on a virtual worlds game. It never really quite happened at the time, but we kept in touch in a networking context. About five years ago he said, I have this client that needs some AR done. So, I started building projects for that company. He was also working as their product manager/producer/project manager.
Josh Welber: About a year after that Josh got involved as well. We’ve been working on various projects for this one client, mostly in the home decor space. We did some other stuff, like VR games for kids. We’ve been working together pretty much daily for the better part of four years.
What was it like to collaborate with the 6D platform?
It’s pretty easy. As we start talking about taking the demo and turning it into a full-fledged app, Josh started taking on some of the coding tasks. Wade is acting as the product manager, UI/UX designer, project manager. Wade is working on things like what the game mechanics should be, what should the user interface look like. We all brainstormed together on that. He’s ultimately the expert there — we’re handling the technical details.
In terms of working with the 6D technology, it adds a whole new palette of things that we can work with that we didn’t have before. The whole idea that we can mesh the world, it opens up so many new gameplay possibilities. 6D provided the playground — now we can brainstorm what would be interesting to make use of that playground.
What’s it been like to see your projects turn into a business?
It’s been super exciting. We’re coming up on ten years since I started this company, Patched Reality. I first had an aha moment where AR, this thing I’d been interested in since 1994, was finally going to be something that people could actually have on their phone and use on their hardware and software platforms.It’s been great to exclusively work on AR for almost ten years now.
What’s next on your list?
We want to see if local creation, using spoken word to interact with the real world, is that a thing that will catch on.
For us, what we’d like to get to is a scalable business model most likely with 6D or at least something in the AR cloud space.
Right now, to me, 6D seems like the best contender. We’re still working out what exactly that means and what vertical — it could be this entertainment space. We’re going to put this game out there as a showpiece, but also potentially as a revenue generator to see whether a virtual pet in the real world is something people want. And of course, we still have our business tackling challenging and creative AR projects for our customers.
