Facebook F8 : The VR and AR Perspective

The Oculus Go awaits you, F8 conference.

We went to the Facebook F8 conference this week for the first time. If you’ve never heard of it, F8 is Facebook’s developer conference.

So as the title implies, we at PTE are VR and AR creators. So we went to F8 in search of our favorite toys: computer vision, Oculus Go, AR, and anything related to 3d. And let me tell you, it was nerd paradise. Let’s go over my highlights and impressions.

As a developer conference, F8 is mostly attended by various types of creators and some press. Unlike Apple, Facebook doesn’t have a bevy of launch events all year, so they use this one to launch their products. And launch they did!

Announcements

Day 1 was “big announcement for the press” day. So in the keynote, Mark Zuckerberg did a quick overview of everything new, and then some other execs dove in deeper.

It’s been a wild couple of years for Facebook and Zuckerberg.

The Politics of Privacy

Elephant in the room time: I don’t write about politics, but you know all about Facebook’s role in the 2016 election and Zuck’s subsequent testimony to the Senate. Obviously, the event was flavored with a bit of this brand of spice. Im sure it has a lot to do with getting ahead of regulations, stock prices, and all the stuff you didn’t come here to read about. But, those politics do affect the development priorities of Facebook. In which ways? Here are a couple:

Facebook is going to let you “clear your cache” of all your activity. Facebook has a new dating feature, and they spent a lot of that segment talking about privacy in dating. They are also very focused on Groups. As in, lets try to keep things smaller and more closed. “Facebook is about connecting with people you really care about” was the slogan of the day.

If you work in tech, you are probably thinking, this is quite the turnaround from SXSW 2013ish when BIG DATA was the buzz phrase, and “we’re gonna grab everything we can and who cares about privacy, we want to eat all the data!” Oh it sure does ebb and flow.

Now to the fun stuff. Facebook in general has some pretty interesting computer vision/3d scanning experimentation going on. If you combine that with their machine learning (called M), there are wonderful possibilities.

3d Memories

Probably my favorite thing all week was this thing where they take a regular photo, and use it to roughly recreate a 3d world from it. Sort of like a dream world.

Why I like it?

1)I can see regular people using it. And it’s usable on both regular Facebook as well as VR. On the FB app, it will be like a magic window mode, just like you currently see for panoramas.

2) Future applications: At our company, we have been exploring various ways to place memories in AR, yet still give a sense of geographic space and time. It’s been a passion of ours. We also have an obsession with 3d scanning and the role it will play in the future of XR.

That being said, we imagine a future of collective scanning, where we all anonymously add to an “everywhere 3d map” that can be used by anyone. With public places 3d mapped and available, we can easily augment reality (by tapping into this 3d geometry), but we can also easily make reality into VR (using the same geometry). From there, it’s easy to make a mental jump and think of cross-reality apps for both AR glasses and VR headsets; for example, I’m on a street in VR, but you’re actually there in real life, and you see my avatar via Augmented Reality (and I see your avatar in VR). It makes for an easy telepresence, no matter where you are in the world. These type of experience truly blur the lines of VR and AR. And it was sort of implied that FB is thinking along similar lines.

Welcome to your own creepy memories!

Oculus Go

As expected, Facebook released the Oculus Go at the conference. There have been entire articles written about how great this device is, so I’ll keep it brief.

Oculus Go has some great social features where you hang out with friend avatars.

Why I like it?

  1. Easy of onboarding. Tethered VR like Rift and Vive are cool, but for a non-technical person to acquire a fast enough computer, and then go through the trouble to set it up…it’s just a lot to ask. This barrier has kept these devices to niche audiences. The “phone” devices like Gear and Daydream are easier to set up, but it’s still a pain to put your phone in and out of a headset, and sometimes it’s not aligned right. The Go, on the other hand, is built to only do VR. It auto-starts when you put it on your head.
  2. It’s cheaper than anything on the market. We think of phone VR as being cheap, but not if you have to buy a compatible phone. Then it’s really expensive.
  3. Great optics: It is clearer than most VR devices on the market. You can imagine how this will be great for casual usage and film. Even though it’s not a powerhouse computer, it will show things like film and social apps with clarity.
  4. Facebook Features: This isnt something a VR creator usually mentions. But the fact that a lot of Facebook’s plans involve some cross platform and social uses of VR will entice non-technical users to try it and to keep returning to it. The social experiences were actually some of the most novel to me, like playing board games with friends in a VR lounge.
Oculus Go: Best played in fishbowls. Ok not really but it’s catchy.

Oculus Half Dome

One of the pain points of VR has always been FOV and depth of field. Those are a couple of the important ways that VR could seem more real and cause less of the current sickness and weirdness.

The best thing about the Half Dome prototype is that the screens inside it actually move slightly based on what your eyes are looking at. This works to vary depth of focus and focus aiming. It’s a prototype, so we weren’t able to see it. But promising nonetheless.

Half Dome. Just so you know the headset is not dome-shaped.

AI and Machine Learning

Day 2 went all-in on AI. The keynote had over an hour on it. Facebook is pushing their machine learning hard. But given recent politics, they are trying to push it with privacy and ethics in mind. The most interesting part of this discussion was based on ways to create algorithms that don’t accidently create bias against any group. They implemented a tool called Fairness Flow, that lets you test your algorithm for bias against groups of people. Machine learning is exciting (and will soon be in near every app in some form IMO), but it’s obviously a beast that needs to be controlled.

On the positive side, people in our industry of VR and AR have so much to explore with ML. Especially when it comes to identifying real-world objects in order to interact with them, or augment them.

Facebook was mostly discussing their ML efforts in the XR area on recreating spaces in 3d and intelligently tagging objects (and inversely, using user tags to apply to unknown objects), which makes sense, given their social focus.

Beyond the XR space, they have rolled out a lot of chatbot usages of ML. That was the main focus of the Facebook’s Messenger announcements; automatic language translation, business chatbots. Chatbots are a big deal right now for business and websites, but once they mature a little bit more, they will have great usage for imaginary characters in VR. Especially since you don’t have a keyboard. Natural language with truly intelligent replies will add a lot to our VR worlds.

Impressions

A few things kept popping up as “this is a common theme”. Here are the notable opinions.

Cross Platform XR

Facebook is pretty bullish on XR. But I think it’s smart that most of the software they’re creating for it is actually cross platform. From avatars, to chatbots, to 3d spatial scanning, you can use most of these features on regular FB mobile or in VR. This approach is not only more engaging for most users who don’t use VR yet, but it creates much more early adoption of the tech.

Facebook as an Engineering Company

Just like most folks, I tend to be Mr Forgetful and think of FB as a web site. F8 is a stark reminder that they are an engineering-led tech company. They just announced that they are opening a research arm in my home town of Pittsburgh. Amongst other branches. That should show how committed they are AI.

This all matters to VR/AR because the commitment to open-sourcing and research friendly technologies will speed up the technologies we need to make our next-gen inventions. Especially in the realm of computer vision and machine learning.

Convergence of Tech

Just like in the early days of mobile (when it was mobile-first this and mobile-tech that), people tend to think of XR tech as XR tech. As XR developers, it was useful to attend F8, because it’s a reminder that we can’t just live in our own bubble of VR/AR. A majority of the new tech created at FB is useful for XR, not just on Facebook as a social platform. Whether it’s chat, computer vision, or AI-created avatars, it’s all useful for our XR platforms. At PTE, we think machine learning and computer vison are some of the most important parts of the future of XR. So if we are to be innovators in XR, we need to be really interested in other technologies and how they can be applied. A pinch of this, a dash of that, and what do you have? Usally a big mess. Then you keep trying and trying again, and you make something great. That’s the fun of innovation.