Google Discontinues Daydream VR

That just happened yesterday. Google’s cardboard style products are what inspired my to pursue career as AR and VR developer. But it seems these products are now outliving their usefulness.

Google is discontinuing the Daydream View VR headset, and the Pixel 4 won't support Daydream

Samsung Galaxy Note10 will not support Gear VR headset

Oculus and Samsung Gear VR had a partnership pushing forward the VR for some time. But Samsung somewhat silently discontinued investment in slot-in VR shortly after the products like Oculus GO and Oculus Quest were announced.

Samsung Galaxy Note10 will not support Gear VR headset - MSPoweruser

Oculus Mobile

Oculus now ship two devices: Oculus GO and Oculus Quest. They are mobile, their hardware spec is very close to that of a mobile phone. However they do not waste the battery of your phone. And are based on Android as well.

Oculus

According to Mark Zuckerberg Quest sales go:

“as fast as we can make them”

Oculus also provides a VR dedicated app store and delivers high quality content.

Mobile VR will be at DevReach 2019

Do you want to know where the mobile VR is going? The slot-in VR is dead now, its successor is, in my humble opinion, superior. If you are interested to learn more about the mobile VR development for the Oculus family — I would like to invite you to my talk at DevReach 2019, Sofia:

VR Quickstart with Unity 3D - DevReach 2019

22 Oct. 4:40 PM — 5:30 PM (hall Alpha) “VR QuickStart with Unity 3D”

For the past few months my colleagues, from the AR/VR team at Progress Software, and I have been working on line of business applications:

VRLabs - Gain Competitive Advantage by Applying VR

And I am going to share part of our experience.

See you there,

The Mobile VR is dead, long live the Mobile VR was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s been nearly 2 years since we started our AR & VR journey at Progress Telerik. For that period we created some awesome stuff, made some mistakes and also learned a lot about the ecosystem and how to operate in an early market.

We’ve Been Inspired

End of May this year, we had a training. A training that turned our point of view and thinking at 180 degrees. It was based on the “INSPIRED — How to Create Tech Products Customers Love” book by Marty Cagan.

Since then we re-assessed our approach to VR and redefined how to further develop our endeavor.

The most important takeaways from this book for us are:

• Nothing important happens in the office
• Our opinions, although interesting, are irrelevant

In other words — if we want to create a valuable product for VR, we must find real clients and talk to them, BEFORE we create a product. This is what the book calls “Product Discovery”. Of course, the process of product discovery is built upon multiple steps and risk assessment, but in essence, there is little to no chance for you to create a valuable product unless you validate it with customers prior to any real development.

A New Website is Born

Following the best practices of the Inspired methodology, we are thrilled to announce a new website, entirely dedicated to VR, that we believe will enable us to have more streamlined conversations with people about how VR can add value to different business processes.

Here is our new web portal:

If you’ve been planning to start exploring VR for business, you can start by exploring our new site. One of the top reasons that make VR ideal for business, is that this is an early market and still an uncharted territory — the perfect setup for future-looking companies to create competitive advantage and innovate on existing processes.

Why a New Website?

To better streamline the process of discovering new prospects, we needed a focused landing page. While our existing page on www.telerik.com is excellent, it brings some distraction to people new to the Telerik brand, and who come solely for VR. As a rule of thumb, when you explore new products in a new ecosystem, you should not force people to learn more about your existing brand than needed.

What Happens With Our Plans for DataViz and Developer Tools?

Our efforts in this direction remain unchanged. While building real VR simulations for various businesses, we will generate a set of reusable components that are fundamental for most of the applications. As an example, some such, already existing, components are:

• Networking (multiplayer) with VoIP, avatars, gestures and 3D models import/collaboration
• Drag-and-drop laser pointers, 6-DoF hand gestures and 3D object interaction
• DataViz components like Charts and Graphs
• Hand-based interactive menus
• More…

If you are building something for VR and a set of developer tools would prove handy, please drop us a line at https://www.telerik.com/ar-vr-lab#contact-us.

Interested to Talk VR?

We want to hear from you! We have ton of ideas and many PoC applications about how VR can add value to businesses, but it is your requirements that matter on what a VR application needs to do. So let’s partner:

Request a Demo | VRLabs - Competitive Advantage Through VR

Or, you can reach out on our new social profiles:

Discovering VR for Business — an INSPIRED update from VRLabs was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

Short answer is “Yes, definitely”. In this post I will share my personal opinion why I think so. There are a lot of great in-depth reviews of Oculus Quest and my intention is not to write yet another review, but to rather summarize all the details that delighted me and made me smile.

Disclaimer: I am a huge VR fan, part of the VRLabs team at Progress, and there is certainly some bias, so if you are new to VR don’t rely solely on my opinion. Still, I hope that you will find these points valid when you enter this new world :)

#1: Premium package quality

This device is premium from every single aspect. The entire package implies the perception that Facebook put a lot of thought process and attention not only to the hardware, but also to little details like magnetic close on the internal box that contains the charging cable. Or the included batteries for the controllers. And for those of us that love to stay in VR for more than an hour, there is this 3 meters-long charging cable that allows you the freedom you need.

#2: Craftsmanship from the outside

Oculus Go, although a really nice entry-level device, felt cheap to me. And it can’t be otherwise, for the price tag of $200. The Quest, however, is rather premium. It is easily adjustable in size and fits perfectly on the head. Although not the most ergonomic device that I’ve tried, it is ergonomic enough to let you stay comfortably in VR for hours, especially if you are watching movies/videos.

The controllers are just awesome. Did you know of this rubber coating on the bottom half that makes the grip extremely comfortable? To me, these are simply better than the original Oculus Rift controllers.

#3: Outstanding lenses and display

The Quest is not the VR device with the best picture quality. But for an All-In-One device, the quality is mind-boggling. For example, it is better than the first Rift, which is a tethered device that uses the graphics capabilities of a powerful gaming PC. Our applications, by just porting them to Quest, felt times more crispy and vivid.

#4: Unexpectedly good tracking & guardian system

I was excited by the Quest the moment Facebook announced it. But, honestly, I wasn’t expecting its tracking and guardian systems to be great. And, besides working flawlessly (well, on rare occasions, when you get too immersed in the game, there are some minor flaws but they are indeed easily accepted as insignificant), it is also a system that you get used to in a matter of several minutes.

#5: As easy as possible app migration

If you’ve built apps for the Rift and Go, then chances are that you will be able to easily migrate them to the Quest.

Our team focuses on the business aspect of VR and we do not have experience with games, so I can talk only about line of business applications.

In fact, the biggest difference between the Go and the Quest lies in how an app handles user input and what level of interaction it allows for. We spent like a week to develop a cross-platform Pointer and Interaction abstractions, based on the latest Unity3D input device APIs, that just work on every Oculus device. Then, building and deploying was like just unplugging the Go and plugging-in the Quest.

Conclusion

To me, Oculus Quest is well prepared to steadily democratize the consumer VR space. It has everything you can wish for from an all-in-one device and delivers even more than expected. And most importantly — it is at an affordable price. $400 is a really good price tag for the quality that we get. And, as we all know, the mass adoption of a technology starts with the consumer market.

So, no matter if you are new to VR or you’ve been a fan of another device, I can definitely recommend the Quest — you won’t regret it!

Is Oculus Quest Really The Best All-In-One VR Device Today? was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

Explore and Analyze Your Data in Stereoscopic 3D with AR and VR - Progress Telerik

Vision for each eye span about 120 degree horizontally and 100 degree vertically. If we take into account, the eye rotations this spans the view to 210 degree horizontally. If in average the smallest visual offset between the visual features that can be detected is about 0.0006 degrees, this gives us a resolution of more than 350,000 pixels horizontally necessary for each eye to match what we can potentially see in reality.

The VR Book explains this, and a lot of other things about the VR in depth. I do highly recommend it:

The VR Book

Today consumer grade VR devices are getting shipped by millions. Sony sold over 4 millions PSVR headsets, that makes 1 in 20 PlayStation 4 buyers. Oculus Quest standalone units are shipped rapidly, with overall high user satisfaction.

But all these consumer grade devices are still VR generation one. They have nice controllers, ergonomic headsets, good positional tracking, easy setup and maintenance. But when you take a step back and take a look at the technology the innovation is not revolutionary, it is evolutionary — the hardware is little more than lenses attached to a phone screen in a single package mounted on your head. The hand input is done through infrared camera tracked controllers, but finger input is captured through buttons so fine motor skills has little play. Feedback is provided by haptics — controller vibrations. Sound is decent quality. There is no body tracking. There is no skin touch, heat, smell, etc.

How do we See in VR?

The human vision relies on two eyes. We receive two slightly different images. Out brain combines the state of our eye muscles eyes, our head position and those two images, to form a single perception of a world rich in depth.

Visual Acuity

Each of our eyes contains several types of sensors. We will not go into details into these types, but we will note their distribution. The visual acuity, the ability to detect clearly and in high-resolution, is concentrated in the fovea — the center of the retina.

So those 350,000 pixels we can potentially detect — we do not receive all that information at once, the fovea sees only the central 2 degrees of the visual field in high resolution.

Saccades

To perceive more detail of a 2D or 3D picture our eyes move simultaneously to bring different details onto the fovea — Saccades. The brain does these in succession and extrapolates the fine details in missing pieces.

If you could track these movements, you could generate an attention map. But the most important thing for the consumer grade VR headsets is that they simply cannot deliver the resolution to fill completely the fovea.

To present high quality models, high quality textures, and the perception of the detail of a real-world premium object, the VR display will have to match the fovea resolution. This can be done by eye tracking, streaming a very small but high resolution image targeting the fovea. But the VR is not there yet.

Consumer VR may not be the right choice to present premium, high detail, products, pictures, textures, tiles, fine art etc. the headsets cannot deliver the level of detail.

Horopter Plane

We see two images, one with each eye. Objects at a distance, that has no disparity in those two images, form a plane — the horopter plane. Those objects are perceived as crystal clear. Near that plane is an area where objects generate disparity, but the brain can still fuse them together. Everything else outside that area, contains disparity too great to fuse, causing objects to appear as doubled.

Vergence

Vergence is the rotation of the eyes in opposite direction with the purpose to obtain sharp and comfortable vision. The position of the eyes in this type of movement provides depth cues effectively of up to 2 meters.

Accommodation

Our eyes contain lenses that change trying to focus the incoming images on the retina. That process is called accommodation. The feel we have of the eye muscles contraction that change the lens, provides a cue for distance of up to 2 meters.

When the effects of vergence and accommodation are combined they make objects that are on focus to appear as single clear image. Objects that are not on focus appear as blurry doubled objects.

The consumer headsets have screens and lenses that make the images focus at a fixed distance. But present two different images to both eyes. This causes vergence vs accommodation conflict. The consumer headsets benefit from that the vergence contributes to depth perception more than the accommodation.

This enables the consumer VR headsets to deliver information rich in depth suitable for engineering visualizations, buildings, machinery, floorplans, charts.

Depth Perception

Accommodation and vergence play role in depth perception. But the overall process is much more complicated. Our brains process automatically array of stimuli and mix them with our knowledge and prior experience. This allows us to perceive depth from 2D media or even from a single eye.

Consider the following list of factors contributing to depth perception again from “The VR Book”:

Pictorial Depth Cues

• Occlusion (front objects hide rear objects)
• Linear perspective
• Relative/familiar size
• Shadows/shading
• Texture gradient
• Height relative to horizon
• Aerial perspective (rear objects look dull disappearing in fog)

Motion Depth Cues

• Motion parallax (when you move, near objects seem to move faster)
• Kinetic depth effect (changes caused by moving objects)

Binocular Depth Cues

Oculomotor Depth Cues

• Vergence
• Accommodation

Contextual Distance Factors

• Intended action
• Fear

This raises the bar of designing 3D stereoscopic display experiences that rely on depth perception. You must take into account the oculomotor depth cues thanks to the stereoscopic displays as well as the motion depth cues thanks to the head tracking.

2D vs Stereoscopic 3D Displays for VR DataViz

Traditional data visualization techniques present data on a single 2D screen. This media has the property to fit all the presented information within focus satisfying the vergence and accommodation easily. That is the whole 2D presentation lies very closely to the horopter plane.

There is a trend for curved displays and TVs, that positions even the edges of the screen perfectly on the horopter plane. In VR curved displays are used frequently for visualizing 2D movies, sometimes menu systems.

On the other hand, the human brain can only consume a limited amount of input at a time. This leads to a conflict where bigger screens and better resolutions allow for increased amount of information to be displayed at once, but brains are not capable of handling it all. On 2D you can rely only on the visual acuity distribution to control content at focus, leading to grouping of data, guidelines for content paddings, etc. This leads to a strive for an overall simplicity in 2D data visualization.

Don’t go 3D on 2D Media

This is as a rule of thumb. Any old book on DataViz will talk about avoiding 3D in general. It leads to noise, high ink per data ratio, visual clutter.

Perception of 3D data on 2D display cannot generate the accommodation and vergence depth cues.

On printed media, where the models cannot be moved, occlusion hides rear data and there are no cues from motion parallax. This can be overcome by interactive 2D displays but it still requires active input from the user or position animations to change the point of view in order to read the data.

Shadow/shading, areal perspective requires photorealistic rendering, for DataViz on 2D displays these type of visualization often lack the context of the environment — where lights come from, room corners forming cues of linear perspective and so on.

Go 3D on 3D Media

Stereoscopic 3D displays are very different. Rich 3D models of the data can be presented. Moving your gaze through these models will push and pull that horopter plane, moving data sets in and out of focus. Depth cues from vergence are provided thanks to the stereoscopic display. Head tracking will allow you to naturally change your point of view triggering depth perception cues that come from occlusion and motion parallax. Occluded back objects can be easily and naturally revealed. Using proper geometry can also trigger depth cues from linear perspective and relative/familiar size.

The consumer grade VR headsets can generate exceptional depth perception in users, that enables a 3rd dimension to be used, and the perception of 2D areas can be extended to 3D volumes.

So how can we make a good use of this property in AR VR DataViz?

Sales Map

One thing that comes to mind is to use clusters of data placed at various distances. Having a stereoscopic AR VR DataViz horizontally placed map, with clusters of charts positioned by regions, can effectively replace a drill-down by regions in a traditional 2D story. Instead of checking out sales by navigating back and drilling down to a different region, you just use the most natural interaction possible — slide your gaze.

Yes, you can visualize a map on a 2D display, but if you overlay the charts, then they all get perfectly focused and you get quickly that too-much-information effect. You can rely only on visual acuity distribution to separate groups.

The shading of the map provides height information both in 2D and AR VR DataViz, but in stereoscopic 3D depth cues are also triggered from vergence on high contrast areas — road lines, city names, pins.

The 3D Bars we selected are aligned with the table sides and the floor tiles so depth cues come from linear perspective and the familiarity with the bar sizes.

Combine Reports and Share Context

Combine several 2D reports in a single stereoscopic 3D AR VR DataViz experience. The closest thing to this in 2D is having 2 monitors running a single reporting app — multiscreen app that should be aware of the monitors placement.

In our SalesDashboard application we placed the products in one axis and shared it between multiple 2D reports placed amphitheatrically. Each report focuses on different aspect of the data and you can switch the context by moving your gaze, while mapping the products from one report to the other. The navigation feels much more natural than switching tabs in a browser.

Keep in mind the above image is a projection of the VR experience onto 2D media. When experienced on device, the vergence will mask the interior of the room, the sharp edges of the TV screen and room corners will not interfere with the actual chart. Switching attention from one report to the other is followed with head and eye movement. When you look at the leftmost dashboard the rightmost hides behind you. Only when you take a few steps back you capture the whole picture at one sight.

Insights from Trends

We have laid bar charts on a table. On one side of the table we have products, on the other we have mapped time. The amount of data that is visible is huge — 324 points. On 2D display you can gaze at one bar and only visual acuity distribution will limit visibility of faraway bars.

When you stand in front of that table in VR and gaze at a product, now the vision is affected by that horopter plane. It enables you gaze at the bars for a product, while your brain is also filtering the nearer and farther products.

The bars have clearly visible square caps, that familiarity allows us to detect changes in size and contributes to depth perception. They are also placed on a grid at the bottom and with bar walls in parallel to the room walls. This makes it easier to follow the array of bars for a product.

Graphs

The 3rd dimension allows for richer graph layouts. Naturally occurring graphs like those from social networks cannot be presented on 2D screens without having their edges overlapping. In 3D the layout constraints are much more relaxed as lines can easily go one beyond the other. In combination with head tracking, head movements triggers depth cues from vergence, motion parallax and kinetic depth effect.

The nodes of the graph can display rich data emphasized in node size and colors. However, displaying quantitative data in size can break depth queues from relative/familiar size. Thus, our choice was to use qualitative values for size, by snapping node sizes to common values. In the Twitter Graph, all users that only retweeted content are displayed using a common fixed size circle, thus triggering relative/familiar size depth cues for the masses of users, while only top connected users are displayed using far greater sizes based on the produced content.

Bubble Charts

Bubble charts share common traits to graphs. However, the represented data points are not interconnected and instead of topology they communicate data values.

This kind of data exists on 2D by the form of 2D bubble charts. In these charts the position of each bubble identifies 2 of its properties and another 2 properties can be encoded in size and color.

The 3D version of a bubble chart has 3 axes to plot onto. Another property can be represented by color. But what about size? If all data points share the same size, then this triggers depth cues from familiar size, boosting the readability of the 3 properties encoded in position. So, this is a tradeoff you can take.

The depth cues from shadows/shading are lost because the large amount of data points makes it hard to relate a bubble with its own shadow. There is one quality of the VR bubble charts that is very important through — they can easily visualize grouped clouds based on 3 properties, not just 2 like on 2D.

For our bubble chart the axes are drawn in parallel to the room corners and the desk to enable linear perspective depth cues, and we’ve decided to keep uniform sizes for the data points.

Point Clouds

There is a special case of bubble charts — point clouds. These usually present points of the same size but in huge quantities. A “point” in general is not supposed to have a volume in 3D or area in 2D, so consider it as a single pixel in space. These cannot generate depth cue based on familiar size, because of the very small size.

To compensate these are displayed in context that it is either very easy to rotate the point cloud and boost the depth cues obtained from parallax. Or display reading from sensors that display a well-known object where the cloud itself forms a body we are familiar with, triggering cues from relative/familiar cues for the whole group or individual clusters within the point cloud.

AR VR DataViz Gone Bad

We’ve seen some of the sources of depth perception underutilized in AR VR DataViz. The following examples have been taken out of their context, so I would like to apologize in advance and pay some respect to their authors before I use the images for illustration of what I think could be done better in VR.

Bubble Charts

Real world: The biggest screen where you can analyze the 3D data.

The whole world really provides huge real estate to present data, this kind of visualizations are better placed into environment. The beach is a non-uniform structure where linear perspective is totally lost. The following chart will be far better suited in an office room, with axes aligned to room walls, corners and windows.

Globe Chart

Globes in AR and VR look wonderful. Placing 3D bar charts on them has caveats.

When you rotate the globe to position a country facing you, the bars occlude themselves — that is the top of the bar hides its height, depth perception is not good enough to rely on it for communicating values. To compensate this, colors can encode the value, but then the whole thing becomes a heatmap.

Data visualization must not lie, including AR VR DataViz. The above presentation has yet another issue. It looks like a heat map, but bars at the sides are seen in their entirety while bars at focus are represented only by their cap. And this may end up in small bars at the sides appear larger in value compared to larger bars at the center.

3D for the Sake of 3D

Simply because you have a third dimension in AR VR doesn’t mean you must use it. If you have a two-dimensional data you may be better off presenting it on a 2D plane or with a 3D geometry objects, but still placed on simple array. It is easy to affect occlusion or form false depth perception based on relative/familiar size.

The top ring of the above presentation looks fine. August is occluded a little by June at the far left. Still enough from the August bar is visible so you can read its height without moving. The ring may present a cyclic yearly recurring pattern, so the layout of the array makes sense.

The lower ring however, represents countries — the layout doesn’t represent anything cyclic. The bars at the far left and right occlude each other so the values there are not readable, you can obtain trend information by the difference of the cap’s height, but not the overall value of the bars.

The bottom caps look the same as the top caps and as the bars are semitransparent the bottom caps limit readability at the far left and right sides of the circle. It may have been better for the bottom ring to be a 2D map placed on the ground.

You cannot Experience an AR VR DataViz on a 2D Display

Whether a 2D image or video, captured from AR VR experience, projecting it on your monitor limits your ability to perceive depth. All of the 3D objects are flattened on a single plane — your horopter plane, making everything perfectly clear, you no longer can focus clusters like you could in AR and VR, depth cues from vergence are gone.

You lose the context distance factors as intended action because someone else is taking the actions in the video. And because you are not immersed in the AR and VR experience, fear has little play. Presenting a VR on 2D screen ends up relying on shadow, texturing, linear perspective, relative/familiar size, height relative to horizon.

Occlusion happens, but you cannot interact with it, together with the motion parallax and kinetic depth effects, they must be explicitly exaggerated for the recording session of a 3D video.

Here are a few secrets we used to capture the videos for the Twitter Graph video on our landing page. View the following video and notice the motion blur during teleportation and the parallax effect:

During the recording we used a homemade 3rd person VR camera to capture 2D video, so the hardware allowed us to enabled motion blur and trigger kinetic depth effects - it happens very clearly near 2:15 during teleportation. The motion blur and the animated transition are supposed to also compensate for the lost intended action cues. If these are truly applied in VR they will cause motion sickness.

Then shortly after 2:15, excessive sideways movement strengthen the parallax effect on the graph. This compensates for the lost depth cues from vergence due to the lack of stereoscopic 3D display.

Thank You!

Thank you for reading this far. I would be happy to hear your thoughts on the matter — drop a line in the comments below.

Also, if you are interested in what we are doing, you can find out more, contact us or request a demo on our page:

Explore and Analyze Your Data in Stereoscopic 3D with AR and VR - Progress Telerik

For future blogs, follow our publication “Telerik AR VR” here on Medium.

If you are into short updates, follow us on Twitter: Georgi Atanasov, Deyan Yosifov, Hristo Zaprianov and me Panayot Cankov.

Perception of Stereoscopic DataViz in AR and VR was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

ProgressNEXT 2019

This event for modern applications development took place May 6–9 in Orlando. The audience of the conference consists of people who are interested in Progress Software existing products and want to learn more about the future plans of the company. As part of Progress Telerik, our main target was to show the attendees of the event that we are actively working on enabling VR for different business use cases, solving problems in a better way than they are currently solved on 2D displays. We presented our VR applications and concepts using three separate approaches — booth presence, live demo on the CTO Keynote’s stage and a separate VR session with DataViz related presentation. Let’s see how these approaches worked from my point of view.

Booth presence

On the Progress Telerik VR booth we have prepared three sample demos, showing them with Oculus Go and Oculus Rift devices. The first demo is a virtual conference room that can be joined by several people, where they can discuss a sales dashboard data visualization. The second demo is a twitter graph application which visualizes twitter communication in a way that has no readable alternatives in 2D. These two applications may be seen in more details and downloaded for free from our web site. The third one is a healthcare application allowing remote collaboration between a doctor and a patient in a virtual medical room. This application helps the doctor to analyze the data from patients medical test and explain the results and the needed surgery procedures with a descriptive and immersive 3D visualization of a sample human body.

As expected, there was excitement for the technology and our demos from the people that visited the booth. The ones who were skeptic about the technology were rather an exception. It was exciting to see people who showed real interest in our project and were happy to take our contacts to be able to reach out after the conference.

CTO Keynote live demo

About a month before the conference we realized that the CTO Keynote will be focused on AR/VR applications in healthcare, so we suggested to create a demo that can be shown on the big stage. Using the knowledge and reusable components from our existing applications, our team managed to fit the short time span and build a healthcare application that shows how collaboration between a doctor and a patient can be achieved in VR. And voilà — one of the most exciting moments from ProgressNEXT for me may be seen in the photo above! I and Georgi appeared on the stage in front of the whole audience of this conference, taking the roles of a doctor and a patient. In my opinion, this presence on stage, even for a few minutes, was the best exposure of our work that managed to reach the maximal amount of attendees, who are now aware of the existence of Progress Telerik’s VR labs project.

VR DataViz session

Our separate VR session was based on a presentation titled “From 2D to Stereoscopic 3D Data Visualization in Virtual Reality”. We had the chance to present for about 45 minutes our VR DataViz explorations explaining the following points:

• What is DataViz in general and why it is important for businesses?
• How Stereoscopic display works, how it differs from the standard 2D display visualization and how it is similar to the way we see the actual world around us?
• Why 3D DataViz is better than the 2D alternative when talking about real 3D visualizations on a stereoscopic display?

At the end of the session, we showed our Sales Dashboard and our Twitter Graph demos, emphasizing on the strengths of 3D DataViz in VR. After the demos, there were several questions from the audience showing that the people were interested in the technology and wanted to know how they can apply it to their existing businesses. Some of the attendees came to us after the session to discuss concrete ideas and scenarios which they see valuable for their business.

ProgressNEXT summary

We did our best to present our ideas and show our working projects, trying to reach the largest amount of the audience at this conference. We were happy to have these chances for meeting people from different business verticals and discuss with them what values the VR technology can bring to their businesses. There were conference attendees who came to us asking for more details on concrete scenarios. They took our contacts and chances are that some of them are going to reach out to us when the time comes for them to implement a VR solution.

Microsoft Build 2019

This is one of the largest conferences based on Microsoft technologies with over 6000 attendees. It took place May 6–8 in Seattle. Our team targets at this conference can be organized into three main groups — booth presence with VR demos, build connections with potential partners, and research what others are working on in the AR/VR area. Let’s see how this event went from our point of view based on these three groups.

Booth presence

Progress’ booth was big enough to welcome a good amount of conference attendees passing by. For VR we were showing the same three demos already mentioned for ProgressNEXT, with one small difference — the demos were shown not only with Oculus devices but with Windows Mixed Reality VR device as well.

Because it is Microsoft’s conference and their VR platform is Windows Mixed Reality, we decided to port our demos for WMR devices. Being able to achieve this task in less than a week proved that our decision for building components on top of Unity was the right thing — Unity makes it really easy to target different platforms and devices.

On the VR front, there was a lot of interest and enthusiasm in the demos. Most of the visitors were just experiencing the technology, some of them were even confusing the Samsung Odyssey device with a HoloLens 2 device. Developers that had some experience were happy with the demos and took our contacts. Some business people were present, those were enthusiastic about the sales dashboard demo and we gave them our fliers.

Connections building

Besides connecting with people at our booth, we also explored what others are doing in the VR area that may be helpful for our projects as well. We met people working on Microsoft Maps for Mixed Reality, which is based on Bing Maps. They had just released an SDK and wanted to get us on it. This corresponds to our current plans for exploring the geospatial data visualization scenario in VR and we see this connection as a good opportunity to build a good use-case together.

We have also tested the HoloLens 2 device. It is a very well put device. The balance is perfect, just as advertised, the straps are very thin and a bit stretchy which holds the thing perfectly on the head, the holo rings are gone. The display has slightly larger FoV, the colors are a bit off. But the biggest improvement in v2 is the hand tracking. The controls in these devices come just naturally, which is a big leap compared to the v1. Overall there were just few HoloLens 2 devices on the conference and there were several exhibitors that still used HoloLens 1.

Others work for AR/VR

We learned more about an open source project called Babylon.js. It is built by Microsoft and they are considering pushing deeper the JavaScript stack into 3D and by that extent VR. The main site has featured demos, one of the business cases is SharePoint spaces, showing this kind of experiences. They are exploring the options of building a “Babylon.js native”, which is “React native” for 3D. Allowing the JavaScript developer to quickly/cheaply prototype decent VR experiences may disrupt the Unity ecosystem. If played well, this may democratize the VR for JavaScript developers ecosystem as well.

MS Build summary

Honestly, we expected some more AR focus from Microsoft, however, most of the sessions were focused on Azure and other Microsoft projects which are currently unrelated to the Windows Mixed Reality topic. We managed to share contacts with some attendees who were interested in our work and probably some of them are going to reach out to us when they start looking for an implementation in their businesses. We are also happy about making some contacts that may result in future partnership opportunities on mutual projects.

VR Labs’ Thoughts and Takeaways From May’s Conferences was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is the sequel to the article written by fellow VR dev Deyan Yosifov about his endeavors trying to record a high-quality (stereo) 360° video. This time, I will go into more detail about the tool used for recording the motions of the Oculus avatars and some “common” objects like the chart and input pointer, prior to recording the video. MotionTool was written out of necessity, since there were no ready-to-use tools, plug-ins or assets in Unity’s Asset Store or elsewhere (and still aren't today), which get the job done. So if you want to record the movement of Oculus avatars or plain Unity objects, or are curious how this can be accomplished, then download our free MotionTool and give it a try.

Recording Avatars, but why?

One possible reason, as mentioned above, is the need to record a stereo 360° video containing Oculus avatars among other stuff. But being able to record and replay the actions of avatars opens up a whole new world of possibilities. You can:

• have a prerecorded avatar that greets the user and guides him through your experience.
• simulate different scenarios in environments like classrooms, workplaces, public institutions, banks, hospitals and so on.
• use avatars to teach or demonstrate certain actions inside your application
• prerecord avatars and replay them as reactions to your user’s actions
• use recorded avatars for aiding your development and testing your application
• use Avatars as NPCs

There are many more ways in which to make use of prerecorded avatars and in the end it’s up to your own, specific use-case. But all those cases require the same prerequisite, namely the ability to record and replay them. This is where our tool comes into play and gives you value by saving you time and pain.

But why a tool?

Why would you bother using a tool in the first place? If you are reading this article, then you probably know the answer already — Oculus do not have a tool of their own that does that, neither did we find any 3rd party tool that does it. The other, more important reason is, that it’s a bit of a complicated process and it’s lacking transparency. Oculus have created a mechanism to transfer avatar pose information over the network, as the core idea of the avatar construct as such is to give its users an appearance, which improves the social experience and interaction between them. But the way they accomplish this is by writing binary “packets” at fixed time intervals (most commonly 30 times a second) and sending them over the network, where they are read by the other participants or clients, “decoded” and applied to the avatar that represents the person that sent them. The process of encoding and decoding the packets and applying them to some avatar is completely opaque to the developer, so they have to use those packets themselves, whether they like it or not. This might be a bit too “low-level” for many developers, especially when they have other tasks at hand and their time is precious. Based on those packets and a couple of examples from the Oculus SDK, we created our tool, which can store all the avatar pose information for a given period of time in a separate file. It can later apply the stored poses on one or more avatars, thus creating something like “canned” avatar animations.

How to use it?

The tool comes ready with examples and there is a series of tutorial videos, which I encourage you to watch before using the tool for the first time:

1. Baseline tutorial
2. Working with Avatars

It contains different Unity components that you can attach to GameObjects in your scenes and assign them the objects that you want to record or play back on. The tool stores the data in its own .asset file formats. It also has a couple of synchronization components in case you want to record or play back to/from multiple data files at once. If your use-case is similar to ours, i.e. you want to play back the motions over the same avatars and objects that you record from, then you can use our special “Record Director” editor utility. It can automate the setup process entirely, you just need to tell it which avatars and objects you want to record and it takes care of everything.

At the time of writing this article, you can use the MotionTool for recording only in the Unity Editor. You can’t record in a built Player, no matter the target platform. You can though, play back recorded data in a Player.

Share your feedback

Share your experience with the tool, send us suggestions for useful features or improvements, what you would like to see and what would be beneficial to you (for example record-in-player functionality) or report issues or bugs that you have stumbled upon — every feedback is greatly appreciated and will help us improve even further!

We are looking forward to get in touch, you can reach out here!

Motion Recording for Oculus Avatars & Game Objects in Unity With VR Labs’ Free MotionTool was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

VR-wise we are well equipped with three different applications and use cases that feature dynamic DataViz, multiplayer (networking), avatars, VoIP, controller integration and more:

• Sales Dashboard
• Twitter Graph
• A fresh new demo, yet unofficial, related to healthcare:

We also have brand new beautiful T-shirts to give away to VR enthusiasts that play with the demos:

So, if you are looking to enable VR for your business we will be happy to chat with you and find a way to help.

Microsoft Build

Progress is a silver sponsor of MS Build and we will be exhibiting our entire set of product families at booth 300. Our VR demos also have their place at the booth and if you drop by, you will be able to experience the applications and see how this technology can add value in business use cases. We have a Windows Mixed Reality Samsung Odyssey+ headset as well as stand-alone Oculus GO devices.

ProgressNEXT

This is the largest global gathering of the Progress customer, partner and developer community, where people come together to connect, learn and collaborate. Plan your own agenda with 150+ sessions, trainings and workshops.

Top three reasons to attend are:

1. See how new technologies can work for you and help your business save time and money
2. Meet some of the best technology strategists and bright minds across diverse industries
3. The Expo Hall — see the technologies of today and tomorrow in action

Progress Telerik will have a dedicated booth in the expo hall and our VR demos will have a dedicated section there.

I and Deyan Yosifov will also have a dedicate session on Stereoscopic 3D DataViz that we are positive you will find interesting:

Already Excited?

I can’t wait to meet you in person and talk anything VR. See you there!

VR is Coming at Microsoft Build and ProgressNEXT 2019 was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most important phases of project management is validation. Ideally you will not invest time in development before you validate the idea for your product. VR is old, View-Master stereoscope is patented 1939, but still very few people have experienced modern VR technology.

Here is one of the unique features of this technology. If you must create a web-based solution, a sketch prototype is enough to express your idea and talk to customers. That is because they are preconditioned by seeing thousand such projects and little is left to the imagination.

VR on the other hand — they have not experienced the technology, they are uncertain of the benefits, they are uncertain of the technical challenges. Low fidelity prototype, presented in the form of video or images, does not deliver the necessary credibility nor proves the technology capabilities of your team.

You are now making high fidelity prototypes and present them on real devices.

Presenting AR

We’ve been doing AR and VR R&D for quite some time. We have learned a lot by presenting our AR and VR applications since the beginning of our journey.

Microsoft Build 2018

For Microsoft Build we created the HoloStock application for HoloLens and presented it as part of a social meetup and later at the Progress booth. The application was high fidelity demo, with some Data Visualization controls. Since it was our first public appearance, we run several test runs inside the company and gathered and addressed all the feedback we could.

Spectator View Marketing Videos

For marketing materials, we implemented networking and set up the HoloLens spectator view. At its core by that time this involved running a capturing application on a computed wired to a DSLR. The setup was too heavy for what we could place at the night club or at the booth. But we had to do it to capture videos to publish on our site. And to have something to show first-time users what to expect when running the app.

A Night in the Club

The day before Microsoft Build 2018 we gathered people from the IT industry in a gorgeous night club. To talk, to socialize, to present our demo. We planned to connect the out-of-the-box streaming of the two HoloLens devices to projectors on a big screen. While the devices and the app performed well, the HDMI cables between our laptops and the projectors failed. People were interacting with the app and we had little idea what they are doing.

At the Booth

The app was well polished, people were quickly getting how to use it. We continued without the streaming on the following days at the booth.

Going Home

We had to leave back home, with some key take-away:

• AR headsets obstruct communication
• First-time users may feel awkward from the unknown
• You must see what your users are doing, sometimes they need guidance
• The technology generates a lot of excitement
• You cannot teach people how to use headsets with videos

ProgressNEXT 2018

ProgressNEXT is an US conference organized by Progress. It is a place to get inspired, get connected, get creative. It is the largest global gathering of Progress customer, partner and developer community.

We had presence on a booth there and were demonstrating our demo in a similar way to Microsoft Build.

Presenting VR

Our focus has always been in delivering a data visualization suite. VR offers a much better arena for immersive data visualization compared to AR. Head-mounted AR focuses at on-field workforce and assistance.

We developed the next set of demos for the Oculus Rift and Oculus GO VR devices.

DevReach and ISTA Conference 2018

These are two events that happened in a same week. And our team talked at both of them.

DevReach is a conference organized by Progress at Sofia. We gather developers from all over the world, for 2 days. This time we were not only doing demos on our booth, we were also presenting. And ISTA is a QA focused conference, again in Sofia.

Presenting on a Big Screen

The presentation starts with intro of what AR and VR is, use-cases, the projected AR and VR economy growth, and ends with a live demonstration of our applications.

We made training presentations internally in the company a few times. The presentation of the demos was done using Oculus Rift on a laptop. The laptop monitor was mirrored onto the big screen.

This kind of demonstration reaches much broader audience. Even in smaller numbers it is much easier to present a VR application to group of peers or business clients like this instead of letting them take turns with the headset. However, this comes with challenges.

Giving a presentation of the application and then inviting the people to try the demo on the booth works well.

Presenting at a Booth

Presenting at a booth is fun. It makes new friends and twitter followers. But whether you are in a small room demoing to your peers or your boss, or on a booth for an event, you need a high-fidelity demo app and you need them to experience it.

Just like you can’t teach a person to ride a bike on a video — you can’t teach a person what the VR application is on a video.

These kind of demos however:

• Blocks you out of the user’s view
• You cannot explain gestures
• The headset obstructs their hearing
• You must watch out for symptoms of motion sickness
• You must accommodate for glasses
• You must take case of hygiene

Spectator View

At this point it was clear that for presenting your applications you need two things:

• Be able to show it in front of a crowd on a TV like screen
• Be able to guide first time VR users through your app

While experienced user will work flawlessly with a well-done app, first-time users will need some guidance. They are not used to the controllers. They are not used to the system gestures. It takes several minutes to go through the system welcome apps and get known with the controller and gestures, but this moves the focus away from your app.

We had two options at that moment.

Use System Streaming

One option was to use the mirroring or streaming when doing demos on the booths and presentations. This thing is supported by the devices, both HoloLens Oculus Rift and Oculus GO have capabilities to stream. However, during presentations subtle head movements shake severely the picture. You must present very carefully trying to keep your head steady. That extra attention draws away focus from your talk. At booths, the standalone devices spend additional power to capture, encode and transmit the video. This generates additional heat and wears of the batteries faster. The quality of the image may be reduced for the users.

VR Spectator View

The second option was to implement a spectator view for VR experiences.

Spectator view is rendered on a PC, thus it required proper networking. Collaboration is one of the key pillars for AR and VR, so we were going to implement networking anyway.

It shows us what the first-time users experience while in the app. It gives us precise control on the camera that the viewers see on the big 2D screen. It smooth movements like a steady-cam so it looks good when streamed on a big screen. Requires less attention from the presenter.

The stream must look like shot with a steady-cam. However, in the virtual experience you don’t need all that hardware. Instead, software simulates the camera movement.

We decided to go for it!

Iterations, Iterations

After the DevReach and ISTA conferences, we continued giving presentations, in similar format, to prospects that reached to us.

Internally at the company, we have meeting rooms where we used the following setup — we connect a laptop to a TV, we present the app using Oculus Rift with display mirrored to the TV, then we use the stand-alone Oculus GO devices to let the prospects experience the app.

This setup, although in very small area and group of people, uses the exact same hardware and configuration we used on the presentations for the conferences and at the booths.

Networking

We were addressing iteratively the feedback and that led to building our multiplayer environment based on the new Unity networking system. The Oculus GO devices now connect to the laptop over WI-FI.

First, we integrated a way to share highlights tooltips and the laser pointers, so users can share insights on the data visualization dashboard. Then we integrated the Oculus Avatars, so users can see each other’s virtual self, and exchange head and hand gestures.

Building the VR Spectator View

The multiplayer made the Oculus Rift optional. With the networking in place, the laptop now could participate in the experience without a VR device — it became our steady-cam or spectator view.

The image on the laptop renders the same 3D scene but designated for 2D displays. The camera that it uses has an algorithm to follow some of the users of the Oculus GO devices. The camera will interpolate linearly its position and rotation, so the movements are in real time, follow closely the presenter, but are also smooth.

For Full-HD or 4K TVs the picture rendered by the spectator view is in native resolution. It is not up-scaled from the headset resolution to the TV resolution. This further improves the quality.

Telerik Campus Presentation

With the networking and spectator view in place we had the chance to present at a local meetup before 30 people. Everything went smooth. HDMI cables to the projectors failed us again. This time we had replacement!

Website Redesign

At this point we were ready to redesign our website and include our new demos. You can check the site at:

telerik.com/ar-vr-lab

The presentation setup allowed us to also capture the promo videos for our two demo applications for our website. Smoothing the camera movements and locking its up direction to always face up — to prevent side way tilt. This gave the video a professional look.

You can see the difference by comparing the following “shaky” video captured directly from Oculus Rift:

Capturing 2D videos was not the only thing we had to do. To enable people see the potential of data visualization for VR we also created 360 movie to put on YouTube, this is an easy distribution channel.

Read the whole story about generating 360 movies here.

Microsoft Build and Progress NEXT 2019

These are two upcoming events and we plan to present and show the demo applications in their full potential. There are a few things that we aim to achieve.

• Give a talk, present the demos with high quality on big screen
• Let the attendees try the apps in a shared experience
• Capture videos and images for our blog and website

And here is our growing check list. We would like to share it with you, it may help you present your own VR creation.

Capture Images and Videos

• After the talk you will need videos and images for reference.
• Be proactive and get consent from users to capture marketing photos.

Present on Big Screen

• Practice in advance.
• Ask for dedicated presenter WIFI network.
• If possible test your setup in the actual presentation room a day early.

Let People Experience the App

• Ask for dedicated WIFI network if you depend on internet connection.
• Charge the devices before the event.
• Prepare power bricks for mobile VR if you expect long working hours.
• Prepare extra batteries for the controllers.
• Test with a few first-time users in advance.
• Share the experience on a TV or laptop.
• Prepare visit cards and flyers to give away.
• Care for hygiene — take tissues and disposable masks.

Wrap-up

Doing proper and quality presentations is key to promoting your AR VR product idea. I hope that you find this post useful and we will be looking forward to meeting you at Microsoft Build 2019 or ProgressNEXT 2019. If you have some comments or feedback you can share them here or contact me over Twitter:

Panayot Cankov (@PanayotCankov) | Twitter

How to Present Your AR and VR Application was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post is inspired by my experience with building a 360° video for Progress Telerik’s AR-VR web page. The video purpose was to show our “Sales Dashboard Data Visualization” demo from the eyes of several participants in a virtual room, who are discussing the presented 3D charts. As this visualization displays a lot of text and graphics, it is important to ensure that the rendering quality is sharp enough so that the participants can easily read the presented information. Although our demo application is written in Unity and manages to provide quality rendering when running in Oculus VR devices, it turns out that with the existing capturing tools for Unity it is not an easy task to preserve this quality in a 360° video. In order to see the pros and cons of the different approaches which I have tried, you should continue reading this post.

Why 360° video?

First of all, let’s see what are the benefits of having a 360° video and why we wanted to provide one, together with our demo application source code.

• Providing the source code of the demo is not enough. Although we have uploaded it with detailed documentation on how to use it and build it for different VR devices it certainly takes time for newcomers to download it and get acquainted with the development workflow.
• Providing already built APK and EXE files is still not enough. Although this would make it easier to install the demo on an Oculus device, still you will have to follow some steps for turning the device in developer mode in order to install and run the application.
• Even if you manage to successfully install and run the application on a VR device, you will be the only participant in a virtual room. In order to see the full functionality of the demo, you should install the app on several devices and have a group of people that will join the virtual room together.
• A 360° video would allow us to overcome the drawbacks from the previously listed approaches and practically provides the fastest way for the visitors of our web page to immerse in the demo room and see what functionalities are implemented in it.
• A 360° video also provides access to a larger set of people that may want to see the demo. With it, you are not required to have a VR device. You may easily see it on your smartphone and even the visitors that are using a desktop browser will be able to drag the mouse over the browser player and look around the virtual room in any desired direction.

How to record 360° video?

First, let’s say a few words on the scenario we wanted to record. It is a sample collaboration between several people in a virtual conference room. I and my colleagues Panayot and Georgi have run the demo application on Oculus devices and have joined the virtual room over a WiFi network. During this collaboration, there is always one of us that is in “presenter mode” and points different aspects on a chart data visualization. At the same time, the others are in “viewer mode” and see in front of them whatever is currently presented. When any of the viewers requests control using the Oculus controller, he becomes the currently presenting participant. We wanted to record the video in such a way that the visitors of our web site can easily look from the position of each of the presenters in the virtual room.

Recorder and Replayer

Most of the tools for capturing 360° video require quite a lot of time for capturing every frame of the video. This practically means that the video cannot be captured in real time and instead we need to somehow record all moving objects in the scene so that the movement can later be replayed frame by frame. We have created two Unity scripts that will be responsible for recording and respectively replaying the changes in our scene that occur during the collaboration. In this case we have recorded the following object properties:

• Chart transform. It changes every time the presenter moves or rotates the data visualization.
• Input transform. It changes every time the presenter moves his hand in order to show some aspect of the data.
• Avatars transform and avatar packets. These are the properties describing the current state of a participant’s avatar. Whenever someone moves his hand or head, new avatar packets are received and we should record this data in order to replay it later.
• Camera transform. When we project our demo on a 2D screen we use a spectator camera implementation which allows you to see from the eyes of the currently presenting user. In 2D display representation, it is fine if the camera moves together with the presenter's head. However, if we capture such movement on a 360° video and then play it on a VR device it is very likely that the VR user receives motion sickness from the intense camera movement. That is why instead of moving the camera with the head we have made it static and with the same position as the currently presenting participant. Changes in camera positioning are performed only when the presenter changes.

I will not get deeper into the implementation details of the recorder and replayer scripts in this post. We plan to publish a separate post on this matter in our Telerik AR VR series, so if you are interested in the implementation details, stay tuned for one of our next blog posts.

Tools for recording 360° video in Unity

So now that we have set up our scenario we are ready to replay it and try the different tools for Unity to see which best suits our needs for recording a 360° video.

The first approach — Unity built-in capturing API

At first glance, the built-in capturing functionality seems very tempting. It uses the Camera.RenderToCubemap method to project the space onto two cubes — one for the left eye and one for the right one. Then the result RenderTextures can be converted to an equirect image and a sequence of such images can easily be recorded by Unity frame recorder to a 360° video.

However, while testing this approach with our scenario we encountered the following issues:

• The graphics were a bit blurry even on higher capturing resolutions. This may not be an issue for some dynamic scene that does not require pixel perfect rendering but in our chart data visualization, there are a lot of text elements that become unreadable (even the ones that are close to the camera).
• The RenderToCubemap method has known limitation that it does not capture any UI elements. As our visualization shows some 2D graphics positioned in 3D space in the video it turned out that these graphics are missing. We have managed to find a possible workaround in Unity forums, however, it did not entirely solve the missing graphics issue in our scenario and we needed to extend its implementation for some of the TextMesh Pro instances in our scene.

As the built-in Unity capturing API did not work well in our case, we have decided to try some of the paid plugins in the asset store.

The second approach — VR Panorama 360 PRO Renderer

This paid asset is one of the popular assets for rendering videos in a Unity scene. It generally creates an image for every frame of the video by making camera snapshots in several directions and then stitching these snapshots to create the 360° image. After all frames are generated you can render a video file from the existing images. The video should then be injected with specific metadata that helps the video players to recognize it as 360° content.

VR Panorama asset provides a large set of options in its VRCapture script related to video format and video quality. One important option is the Capture Type. Other options regarding the quality are the Sequence Format (JPG or PNG), the Resolution and a Speed vs Quality parameter which controls the Anti-Aliasing. Let’s see what our findings are after testing different combinations of these options.

As you may see in the picture above there are two stereo options (the first one is with top-bottom layout while the second one is with side-by-side layout). As we target our 360° video for VR devices we would like to benefit from the stereoscopic displays for human realistic 3D visualization. That is why our first try is to render the video with some of the stereo options.

As we wanted to upload the video on Youtube with the best possible quality we have selected the Youtube 8k resolution preset and the Sequence Format is set to PNG (instead of the default JPG). At first glance, the result frame image looks with really good quality without blurriness of the graphics.

However, having a better look we have noticed that only the text and graphics in the center of the view are perfectly rendered. Looking left or right one may notice that the text and the chart bars are becoming double and this is reproducible both on the left eye image part and right eye image part.

This result feels particularly unpleasant when viewing the video with a VR headset and is totally unacceptable for our use case where the viewers should be able to easily read the labels in the chart. We have tried several options changing the resolution and the sequence format, but whenever the Capturing Type is set to some of the stereo options this double vision issue is persistent. Most probably it results from some error during the stitching algorithm that combines images from different view angles into a single 360° image.

With these issues in the stereoscopic capture type, our only option with VR Panorama asset was to try the monoscopic rendering. With this option, we won’t be able to benefit from the sense of depth provided by the VR devices, however, people will still be able to enter our virtual room and look around from the presenters' positions in all direction. So here is what our test with monoscopic rendering managed to produce for a single frame of the video using the maximum quality setting both for resolution and anti-aliasing:

Generally, the graphics were captured really sharp and all texts were easily readable without any double vision effects. When testing the video on an Oculus VR device it also looks sharp and with good quality. The only issues we had during the full video capturing were related to the performance and memory consumption of the VR Panorama tool — it took more than 10 hours to capture a 4-minute long monoscopic video and what is worse — the capturing process often crashes in the middle of this process. The calculations for generating every frame as a high-resolution PNG image are also very greedy of hard drive space — for a 4k monoscopic video, the images retained about 100 GB of memory. For 8k video, the needed resources are more than twice that big.

So, let’s sum up the pros and cons of the VR Panorama asset in our use case.

Pros:

• We managed to record 8k monoscopic 360° video which was looking fine on Youtube both for desktop, for mobile and for VR headsets.
• The tool is easy to use and has good documentation on the different provided options.

Cons:

• Recording stereoscopic video has stitching issues which lead to unacceptable quality in our scenario.
• The process of capturing the video takes a lot of time as it required rendering of one big image for every frame.
• The process of capturing the video requires a lot of disk space as the frame images are kept on the hard drive before the video generation.
• The capturing process often crashes the Unity editor for longer videos — in our case, the crash occurred when the captured video length was longer than 2 minutes.

The third approach — AVPro Movie Capture

As we could not achieve our go to record a stereoscopic video so far, we have decided to test one of the other paid assets from Unity Asset Store. AVPro asset provides a free version with 10 seconds recording limit which was enough for us to try its capabilities. In general, this asset uses a similar approach as VR Panorama and generates high-resolution PNG images for each frame. Here is what our findings showed when trying AVPro and comparing it to the VR Panorama asset.

Pros:

• AVPro managed to render sharp and high-quality images when rendering in stereoscopic mode.
• AVPro frames did not have the VR Panorama issue with double-vision text and graphics when looking away from the central view direction.

Cons:

• When viewing the result 360° video with a VR device the chart labels were blurry and hardly readable. This is probably related to an issue with correctly overlapping the left and right eye images and the result was unacceptable for our use case.
• The rendering of the frames seemed even slower compared to the VR Panorama asset.

As we could not create a stereoscopic video with acceptable text quality we decided not to test further this asset. We were already able to create a good quality monoscopic video with the VR Panorama tool.

The fourth and last approach — Facebook 360 Capture SDK

As a last try to capture stereoscopic video we tested the FBCapture SDK. It was looking very promising in sense of speed, however, it turned out that the produced quality was not acceptable for us (even for the monoscopic scenario). It uses the same RenderToCubemap method as in our first approach but provides an easy to use FBCapture prefab which has a variety of options for controlling the output quality. Because of its easy usage and really performant rendering, I still believe that it is worth mentioning the pros and cons we have found out while testing this SDK.

Pros:

• Real-time capturing. As it uses RenderToCubemap method combined with shaders for creating the video frames it does not take hours to render the video which is a good advantage compared to the previous two approaches.
• Small memory consumption. The result MP4 files are also some times smaller compared to the ones produces with VR Panorama and AVPro assets.
• Easy to use — simply drag and drop the FBCapture prefab and use its Hotkeys for start and stop encoding the video.

Cons:

• The SDK restricts the maximum size for video capturing to 4k resolution. Both VR Panorama and AVPro were allowing 8k resolution.
• The captured graphics are not sharp enough which makes the labels blurry and hardly readable. This is reproducible for both monoscopic and stereoscopic videos.
• The usage of RenderToCubemap method comes with the limitation for capturing UI elements. As in the first approach we had to try the workaround from Unity forums with some additional implementation modification in order to make it work for our scenario.
• We could not make the stereoscopic video to work in any of the 360° video players. In order to capture stereoscopic video with this Facebook SDK, you should select the RGBD_Capture texture format which creates a gray depth image parallel to the original monoscopic 360° image. However, neither Youtube nor Facebook video players were able to recognize this video format and we could not find any metadata injector that could make it recognizable. All metadata injectors seem to be working with top-bottom or side-by-side image formats and none of them is providing the option for a depth channel.

Conclusion

Although our demo application manages to show text and graphics with sharp edges on VR stereoscopic displays, we were not able to find a suitable 360 capturing tool that manages to persist this visualization quality in a stereoscopic video. That is why we ended up creating a monoscopic video which loses the sense of depth, but at least allows the viewers to easily read the text and graphics presented in our demo application.

I hope this shared experience will be helpful and time-saving for someone who is about to start their own challenge of capturing a 360° video. Feedback is welcome — feel free to share your thoughts about our or your own experience with the available tools for Unity.

Capturing 360° video from a VR DataViz application — trials and errors was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

At Progress Telerik we’ve been on the AR VR journey for more than a year now. The path has always been challenging and rewarding and it is now even more exciting. We learned a lot, made some mistakes and even shifted our initial strategy. In this post we will look retrospectively at our findings and talk about the new path that we’ve taken.

This post assumes that when we talk AR VR, we relate to Head-Mounted Devices. Smartphone AR VR is out of our scope today.

Web Presence

Before we continue, we have an important and super exciting announcement to make: a completely new and redesigned page that fully matches our current vision, strategy and short-term roadmap is now live on our company web site!

We will keep this page updated with our latest product announcements and plans. This is also the place where you can reach out to us and/or subscribe to our news and updates.

Besides some plain text that explains where we are heading to and what problems we solve, the page also features delightful graphics and several exciting videos, including a 360-degree panoramic capture of one of our applications. We have even enabled our recent DataViz applications for download, including complete Unity3D projects and source code!

In case you are already curious to see our new web face in action, please go ahead, you may continue reading the post afterwards :)

Visit https://www.telerik.com/ar-vr-lab

A Retrospective Look Back

We are developers. It is in our DNA to help other developers and businesses solve their challenges in an elegant and efficient way. Our long-term mission is to democratize the AR VR business application development and reduce the entry level barrier for developers. One way to accomplish our mission was to start building what we have most experience with — a set of UI and non-UI tools, frameworks and components for Unity3D that would reduce the entry level barrier for developers and would eliminate the need for complex user experiences to be continuously reinvented for each and every application.

Naturally, we took that path. If you remember our first post, it was published straight after Microsoft Build 2018, where we publicly revealed our Company’s plans to invest in this space. We imagined that once we create our initial UI for AR VR offering developers would jump on it and we will slowly start delivering on our mission.

Obviously that was not the case and we are on a different path today.

Early Market State of Affairs

You cannot sell components in an early market. There is simply no need.

We learned that from the feedback and traction that we gathered after MS Build ’18. Which was close to zero. Although related to Microsoft’s HoloLens 1, this post summarizes our findings back then. The following list outlines our most recent conclusions:

1. Most of the businesses are looking to enable AR VR in some form and take competitive advantage from this technology but they struggle to find a valuable use case that’s worth investing in.
2. Often times businesses have this emotional barrier to use today’s AR VR hardware as it is far from being socially acceptable. They would rather stick to conventional methods for solving a problem than putting a helmet on their heads.
3. This is a completely new technology. The development stack heavily involves 3D and game development-like expertise, therefore most of the businesses lack AR VR R&D team(s) and consider applying this technology expensive, with a steep learning curve and with significant initial investment.

As you can tell from the list, what we were trying to solve was directly the third problem — to make the development far more easier. Needless to say, that won’t work until the first two problems are solved. That is, valuable use cases to be defined and businesses being able to see value in the technology. Only then will they be ready to challenge their socially-related barrier of putting a helmet on.

What Else Could We Do?

We spent a lot of time to think what to do and how to do it. We challenged ourselves to leave our comfort zone of building what we can easily do — UI for AR VR — the truth is that the market is simply not there. Instead, we decided to tackle the first two problems first.

While this may sound easy, it was challenging to implement because:

1. The use case(s) that we needed to find should clearly demonstrate the full glory of the technology.
2. Most, if not all, of the business verticals where AR VR adds value are occupied with various solutions, coming from small to large companies.
3. Our company builds horizontal solutions that fit multiple business verticals — our ultimate goal is to build a horizontal solution as well.
4. Just a use case on a Power Point presentation wouldn’t suffice, the technology needs to be experienced. Thus, a working application should be demonstrated, instead of text and images.

Top Values of AR VR for Business

When we talk about “Technology Value”, we relate to:

Immersive experiences through Stereoscopic 3D

Like in 3D movies in theaters, Stereoscopic 3D is a technique that produces human-realistic sense of depth by displaying two slightly different images to the left and right eyes of the viewer. The ability is based on the characteristics of the human visual system.

This aspect of the technology makes our brain to literally experience every part of an application. It is just how our brain it works. The difference between conventional 2D displays and Stereoscopic 3D displays is like watching a movie of the Pyramids and being on a trip to Egypt.

Natural interactions with digital content

Much like in real life, we can interact with digital content through natural gestures. For example, if I need to push a button in VR, I would simply reach out and press it with my hand. Or, if I need to move an object somewhere in space I would simply grab and move it.

Truly social multi-people interaction and collaboration

Because of the human-realistic experiences in Stereoscopic 3D, sharing an experience with other people builds long-lasting memories. For example, instead of being bored to death at a remote meeting, why not have it in VR where you can interact with participants as if they are next to us?

Data Visualization

What is the single property that is across all business segments. What lies in the pulse of every business? The answers are simple — it is data. Visual communication — or Data Visualization as we call it today — has always been a significant part of humanity. Technologies change, new ones are invented, data evolves but producing visual means from a raw data-set will always be part of how we interact with information and ourselves.

Following that logical path we asked ourselves — can AR VR add value to the way we visualize data today? Can Data Visualization take advantage of the above listed strengths of the technology?

We did some experiments and were blown away by the enormous value Stereoscopic 3D adds to seeing even simple data:

• Many people will argue that data is best seen on 2D. That is true, given we talk about 2D-displays. 3D on a 2D display is almost every time a bad thing to do. But it is a completely different story when you see 3D on a 3D display. And when we talk about Big and Wide data, these human-realistic 3D experiences offer enormous potential, unexplored before.
• Using human-natural interactions like moving charts all around the place, zooming in-and-out, walking through the data and seeing it from different angles, improves readability and understanding of data multiple times.
• Presenting data to other people in VR takes full advantage of the social-like collaboration experiences and contributes significantly to the collective understanding of data.

VR or AR?

After we defined our path and strategy, we needed to choose the right device that is suitable for displaying data visualizations in the most optimal way in terms of image quality, field of view and price.

The requirements that we have are:

• As large as possible field of view. Big or Wide data requires significant virtual space and viewing angle. The minimum acceptable angle is 100 degrees diagonal.
• Stand-alone. Pitching a use case that requires tethering machine would be less appealing than a stand-alone device that one may carry anywhere on the planet.
• Affordable price. One of the barriers for businesses to embrace the technology is the investment needed. We defined a price tag of a maximum $800.

Where is AR Today

This engineering masterpiece features:

• MEMS displays, 52-degrees diagonal field of view
• Full hand and eye tracking for natural interactions
• Perfect balance for long hours of comfortable hands free work
• Pumped up with field worker productivity software
• Azure software sharing HoloLens, iOS and Android experiences
• Enterprise oriented

Producing AR capable hardware is orders of magnitude more complex than building VR hardware and this explains the $3500-ish price tag on the device.

Although a solid device, HoloLens 2 is incompatible with our requirements of 100+ degrees field of view and $800 or less price tag. These aside, the device, as it is designed, is practical for a variety of enterprise use case but these are already taken.

Where is VR Today?

We have a stand-alone Oculus GO device with a price tag of $200, and we are expecting Oculus Quest any time soon: “Coming in 2019. Starting at $399”.

At a nearly “impulse buy” price tag you get:

• Same FoV as desktop attached Oculus Rift — that is 110 degrees diagonal
• Tracking with 6 DoF touch controllers for natural interactions
• Shared arena-scale experiences
• Excellent image quality

There is a trend there, based on the better imaging capabilities and availability of VR devices — higher FoV, real life-like imaging, completely immersed experiences.

While AR is more exciting and somewhat more socially acceptable when working with multiple people in the same physical room, its price tag and limited visualization capabilities turn the tide in favor of VR.

The Future of Data Visualization

We believe AR and VR at some point will converge at a single device that will be capable to deliver both kind of virtual experiences and anything in between. Somewhere along that road AR may become better suited for Data Visualization and thanks to the cross-platform support from Unity3D, our solution will run on AR devices as well.

10 years from now we expect a single socially acceptable device that can deliver both AR and VR experiences, to be wildly available at offices. At that point the decision whether the application will run in AR or VR mode will finally be made on business needs instead of hardware capabilities.

For example:

• Executive live meetings that happen in the same physical room will use the devices in AR mode. These will be capable to deliver report experiences far more richer than traditional 2D displays.
• Remote meetings will use the device in VR mode. These will share environments, avatars or video presence and will transfer micro-expressions. That would be a far better communication channel than 2D video conference calls. It will make people feel they are in the same room. And again, it will be capable to deliver report experiences far more richer than traditional 2D displays.

You can capture a glimpse of that future:

Back at Microsoft Build 2018 we had a fully functional data visualization demo displaying the bitcoin stock movements. The enthusiasm about the demo was huge, however the benefits of the third dimension were mitigated by the form factor, FoV and price of the device.

Now Microsoft Build 2019 is on the door step. HoloLens 2 is a very strong release. But at this point it is focused primarily on enterprises and field workers. It will take few years before miniaturization and mass production can cut the shape and cost of the device enough to bring AR to the state where Data Visualization will shine.

On the other hand VR is already here:

The VR field is fully saturated and competition has already pushed the hardware and software to a point where Data Visualization can show its full potential.

Wrap Up

We will take the VR Data Visualization path, aiming to help businesses find valuable use cases for AR VR and apply the technology successfully today. One of our long-term goals remains a comprehensive framework of reusable components and tools, but this will happen gradually, after the market picks up.

Technologically, we are backed by Unity3D, which is one of the very few truly cross-platform frameworks. This ensures that when AR as hardware picks up we will be prepared. Unity3D also enables us to deploy to literally any VR device that exists today and will be born tomorrow.

To fully demonstrate the potential of Data Visualization in VR, we created two completely functional applications available for download. Head-on to our new page and see what we have there for you:

Visit https://www.telerik.com/ar-vr-lab

Thank you for reading this post! As usual — feedback is gladly accepted — please feel free to share it with us either on post’s comments or at our email:

vrlabs@telerik.com

Happy experimenting and we look forward to hearing from you.

Data Visualization Taken to the Next Dimension through VR was originally published in Telerik AR VR on Medium, where people are continuing the conversation by highlighting and responding to this story.