Vulkan on Android 6 — VR Stereo Rendering Part 5: Implementation

6 min readFeb 2, 2019

Full sample here.

In this post, we are going to combine all the components we mentioned earlier in the series and apply device orientation to the virtual camera.

Command Buffers

_msaaCommandBuffers and _commandBuffers are used for off-screen and multi-view rendering respectively. They are both have the same number of VkCommandBuffers as each element of them is specific to the particular off-screen color(for _msaaCommandBuffers) and swapchain(for _commandBuffers) image.

The procedures of off-screen and multi-view rendering for both eyes are similar except for the arguments’ values.

In off-screen drawing commands we take the following steps:

Bind the vertex and index buffer(_modelResources[0]) we are going to draw.
Starts a render pass instance which takes msaaRenderPass as the configuration and uses _lMsaaFramebuffers/_rMsaaFramebuffers as the rendering targets for the left/right eye.
Bind _msaaPipeline to affect subsequent commands.
Bind _msaaDescriptorSet with _msaaPipelineLayout to access the resources needed. dynamicOffsets is initially 0 to access the viewing parameters of the left eye, and it increases to _dynamicBufferAlignment to access the viewing parameters of the right eye.
Draw the buffer bound and end the render pass instance.

In multi-view drawing commands:

Starts a render pass instance which takes resolvedImageRenderPass as the configuration and uses framebuffers as the rendering targets for the both eyes.
Bind _multiviewPipeline to affect subsequent commands.
Bind the vertex and index buffer(_modelResources[1]) of the square plane which we are going to draw.
As we have specified that we are going to use dynamic viewports and scissors in the _multiviewPipeline creation, we are able to set up the separate viewports and scissor for both eyes.
Bind the element at the location index of _lDescriptorSets/_rDescriptorSets for the left/right eye, perspectively. It is because off-screen rendering results are different between frames.
Draw the buffer bound and end the render pass instance.

Render Loop

As before, we build framebuffers and command buffers in each loop. The particular element of framebuffers and command buffers depends on the index(imageIndex) retrieved from vkAcquireNextImageKHR. _lMsaaViews , _lMsaaDepths, and _lMsaaResolvedViews are color, depth, resolve attachments of _lMsaaFramebuffers for the left eye while _rMsaaViews, _rMsaaDepths, and _rMsaaResolvedViews are color, depth, resolve attachments of _rMsaaFramebuffers for the right eye. And swapchain->ImageViews() are the only color attachments of framebuffers for presentation.

After submitting _msaaCommandBuffers.buffers[imageIndex] we have to wait for it to complete as we demand a complete rendering result as the sampling image for the next submission. When it finishes, it is safe to release the framebuffers we create at the beginning of the loop.

When off-screen drawing completes, we are allowed to submit the multi-view commands and specify a semaphore(commandsCompleteSemaphores[currentFrameIndex]) to signal for a complete. And, QueuePresent can wait for the semaphore before it starts to present the image.

currentFrameToImageindex records the currentFrameIndex/imageIndex pairs since no fences are used between the last submission and presentation. That is to say, we have no chance to know which fence to wait at the beginning of the loop because currentFrameIndex may have been increased many times so we have to check currentFrameToImageindex to see if the element at the currentFrameIndex of multiFrameFences has been submitted before so we can safely wait on it or validation layers will complain about it.

Barrel Distortion

We use Brown’s distortion model to apply barrel distortion in vertex or fragment shader stage. Another method is published on Google IO 2016. it distorts the whole scene at first and renders them which saves tremendous time. However, old Unity Cardboard SDK implementation of the method doesn’t take lights or skybox into account, and I can not find them anymore in today’s Google VR SDK. And I don’t see any further application after the publication. If you ever find anyone, please let me know, thanks!

Implementation in vertex or fragment shader varies in targets to which distortion is applied. In the fragment shader, we distort the image pixel by pixel. While distortion in vertex shader needs a temporary full-screen plane where the off-screen image is mapped for each eye and it distorts the temporary plane. It’s just a trade-off between processing time and accuracy, but I found hundreds or thousands of vertices on plane produce a satisfactory result. So in this post, I chose to distort in the vertex shader. For instance, create a plane with 25 vertices.

We could distort these 25 vertices.

And hardware will interpolate all the other things like normals, texture coordinates, colors for us.

Shaders

texture.vert/texture.frag are simple texture mapping shaders while multiview.vert/multiview.frag do the image distortion job.

In multiview.vert, we can use as many as k-terms we want with a greater granularity of shape-controlling, however, most shapes we see in VR apps use one k term only. Note here we pass our position directly to texture coordinates for the fragment shader.

In multiview.frag, since we know the plane coordinates are in the range [-1, 1], we have to remap them to range [0, 1] for texture mapping.

Apply Orientation of Device Sensor to Virtual Camera

Now we have stereo views, and we want to explore the world with the camera. So we need our camera stay synchronized with our device’s orientation. For Android, four types of sensor are related to rotation. And we are going to use the type rotation vector since it returns a quaternion and thus is free of gimbal lock.

Using sensor in NDK involves

ASensorManager managing sensors and events queues.
ASensor provides information about a hardware sensor.
ASensorEvent contains all data about detected values dependent on sensor type.
ASensorEventQueue provides access to ASensorEvent from hardware sensors.

The constructor of EvenLoop initializes the sensor manager, event queue, and event rate, etc. And later we poll rotation vector sensor data in EventLoop::Run.

We delegate data receiving to onGetRotationVector, which is assigned to OnGetRotationVector in the constructor of StereoViewingScene. It receives a 4-float array representing a quaternion to be transformed into a rotation matrix which is later assigned to cameraRotationMatrix. Rotation vector is a composite sensor which uses an accelerometer, magnetic field sensor, gyroscope, etc.

And its coordinate system is defined as:

X is defined as the vector product Y x Z. It is tangential to the ground at the device’s current location and points approximately East.
Y is tangential to the ground at the device’s current location and points toward the geomagnetic North Pole.
Z points toward the sky and is perpendicular to the ground plane.

https://developer.android.com/guide/topics/sensors/sensors_motion#sensors-motion-rotate

However, it does not fully meet our requirement because we want the other orientation to be our rotation origin.

Rotation from three axes of frame reference 1 to those of frame reference 2 is specified in base of quaternion form, which is rotating on Y axis by -90 °. But it rotates the axes not the frame of reference itself and it just set the rotation axes aligned with the frame of reference 2. Subsequent rotations still apply on the frame of reference 1. So we reorder the quaternion components so that rotations apply to correct axes. When the rotation data is acquired from the sensor in quaternion form, we should transform it into matrix form and it is then be used as the value of view transform of _lViewProjTransform and _rViewProjTransform.