Vulkan on Android 6 — VR Stereo Rendering Part 5: Implementation
Full sample here.
In this post, we are going to combine all the components we mentioned earlier in the series and apply device orientation to the virtual camera.
Command Buffers
_msaaCommandBuffers
and _commandBuffers
are used for off-screen and multi-view rendering respectively. They are both have the same number of VkCommandBuffer
s as each element of them is specific to the particular off-screen color(for _msaaCommandBuffers
) and swapchain(for _commandBuffers
) image.
The procedures of off-screen and multi-view rendering for both eyes are similar except for the arguments’ values.
In off-screen drawing commands we take the following steps:
- Bind the vertex and index buffer(
_modelResources[0]
) we are going to draw. - Starts a render pass instance which takes
msaaRenderPass
as the configuration and uses_lMsaaFramebuffers
/_rMsaaFramebuffers
as the rendering targets for the left/right eye. - Bind
_msaaPipeline
to affect subsequent commands. - Bind
_msaaDescriptorSet
with_msaaPipelineLayout
to access the resources needed.dynamicOffsets
is initially 0 to access the viewing parameters of the left eye, and it increases to_dynamicBufferAlignment
to access the viewing parameters of the right eye. - Draw the buffer bound and end the render pass instance.
In multi-view drawing commands:
- Starts a render pass instance which takes
resolvedImageRenderPass
as the configuration and usesframebuffers
as the rendering targets for the both eyes. - Bind
_multiviewPipeline
to affect subsequent commands. - Bind the vertex and index buffer(
_modelResources[1]
) of the square plane which we are going to draw. - As we have specified that we are going to use dynamic viewports and scissors in the
_multiviewPipeline
creation, we are able to set up the separate viewports and scissor for both eyes. - Bind the element at the location
index
of_lDescriptorSets
/_rDescriptorSets
for the left/right eye, perspectively. It is because off-screen rendering results are different between frames. - Draw the buffer bound and end the render pass instance.
Render Loop
As before, we build framebuffers and command buffers in each loop. The particular element of framebuffers and command buffers depends on the index(imageIndex
) retrieved from vkAcquireNextImageKHR
. _lMsaaViews
, _lMsaaDepths
, and _lMsaaResolvedViews
are color, depth, resolve attachments of _lMsaaFramebuffers
for the left eye while _rMsaaViews
, _rMsaaDepths
, and _rMsaaResolvedViews
are color, depth, resolve attachments of _rMsaaFramebuffers
for the right eye. And swapchain->ImageViews()
are the only color attachments of framebuffers
for presentation.
After submitting _msaaCommandBuffers.buffers[imageIndex]
we have to wait for it to complete as we demand a complete rendering result as the sampling image for the next submission. When it finishes, it is safe to release the framebuffers we create at the beginning of the loop.
When off-screen drawing completes, we are allowed to submit the multi-view commands and specify a semaphore(commandsCompleteSemaphores[currentFrameIndex]
) to signal for a complete. And, QueuePresent
can wait for the semaphore before it starts to present the image.
currentFrameToImageindex
records the currentFrameIndex
/imageIndex
pairs since no fences are used between the last submission and presentation. That is to say, we have no chance to know which fence to wait at the beginning of the loop because currentFrameIndex
may have been increased many times so we have to check currentFrameToImageindex
to see if the element at the currentFrameIndex
of multiFrameFences
has been submitted before so we can safely wait on it or validation layers will complain about it.
Barrel Distortion
We use Brown’s distortion model to apply barrel distortion in vertex or fragment shader stage. Another method is published on Google IO 2016. it distorts the whole scene at first and renders them which saves tremendous time. However, old Unity Cardboard SDK implementation of the method doesn’t take lights or skybox into account, and I can not find them anymore in today’s Google VR SDK. And I don’t see any further application after the publication. If you ever find anyone, please let me know, thanks!
Implementation in vertex or fragment shader varies in targets to which distortion is applied. In the fragment shader, we distort the image pixel by pixel. While distortion in vertex shader needs a temporary full-screen plane where the off-screen image is mapped for each eye and it distorts the temporary plane. It’s just a trade-off between processing time and accuracy, but I found hundreds or thousands of vertices on plane produce a satisfactory result. So in this post, I chose to distort in the vertex shader. For instance, create a plane with 25 vertices.
We could distort these 25 vertices.
And hardware will interpolate all the other things like normals, texture coordinates, colors for us.
Shaders
texture.vert
/texture.frag
are simple texture mapping shaders while multiview.vert
/multiview.frag
do the image distortion job.
In multiview.vert
, we can use as many as k-terms we want with a greater granularity of shape-controlling, however, most shapes we see in VR apps use one k term only. Note here we pass our position directly to texture coordinates for the fragment shader.
In multiview.frag
, since we know the plane coordinates are in the range [-1, 1], we have to remap them to range [0, 1] for texture mapping.
Apply Orientation of Device Sensor to Virtual Camera
Now we have stereo views, and we want to explore the world with the camera. So we need our camera stay synchronized with our device’s orientation. For Android, four types of sensor are related to rotation. And we are going to use the type rotation vector since it returns a quaternion and thus is free of gimbal lock.
Using sensor in NDK involves
ASensorManager
managing sensors and events queues.ASensor
provides information about a hardware sensor.ASensorEvent
contains all data about detected values dependent on sensor type.ASensorEventQueue
provides access toASensorEvent
from hardware sensors.
The constructor of EvenLoop
initializes the sensor manager, event queue, and event rate, etc. And later we poll rotation vector sensor data in EventLoop::Run
.
We delegate data receiving to onGetRotationVector
, which is assigned to OnGetRotationVector
in the constructor of StereoViewingScene
. It receives a 4-float array representing a quaternion to be transformed into a rotation matrix which is later assigned to cameraRotationMatrix
. Rotation vector is a composite sensor which uses an accelerometer, magnetic field sensor, gyroscope, etc.
And its coordinate system is defined as:
X is defined as the vector product Y x Z. It is tangential to the ground at the device’s current location and points approximately East.
Y is tangential to the ground at the device’s current location and points toward the geomagnetic North Pole.
Z points toward the sky and is perpendicular to the ground plane.
However, it does not fully meet our requirement because we want the other orientation to be our rotation origin.
Rotation from three axes of frame reference 1 to those of frame reference 2 is specified in base
of quaternion form, which is rotating on Y axis by -90 °. But it rotates the axes not the frame of reference itself and it just set the rotation axes aligned with the frame of reference 2. Subsequent rotations still apply on the frame of reference 1. So we reorder the quaternion components so that rotations apply to correct axes. When the rotation data is acquired from the sensor in quaternion form, we should transform it into matrix form and it is then be used as the value of view transform of _lViewProjTransform
and _rViewProjTransform
.