How To Encode MV-HEVC Video With FFmpeg

How to encode 3D stereoscopic video into MV-HEVC spatial and how to decode MV-HEVC spatial video into 3D stereoscopic side-by-side.

At SpatialGen, we do a lot of work with custom encoders so we can stream the world’s most innovative spatial and immersive video. For those of intense curiousity, in this article, we will discuss the steps to encode a stereoscopic video into MV-HEVC utilizing FFmpeg. It is not easy yet, so if you just want to get streaming, consider checking out SpatialGen or getting in touch.

What is MV-HEVC?

MV-HEVC stands for “Multiview High Efficiency Video Coding.” It is mostly known in reference to Apple’s “Spatial” or “Immersive” video. This is a type of 3D video usually derived from a stereoscopic source.

A 3D video should look something like this with the left eye on the left of the frame and the right eye on the right:

A stereoscopic video

MV-HEVC is a codec for organizing the video data in a more efficient way. It’s primary advantages are two-fold:

1. Smaller File Sizes
   MV-HEVC can store the difference between the left and right eyes instead of the full view, which provides a smaller file size than a stereoscopic video. This helps push streaming costs lower and makes spatial content available to households with low bandwidth internet.
2. 2D Native Playback Capability
   The MV-HEVC codec is able to be played back in 2D on iOS or macOS. This helps native 3D video files find a larger audience than just those with a Quest or Apple Vision Pro.

How To Decode MV-HEVC Into Stereoscopic Side-By-Side

Make sure you have the latest version of FFmpeg. I am running this on macOS with FFmpeg version 7.1.

The FFmpeg command to decode a MV-HEVC video into stereoscopic SBS is the following:

ffmpeg -i mvhevc_video.mov -filter_complex \ 
"[0:v:view:0][0:v:view:1]hstack" -c:v hevc \
-tag:v hvc1 output_sbs.mp4

How To Encode Stereoscopic Side-By-Side To MV-HEVC

Hopefully one day this will be a simple command, but for today, there is a process. FFmpeg taps into x265 for it’s codec. MV-HEVC is an “extension” of the HEVC codec, which is the common term for x265.

While FFmpeg supports decoding MV-HEVC videos, it does not have easy support for MV-HEVC. To encode a video into MV-HEVC, you need to download the latest version (4.0 as of this writing) of x265 and build it from source. The result will not contain metadata which means the Vision Pro will not recognize the file as spatial or immersive. If this is too much, consider checking out SpatialGen. Now that we got all of that out of the way, let’s get started!

1. Download or clone the repository and make sure you have version 4.0. The September 13th, 2024 release has MV-HEVC support. Make sure to unzip the repo if you download it. The bitbucket repo is where the code is stored.

I have tested this on macOS. Windows may require some extra steps according to the x265 build documentation.

2) Generate the make files.

Open a terminal, cd to the build folder and run:

cmake -DENABLE_MULTIVIEW=ON ../source

That will generate your make files.

3) Create the executable

Since we need support for MV-HEVC, we need to set a flag when we call “make”

make ENABLE_MULTIVIEW=ON

This will generate a “x265” executable we can use for MV-HEVC encoding.

4) Create a raw YUV file of your input video. This is a file x265 can take in. There is likely a possibility in the future of piping the frames from FFmpeg to x265 so you don’t have to write a file. Note that this will create a large file.

ffmpeg -i stereo_input.mp4 -pix_fmt yuv420p -f rawvideo stereo_raw.yuv

5) Create a mv_config.cfg file for the x265 encoder. This file specifies details about your input file, such as the number of views, their format, and your input name. Save this file

#Configure number of views in the multiview input video#
#--num-views <integer>#
--num-views 2

#Configuration for the input format of the video#
#--format <integer>#
# 0 : Two seperate input frames#
# 1 : One input frame with left and right view#
# 2 : One input frame with top and bottom view#
--format 1

#Configure input file path for each view#
##NOTE:Other input parameters such as input-csp/input-depth/fps must be configured through CLI##
--input "stereo_raw.yuv"

6) Run x265 to encode a MV-HEVC source. NOTE: Provide the correct fps and input resolution, which should be half your video’s width if SBS stereoscopic.

./x265 --multiview-config mv_config.cfg \
--fps 23.98 --input-res 960x1080 --output mvhevc_output.hevc \
--profile main10 --colorprim bt709 \
--transfer bt709 --colormatrix bt709

7) Mux the video into a container for playback

ffmpeg -i mvhevc_output.hevc -c copy -tag:v hvc1 output.mov

Congrats! You have a MV-HEVC video!

Need A Better Solution?

If you want a Super Simple Streaming solution for the world’s most innovative and demanding videos, consider reaching out to us at SpatialGen.

Sign up today, or shoot us an email. https://spatialgen.com