FFmpeg — Your powerful video/audio helper in your application

Classifai is extending the annotation of data types on videos, and the team is actively exploring video related libraries. With the team's learning over there, I’ll talk about one of the libraries — FFmpeg library for handling multimedia files.

The name is inspired by MPEG, which means Moving Picture Expert Group. The FF means Fast Forward.
The MPEG group is the alliance of working groups behind setting many of the essential video standards today, including the MP4 format, which we are all familiar with.

Outcomes

1. What is FFmpeg?
2. Why do people use FFmpeg?
3. How to set up FFmpeg in Windows, Ubuntu, and macOS.
4. Libraries are available in FFmpeg for media manipulations.
5. Tools that FFmpeg comes with for command-line applications.
6. Usage and examples for FFprobe and FFplay.

Why do people use FFmpeg?

FFmpeg is often referred to as the Swiss Army knife of media transcoding or streaming. We can use FFmpeg to perform a lot of functions. The code is written in C language and optimized for best performance. FFmpeg commands are simple to run. Once you get familiar with the concepts, it is very flexible to all the filters and options you can use to fit your needs.

FFmpeg supports a wide range of codes, formats, devices, and protocols, making it an ideal choice as a transcoding engine. Unlike many discontinued projects, FFmpeg is still in active development for more than 20 years. There’s a large community of developers, users, and contributors who constantly develop new features and fixes.

FFmpeg has been used in the core processing for video platforms like YouTube and iTunes. Most of us used a media player like VLC to play video files. VLC uses FFmpeg libraries as its core. Some video editors and mobile applications also use FFmpeg under the hood.

Setting up FFmpeg in Windows, Ubuntu, and macOS

There are a few things you need to know before setting up FFmpeg. There’s no official build for any operating systems. But it is an open-source project, which you can get hold of the FFmpeg source code from the internet. So, you can choose to build FFmpeg yourself from the source code.

You can download a snapshot of the current source code here or do a checkout from their git repository and follow the instructions to compile it to your machine. This can be a little complex and time-consuming and may require you to understand and choose some configurations options. It may also need additional tools and compilers to be installed at first. So unless you have a good reason to build it yourself, you may not want to go this route.

Instead, you can choose to download and install a pre-build FFmpeg package for your operating system. There are two ways to set up these pre-build binaries. If you have a package manager like APT in Ubuntu and Homebrew in macOS, it is easier to use it since it takes care of downloading FFmpeg and its dependencies.

Let’s keep things as simple as possible and choose the easiest way to get started with FFmpeg. The procedures below show how you can set up using pre-build packages.

macOS

On macOS, the easiest way to get FFmpeg installed is by using Homebrew. Make sure you installed Homebrew in your macOS, or you can follow this tutorial on how to install Homebrew. We are using macOS 11.4 in this case.

• Open Terminal and type brew --version . You should see the brew version printed out in Terminal if Homebrew is installed successfully.

Brew version in macOS

• For macOS in x86 Architecture, use the commandbrew install ffmpeg , while for M1, please use the command aarch arm64 brew install ffmpeg

Downloading and installing necessary dependencies

• After the installation is done, you can type ffmpeg --version in Terminal. If the FFmpeg version is printed, it means FFmpeg is installed successfully.

Ubuntu

In Ubuntu, we will use the advanced Packaging Tool (APT) to install FFmpeg.

• Open Terminal and type sudo apt-get update to ensure the package information from all configured sources are all up to date.

Done updating the package information

• Type sudo apt install ffmpeg to install FFmpeg.

Downloading and installing FFmpeg packages.

• After the installation is done, you can type ffmpeg --version in Terminal. If the FFmpeg version is printed, it means FFmpeg is installed successfully.

Windows

Unlike APT on Ubuntu and Homebrew on macOS, there’s no well-known package manager on Windows, which you can use to install FFmpeg quickly. For Windows, you need to down and set it up manually.

There are 4 Windows FFmpeg build variants available can be found here:

1. git full - built from the master branch with a large set of libraries.
2. git essentials - built from the master branch with commonly-used libraries.
3. release full - built from the latest release branch with a large set of libraries.
4. release essentials - built from the latest release branch with commonly-used libraries.

We will use FFmpeg release essentials in our case.

• Download FFmpeg release essentials here.
• Extract the files to a place from where you will use them in future. In my case, I put them in my E Drive.

• Inside the bin folder is containing those executables we want.

• Add bin folder to Windows Environment Path Variable to run this tool without having to specify the full path every time. Search for Environment Variables in Search Bar and click on Environment Variables…

• Click on Path in User Variable, click New and paste the full path of the FFmpeg bin folder.

• Open Command Prompt and type ffmpeg -version . If you see the FFmpeg version is printed, it means FFmpeg is installed successfully. We also can check ffprob -version and ffplay -version .

FFmpeg Libraries

FFmpeg has several valuable libraries that you can use directly from your application code. Each library contains different functions related to a particular area.

Some of the famous libraries are:

1. libavcodec — Contains all the encoders and decoders that FFmpeg supports.
2. libavformat — Has all the muxers and demuxers for dealing with various containers format.
3. libavfilter — Consists of many filters that you can use to modify the audio or video according to the requirements.
4. libavdevice— Support several different input and output devices.
5. libavutil — Aid portable multimedia programming.
6. libswscale — Performs highly optimised image scaling and colourspace and pixel format conversion operations.
7. libswresample — Performs highly optimised audio resampling, remixing and sample format conversion operations.

FFmpeg Tools

Besides the libraries which libraries can use from other applications, the FFmpeg package contains a few tools on its own.

Some of the tools are:

1. ffmpeg — Main transcoding engine. It is often invoked from a command line or another process.
2. ffplay — A minimal tool for playing audio or video.
3. ffprobe — Quickly inspecting and media to extract valuable information, like how many streams there are in the media, the frame rate of a video, etc.

FFprobe Usage and Examples

Usage

ffprobe [OPTIONS] [INPUT_FILE]

• man ffprobe — Refer to the manual.
• ffprobe -h — Shows included help.
• -v error — Hide logs unless there’s an error.
• -show_format — Show formatted video metadata using Tag.
• show_streams — Show all the streams in the file (both video and audio).
• -print_format <format>—Print metadata in specific format (CSV/XML/JSON).
• -select_streams <video/audio> —Show specific streams of the video.
• -show_entries stream=<entity name> — Show only the specific entity and its nested dependencies.
• -show_entries stream=<entity name> default-noprint_wrappers=1 — Show only the specific entity.

You can find out more FFprob usage here.

Basic Examples

• Show all video metadata.

$ ffprobe -v error -show_format -show_streams input.mp4
Output:
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1/50
codec_tag_string=avc1
codec_tag=0x31637661
width=320
height=240
has_b_frames=2
sample_aspect_ratio=1:1
display_aspect_ratio=4:3
pix_fmt=yuv420p
level=13
color_range=N/A
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
timecode=N/A
refs=4
is_avc=1
nal_length_size=4
id=N/A
r_frame_rate=25/1
avg_frame_rate=25/1
time_base=1/12800
start_pts=0
start_time=0.000000
duration_ts=384000
duration=30.000000
bit_rate=34761
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=750
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:language=und
TAG:handler_name=VideoHandler
[/STREAM]
[STREAM]
index=1
codec_name=aac
codec_long_name=AAC (Advanced Audio Coding)
profile=LC
codec_type=audio
codec_time_base=1/44100
codec_tag_string=mp4a
codec_tag=0x6134706d
sample_fmt=fltp
sample_rate=44100
channels=1
channel_layout=mono
bits_per_sample=0
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/44100
start_pts=-1024
start_time=-0.023220
duration_ts=1324024
duration=30.023220
bit_rate=56517
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=1293
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:language=und
TAG:handler_name=SoundHandler
[/STREAM]
[FORMAT]
filename=input.mp4
nb_streams=2
nb_programs=0
format_name=mov,mp4,m4a,3gp,3g2,mj2
format_long_name=QuickTime / MOV
start_time=-0.023220
duration=30.024000
size=368644
bit_rate=98226
probe_score=100
TAG:major_brand=isom
TAG:minor_version=512
TAG:compatible_brands=isomiso2avc1mp41
TAG:title=FFprobe Tips
TAG:encoder=Lavf56.15.101
[/FORMAT]

• From the above example, if you would only want duration=30.024000 .

$ ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 input.mp4
Output: size=368644

• If you would only want the value with no key.

$ ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 input.mp4
Output: 368644

• Getting width and height (resolution) of video.

$ ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 input.mp4
Output: 1280x720

FFplay Usage and Examples

Usage

ffplay [OPTIONS] [INPUT_FILE]

• man ffplay — Refer to the manual.
• ffplay -h — Shows included help.
• ffplay <INPUT_FILE> — Play the video on full screen.
• -x <window width> -y <windows height> — Play the video with specific height and width popup windows size in the centre. It will automatically fill the extra space between the video area and the window area if the aspect ratio of the popup window and the video are different.
• -noborder — Hide title bar while playing video in popup windows.
• -[top/left/bottom/right] <value> — Make popup window in specific coordinate. By default, it will pop up in the centre of the screen.

You can find out more FFplay usage here.

Basic Examples

• Play a video file named video.mp4 in fullscreen.

$ ffplay video.mp4

• Play a video file in a popup window with 600 widths, 600 height, and no title bar.

$ ffplay video.mp4 -x 600 -y 600 -noborder

• Play a video file in a popup window in the top left corner.

$ ffplay video.mp4 -x 600 -y 600 -noborder -top 0 -left 0

Conclusion

We had covered all the basics in FFmpeg. Feel free to read their official documentation here if you want to have a deeper understanding!

References

• https://ffmpeg.org/documentation.html
• https://trac.ffmpeg.org/wiki/FFprobeTips
• https://ffmpeg.org/ffplay-all.html
• https://ffmpeg.org/ffprobe.html
• FFmpeg Complete Guide by Syed Andaleeb Roomy

About Author:

This article is written by Han Sheng, Technical Lead in Arkmind, Malaysia. He has a passion for Software Design/Architecture related stuff, Computer Vision and also Edge Devices. He made several AI-based Web/Mobile Applications to help clients solving real-world problems. Feel free to read about him via his Github profile.