Android, CMake and FFmpeg. Part Two: Building FFmpeg
Originally published here.
Series Contents
- Part One: CMake in Android Cookbook
- Part Two: Building FFmpeg [you are here]
- Part Three: Throwing libx264 in the Mix
Building FFmpeg
Preface
This article expands on my previous works and improves them in a few areas (specifically performance-wise, because the previous article used –disable-asm). CMake workflow makes FFmpeg integration much smoother and easier to handle since you’re working in one place, instead of constantly jumping between terminal windows as with ndk-build.
While I hope the material in this article is self-sufficient, so you won’t need to read previous articles on FFmpeg, I think it still might be worth it to glance them over just to refresh some basic vocabulary.
Getting started
Before we start let’s make sure we all on the same page here:
- Make sure you have NDK installed. The version used in this article is 21.1.6352462.
- Make sure you have CMake installed. The version used in this article is 3.10.2.
- Make sure you have your C++ enabled project ready.
That’s pretty much all you need to get started.
Just give me the code
In case you’re not interested in the material in the article and would rather play with code, you can check out according to the article version of FFmpeg Development Kit.
Setting up for the build process
Navigate to your CMakeLists.txt file and add following code here:
link_directories(${CMAKE_LIBRARY_OUTPUT_DIRECTORY})
include_directories(${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/include)
We’re going to build libraries into ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}
and headers into ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/include
. Here we make sure that all targets will be able to find our libraries and headers later on.
Next, make sure you have included ExternalProject like so:
include(ExternalProject)
Now, let’s define all the libraries we will use:
set(FFMPEG_LIBS avutil swresample avcodec avformat swscale avfilter avdevice)
We define it as a separate variable both for readability sake and ability to change it dynamically later on, if necessary.
Also, let’s add an additional QoL variable which will allow us to pass additional configure flags from the main script to the configure process like this:
set(FFMPEG_CONFIGURE_EXTRAS )
You can add a list of additional flags here as a list.
This is pretty much all for the setting up process in the main script. Let’s move all the work related to the actual FFmpeg building to a separate script to make it cleaner and easier to maintain.
ffmpeg.cmake
Preparing the sources
Create a file in the same folder your CMakeLists.txt resides and call it “ffmpeg.cmake”. This is going to be the module that will build FFmpeg for us.
Let’s start from the start i.e. actually getting sources to build. First, we will define a few support variables to make our life easier:
set(FFMPEG_VERSION 4.2.2)
set(FFMPEG_NAME ffmpeg-${FFMPEG_VERSION})
set(FFMPEG_URL https://ffmpeg.org/releases/${FFMPEG_NAME}.tar.bz2)
You can change FFMPEG_VERSION to whatever version you want, but be aware that it might not work.
Next, let’s extract archive name from the URL:
get_filename_component(FFMPEG_ARCHIVE_NAME ${FFMPEG_URL} NAME)
You might wonder why won’t we simply set it like that:
set(FFMPEG_ARCHIVE_NAME ffmpeg-${FFMPEG_VERSION}.tar.bz2)
set(FFMPEG_URL https://ffmpeg.org/releases/${FFMPEG_ARCHIVE_NAME})
My reasoning is that I want to allow URL to change without breaking too much logic, however you’re free to take any approach you want.
Now let’s do actual fetching. First we check if we already have downloaded the sources:
IF (NOT EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME})
Note that we use CMAKE_CURRENT_SOURCE_DIR to make sure we have an absolute path to the folder. The reason is that your scripts are re-configured multiple times in the “cxx” folder for different flavors and architectures. To avoid excess work and messing up it’s better to always use absolute paths, because relative paths will be evaluated relative to cache directory (inside “cxx”).
Now we want to download archive:
file(DOWNLOAD ${FFMPEG_URL} ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_ARCHIVE_NAME})
But we can’t use the archive — we need sources inside it. Let’s unpack it:
execute_process(
COMMAND ${CMAKE_COMMAND} -E tar xzf ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_ARCHIVE_NAME}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
Note that we use ${CMAKE_COMMAND} purely for cross-platform reasons. If you prefer another unpacking method and your host will always be the same — you can change COMMAND to any other extraction command you want.
Well, so far so good, but not entirely. We need to patch sources a little bit to make them usable on Android. You see, “main” inside ffmpeg.c has “exit” invocation just before return. In shell programs, it’s totally fine (as well as calling abort). The problem is that on Android it’s treated as a call to close the entire app. Which means your app will just silently close on the user as soon as processing ends. Talking about great UX huh.
So let’s remove that:
file(READ ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg.c ffmpeg_src)string(REPLACE "exit_program(received_nb_signals ? 255 : main_return_code);" "//exit_program(received_nb_signals ? 255 : main_return_code);" ffmpeg_src "${ffmpeg_src}")
string(REPLACE "return main_return_code;" "return received_nb_signals ? 255 : main_return_code;" ffmpeg_src "${ffmpeg_src}")file(WRITE ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg.c "${ffmpeg_src}")
As you can see we simply comment out the exit call and move proper return code to the return.
Well after that we finished with the fetching process, let’s close IF with ENDIF().
Now let’s add the build system script to the fetched folder. Note that we won’t create it just yet, it will just be a CMake wrapper around configuration and build process.
file(
COPY ${CMAKE_CURRENT_SOURCE_DIR}/ffmpeg_build_system.cmake
DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
)
Don’t worry about the script, for now, we’re going to add it later on.
Configuring build tools
This section is rather simple — we just want to configure specific build tools (C compiler, ASM compiler etc.) for the FFmpeg. Let’s start with tools that already defined in the toolchain:
set(FFMPEG_CC ${CMAKE_C_COMPILER})
set(FFMPEG_CXX ${CMAKE_CXX_COMPILER})
set(FFMPEG_AR ${ANDROID_AR})
set(FFMPEG_AS ${ANDROID_ASM_COMPILER})
We essentially make copies of those variables so if we ever would want to change them (for example, setting different C compiler for specific ABI) — we could do that without breaking stuff in other places of the build process (remember that variables exposed by toolchain are global and it’s probably not a good idea to modify them).
Now let’s move to stuff that toolchain doesn’t expose yet:
set(FFMPEG_RANLIB ${ANDROID_TOOLCHAIN_PREFIX}ranlib${ANDROID_TOOLCHAIN_SUFFIX})
set(FFMPEG_STRIP ${ANDROID_TOOLCHAIN_ROOT}/bin/llvm-strip${ANDROID_TOOLCHAIN_SUFFIX})
set(FFMPEG_NM ${ANDROID_TOOLCHAIN_PREFIX}nm${ANDROID_TOOLCHAIN_SUFFIX})
Note that this set of build tools is specific to FFmpeg. Your particular case might not need ranlib or nm, for instance, so make sure to check beforehand what you need to provide to don’t have useless declarations.
Setting the flags
The first thing we want to do is remove “-Wl,–fatal-warnings” from linker flags provided by the toolchain. While this flag might be ok for your library it most definitely prevents FFmpeg from building. We can do it like this:
string(REPLACE " -Wl,--fatal-warnings" "" FFMPEG_LD_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
Nothing fancy here. Since toolchain doesn’t provide a way to turn it off — we just replace this part of the string with an empty string.
Now let’s proceed to setting flags for the FFmpeg:
set(FFMPEG_C_FLAGS "${CMAKE_C_FLAGS} --target=${ANDROID_LLVM_TRIPLE} --gcc-toolchain=${ANDROID_TOOLCHAIN_ROOT} ${FFMPEG_EXTRA_C_FLAGS}")
set(FFMPEG_ASM_FLAGS "${CMAKE_ASM_FLAGS} --target=${ANDROID_LLVM_TRIPLE} ${FFMPEG_EXTRA_ASM_FLAGS}")
set(FFMPEG_LD_FLAGS "${FFMPEG_C_FLAGS} ${FFMPEG_LD_FLAGS} ${FFMPEG_EXTRA_LD_FLAGS}")
Again, we duplicate flags so we avoid modifying flags set by toolchain and accidentally messing up other targets. Note that we have additional variables for every flag to provide additional flags in case we need to (FFMPEG_EXTRA_*_FLAGS). Also, note that LD flags also include C flags. I figured it out from the build.ninja, it seems to be necessary.
Setting additional variables
Let’s define a few additional variables that will help us control the build process:
set(NJOBS 4)
set(HOST_BIN ${ANDROID_NDK}/prebuilt/${ANDROID_HOST_TAG}/bin)
The NJOBS
variable will control how many jobs Make is going to use (or how much of your CPU capacity it’s going to use, in other terms).
HOST_BIN
has a path to the host binaries inside the NDK. The reason we need this path is that we want to use Make provided by NDK, not the actual host one (assuming the host even has it installed, which is not guaranteed).
Patching up x86
FFmpeg has text relocations by design on x86, while Android from Marshmallow and up explicitly forbids having text relocations in native libraries. Since x86 is a really limited platform (you probably only would see it in the emulator and I recommend using x86_64 images instead of x86 anyway) so we just disable assembly on the x86 ABI.
IF (${CMAKE_ANDROID_ARCH_ABI} STREQUAL x86)
list(APPEND FFMPEG_CONFIGURE_EXTRAS --disable-asm)
ENDIF()
Encoding extras
We will have to pass extras to the script, however, there is a problem here. By default CMake uses “;” as a separator for both lists (which are strings under the hood) and for separating arguments for the CMake invocations. Which means if we just pass a list as the argument for the CMake script we will end up with all flags being treated as separate arguments.
So in order to avoid that we will just encode list with different separator:
string(REPLACE ";" "|" FFMPEG_CONFIGURE_EXTRAS_ENCODED "${FFMPEG_CONFIGURE_EXTRAS}")
Note that I used “|” as a separator since it’s unlikely to appear in the flags, however, if it conflicts with your use case — feel free to change it to any other symbol.
Configuring ExternalProject
Now we have everything we need to actually configure FFmpeg for building. There will be a lot of code, however much of it is explained in part one of this series. I’ll concentrate only on specific to the FFmpeg parts:
ExternalProject_Add(ffmpeg_target
PREFIX ffmpeg_pref
URL ${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}
DOWNLOAD_NO_EXTRACT 1
CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env
PATH=${ANDROID_TOOLCHAIN_ROOT}/bin:$ENV{PATH}
AS_FLAGS=${FFMPEG_ASM_FLAGS}
${CMAKE_COMMAND}
-DSTEP:STRING=configure
-DARCH:STRING=${CMAKE_SYSTEM_PROCESSOR}
-DCC:STRING=${FFMPEG_CC}
-DSTRIP:STRING=${FFMPEG_STRIP}
-DAR:STRING=${FFMPEG_AR}
-DAS:STRING=${FFMPEG_AS}
-DNM:STRING=${FFMPEG_NM}
-DRANLIB:STRING=${FFMPEG_RANLIB}
-DSYSROOT:STRING=${CMAKE_SYSROOT}
-DC_FLAGS:STRING=${FFMPEG_C_FLAGS}
-DLD_FLAGS:STRING=${FFMPEG_LD_FLAGS}
-DPREFIX:STRING=${CMAKE_LIBRARY_OUTPUT_DIRECTORY}
-DCONFIGURE_EXTRAS:STRING=${FFMPEG_CONFIGURE_EXTRAS_ENCODED}
-P ffmpeg_build_system.cmake
BUILD_COMMAND ${CMAKE_COMMAND} -E env
PATH=${ANDROID_TOOLCHAIN_ROOT}/bin:$ENV{PATH}
${CMAKE_COMMAND}
-DSTEP:STRING=build
-NJOBS:STRING=${NJOBS}
-DHOST_TOOLCHAIN:STRING=${HOST_BIN}
-P ffmpeg_build_system.cmake
BUILD_IN_SOURCE 1
INSTALL_COMMAND ${CMAKE_COMMAND} -E env
PATH=${ANDROID_TOOLCHAIN_ROOT}/bin:$ENV{PATH}
${CMAKE_COMMAND}
-DSTEP:STRING=install
-DHOST_TOOLCHAIN:STRING=${HOST_BIN}
-P ffmpeg_build_system.cmake
STEP_TARGETS copy_headers
LOG_CONFIGURE 1
LOG_BUILD 1
LOG_INSTALL 1
)
In CONFIGURE_COMMAND
we invoke ffmpeg_build_system.cmake with STEP “configure” and all parameters required for the configuration process. LIBS_OUT
and HEADERS_OUT
parameters allow us to specify where libs and headers will be installed and TARGET_TOOLCHAIN
is needed for “yasm” to be found (we add it to PATH).
In BUILD_COMMAND
we supply STEP
equal to “build” to indicate what we are doing, NJOBS
to control how much resources Make can use and HOST_TOOLCHAIN
to have NDK’s Make available. TARGET_TOOLCHAIN
is needed to make sure we have yasm available if needed.
In INSTALL_COMMAND
we supply STEP
equal to “install” to indicate what we are doing, TARGET_TOOLCHAIN
and HOST_TOOLCHAIN
are for the same reasons as in the build step.
In STEP_TARGETS
we define an additional step that will take place before the “install” step and it will copy all headers from the FFmpeg’s folder to the headers installation folder. The reason is that since we want to use CLI tools, which use some internal stuff, we have to copy headers in order to make them work.
LOG_* properties just set to make sure if something fails we can diagnose what happened.
Adding step for headers’ copying
As mentioned above we need this step to make CLI tools work inside our library. Let’s start with creating a script that will do the actual copying.
copy_headers.cmake
In the same folder where your CMakeLists.txt resides create a file called “copy_headers.cmake”. This script will be very simple, just a few commands:
cmake_minimum_required(VERSION 3.10.2) file(GLOB libs "${SOURCE_DIR}/${FFMPEG_NAME}/lib*")
file(
COPY ${libs} ${BUILD_DIR}/config.h ${SOURCE_DIR}/${FFMPEG_NAME}/compat
DESTINATION ${OUT}/include
FILES_MATCHING PATTERN *.h
)
Few notes. Firstly, we use ${SOURCE_DIR}
which is passed as a parameter, not ${CMAKE_CURRENT_SOURCE_DIR}
. The reason is that if we use CMAKE_CURRENT_SOURCE_DIR
it will point to the cache directory since the script will be executed at build time, not at configure time. Secondly, we use GLOB to get a list of all libraries folders in the FFmpeg’s sources folder and then expand this list in the copy invocation.
We provide two additional variables: BUILD_DIR
, which points to the FFmpeg’s build folder with proper “config.h”, and OUT
which points to the directory where libs will be installed (essentially, it is CMAKE_LIBRARY_OUTPUT_DIRECTORY
).
Also, it’s worth noting we copy only headers (*.h files), since we don’t need source files there.
Executing script as custom step
Now let’s add actual custom step back in the “ffmpeg.cmake” file to run the script:
ExternalProject_Get_property(ffmpeg_target SOURCE_DIR)
ExternalProject_Add_Step(
ffmpeg_target
copy_headers
COMMAND ${CMAKE_COMMAND}
-DBUILD_DIR:STRING=${SOURCE_DIR}
-DSOURCE_DIR:STRING=${CMAKE_CURRENT_SOURCE_DIR}
-DFFMPEG_NAME:STRING=${FFMPEG_NAME}
-DOUT:STRING=${CMAKE_LIBRARY_OUTPUT_DIRECTORY}
-P ${CMAKE_CURRENT_SOURCE_DIR}/copy_headers.cmake
DEPENDEES build
DEPENDERS install
)
First of all, we’re getting the build directory with ExternalProject_Get_property
and putting it in the property called SOURCE_DIR
(note that since we build in source and build directories are the same). Then we add the actual implementation (definition is in the ExternalProject_Add
) of the step with script invocation. The last piece of the puzzle is that we set it to run between “build” and “install” steps explicitly by defining DEPENDEES
and DEPENDERS
properties.
This finalizes the build process of FFmpeg itself, leaving us to do the integration part.
Defining sources for the CLI tools
We define the list of sources to use later on in the main script:
set(ffmpeg_src
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/cmdutils.c
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg_cuvid.c
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg_filter.c
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg_hw.c
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg_opt.c
${CMAKE_CURRENT_SOURCE_DIR}/${FFMPEG_NAME}/fftools/ffmpeg.c
)
Nothing really to explain here, we just define a path to sources for later use.
ffmpeg_build_system.cmake
There are a few reasons I decided to use a separate script. The first reason is that I had an insane amount of trouble with getting just plain “./configure” to run. Shell wasn’t able to run the script, throwing out “command not found”, parameters weren’t supplied correctly etc. Using CMake script seems to be a better and more stable option. The second reason is that there is some additional logic we might want to have in the script, like exporting variables, modifying flags and so on.
The script is actually very simple:
cmake_minimum_required(VERSION 3.10.2)if (${STEP} STREQUAL configure)
# Encoding string to list
string(REPLACE "|" ";" CONFIGURE_EXTRAS_ENCODED "${CONFIGURE_EXTRAS}")
list(REMOVE_ITEM CONFIGURE_EXTRAS_ENCODED "")# Note that we don't pass LD, Clang sets it internally based of --target
set(CONFIGURE_COMMAND
./configure
--cc=${CC}
--ar=${AR}
--strip=${STRIP}
--ranlib=${RANLIB}
--as=${AS}
--nm=${NM}
--target-os=android
--arch=${ARCH}
--extra-cflags=${C_FLAGS}
--extra-ldflags=${LD_FLAGS}
--sysroot=${SYSROOT}
--enable-cross-compile
--disable-static
--disable-programs
--disable-doc
--enable-shared
--enable-protocol=file
--enable-pic
--shlibdir=${PREFIX}
--prefix=${PREFIX}
${CONFIGURE_EXTRAS_ENCODED}
) execute_process(COMMAND ${CONFIGURE_COMMAND})
elseif(${STEP} STREQUAL build)
execute_process(COMMAND ${HOST_TOOLCHAIN}/make -j${NJOBS})
elseif(${STEP} STREQUAL install)
execute_process(COMMAND ${HOST_TOOLCHAIN}/make install)
endif()
Build and install steps just run “make” and “make install”. Configure step is a little bit more complicated, but also not extremely hard. Essentially here we decode extras back to the list, then we append those to “configure” invocation.
Back to the CMakeLists.txt
Now we actually can go to the main script and link everything we’ve done so far in “ffmpeg.cmake” to our actual library.
First of all, include “ffmpeg.cmake”:
include(ffmpeg.cmake)
Then modify your add_library
call to include ffmpeg_src
, like so:
add_library(native-lib SHARED native-lib.c ${ffmpeg_src})
Next, we need to make sure FFmpeg actually is going to be built. By default, CMake in Android will build only the main target. In order to make sure FFmpeg is built we need to specify FFmpeg’s target as a dependency to your main target like so:
add_dependencies(native-lib ffmpeg_target)
And the last, but not least, we have to actually link all built libraries against our main library:
target_link_libraries(native-lib ${FFMPEG_LIBS})
That will allow us to use libraries in our library.
Using FFmpeg in the library
This is pretty much the same as it was in previous articles. Let’s define a function that will call FFmpeg’s main function:
int main(int argc, char **argv);jint run_ffmpeg(
JNIEnv *env,
jobjectArray args
) {
int i = 0;
int argc = 0;
char **argv = NULL;
jstring *strr = NULL; if (args != NULL) {
argc = (*env)->GetArrayLength(env, args);
argv = (char **) malloc(sizeof(char *) * argc);
strr = (jstring *) malloc(sizeof(jstring) * argc); for (i = 0; i < argc; ++i) {
strr[i] = (jstring)(*env)->GetObjectArrayElement(env, args, i);
argv[i] = (char *)(*env)->GetStringUTFChars(env, strr[i], 0);
}
} jint retcode = 0;
retcode = main(argc, argv); for (i = 0; i < argc; ++i) {
(*env)->ReleaseStringUTFChars(env, strr[i], argv[i]);
} free(argv);
free(strr); return retcode;
}
Note that we have a forward declaration of “main” function before ours to make code compile properly. You can call the “run_ffmpeg” function inside your JNI function, just make sure to pass the correct list of arguments.
Don’t forget to load libraries at runtime before you call your JNI function like this:
System.loadLibrary("avutil")
System.loadLibrary("swresample")
System.loadLibrary("avcodec")
System.loadLibrary("avformat")
System.loadLibrary("swscale")
System.loadLibrary("avfilter")
System.loadLibrary("avdevice")
That’s pretty much it, now you should be able to build and run FFmpeg (hopefully).
Bonus content
This section describes a few additional things that are not exactly needed in the main article, but still are super useful.
Getting logs from native libs
The credit for this goes to this SO answer.
Essentially the problem is that somebody in the early days of Android decided that nobody would ever need stdout and stderr streams and redirected them to /dev/null. The recommended approach is to use Android’s logging library, which is not feasible in case of big third-party libraries (and pretty much impossible if the library is not open-sourced). So what can we do?
The only working option I was able to find is to redirect streams to the Logcat like this:
#include <android/log.h>static int pfd[2];
static pthread_t thr;static void* thread_func(void* in) {
ssize_t rdsz;
char buf[128];
while((rdsz = read(pfd[0], buf, sizeof buf - 1)) > 0) {
if(buf[rdsz - 1] == '\n') --rdsz;
buf[rdsz] = 0; /* add null-terminator */
__android_log_write(ANDROID_LOG_DEBUG, tag, buf);
}
return 0;
}jint JNI_OnLoad(JavaVM* vm, void* reserved){
setvbuf(stdout, 0, _IOLBF, 0);
setvbuf(stderr, 0, _IONBF, 0); pipe(pfd);
dup2(pfd[1], 1);
dup2(pfd[1], 2); if(pthread_create(&thr, 0, thread_func, 0) == -1) return JNI_VERSION_1_6; pthread_detach(thr); return JNI_VERSION_1_6;
}
First, we assign buffers to both stdout and stderr with setvbuf calls. Then we create a pipe and assign both stdout and stderr to the write end of the pipe. Then we start a thread to pool read end of the pipe and write everything it gets to the Logcat.
That’s pretty much it, now you should see logs from native lib in your Logcat. Don’t forget to link the log library to your native library for this to work!
Enabling MediaCodec support in FFmpeg
FFmpeg has supported MediaCodec for a while now, however, due to the nature of this particular library, it might not be suited for every format of the video. Usually, MediaCodec works well only with videos recorded on the device, so I would suggest either re-encoding video received from the internet to make sure it matches expected profile (like h264) or don’t use it at all since it might cause your app to receive native crash (because FFmpeg might call abort).
Anyway, if you want to try it out here’s what you have to do. Make sure FFmpeg is configured with the following flags “target-os=android”, “–enable-jni” and “–enable-mediacodec”. Then add the following call to your native lib JNI_OnLoad:
av_jni_set_java_vm(vm, NULL);
And make sure to include:
#include <libavcodec/jni.h>
That’s pretty much it, you now should be able to select MediaCoded as the codec for encoding/decoding in the FFmpeg call. But as I said be careful, it might crash the app!