Audio mixing on Android
In this article I would like to talk about audio mixing on Android. Everything started when I faced with task of mixing audio from video and audio files by using android.media library which includes:
- MediaExtractor is used for extract meta information and media samples from audio/video files.
- MediaCodec is needed for decoding encoding media samples.
- MediaMuxer packs audio and video samples to video file — MP4 in our case.
If you never work with android media library, better to read documentation of them before continue to read this topic.
So let’s define task which we are going to solve in this article:
We have one MP4 and one MP3 files and as result we would like to have one MP4 file where audio from first and second file are mixed, duration of output MP4 should be equal duration of input MP4.
Firstly we will declare simple method which we are going to implement. We are writing code on Kotlin, let’s begin:
Main idea of audio mixing
Before doing this implementation of course I did research, and no one simple and light library I could find. Most useful article which I managed to find was Mobile Engineering — thoughts and tips on iOS and Android development: Audio mix and record in Android. Those article tells us most important things of audio mixing: each mixed audio sample is average of audio sample 1 and audio sample 2 -remark: this method only guarantee right result for audios with similar sample rate so let’s assume that our input files have similar sample rate and continue.
Now we know that for mixing two audios (with similar sample rates) we have to:
1) Extract raw audio samples from both input files: MP4 and MP3
2) Find average of input audio samples
3) Write resulted audio sample to output MP4 file.
Extracting raw audio from MP4.
For extracting raw audio from MP4 we need to create instance of MediaExtractor for extracting sample by sample of audio/video (video is not going to change so we will write it in output MP4 as it is) and instance of MediaCodec which will encode audio sample — makes it raw.
Process of extracting raw audio sample from file actually takes a two steps:
- Extracting audio sample from file by MediaExtractor
- Add audio sample to decoder input buffer queue
- Getting from decoder output buffer queue decoded/raw audio sample.
Code example of extracting raw audio samples:
Extracting raw audio from MP3.
The process of extracting decoded samples from Mp3 file is the same as for Mp4. Only one difference that MP3 doesn’t have video samples so we don’t need to care about it. Because of that we will use the same class for extracting audio raw data from Mp3 and Mp4 files and we will call it `AudioSampleExtractor`.
Here is the instance creation of `AudioSampleExtractor` for Mp3 file:
val mp3AudioSampleExtractor = AudioSampleExtractor(
mediaExtractor = mp3Extractor,
trackIndex = getTrackIndex(mp3Extractor, AUDIO),
syncsPresentationTime = false
Mixing audios from MP4 and MP3.
We have raw audio data from MP4 and MP3 and we are ready to mix them. As a main timeline we are going to use time from MP4 file.
Audio mixing function implementation:
Write mixed audio to output MP4.
Mixed audio sample is ready and we are ready to write it to the output MP4 file. But before writing to the MP4 container we need to encode it to AAC format. For encoding audio sample we are going to use instance of MediaCodec. For writing encoded audio sample to the MP4 file we need instance of MediaMuxer.
Code example of encoding raw audio sample data and writing it to the output Mp4 file.
In this article we made the acquaintance of Android media library and write simple algorithm for mixing audios of MP4 and MP3 files with similar sample rate.
If you know how to mix audio for different sample rates please share your knowledge with me and community, I will be really appreciate that, I tried to implement upsample filter but without success.
Current implementation easy to adapt for mixing N number of audio tracks and add timing offsets if needed.
If you like to hear more about audio, video, gif rendering on Android please let me know and I can write other articles for community.
Thank you for reading.
Github repo with app sample here.