Robust, continuous audio recording:

The reconnect command in FFmpeg

At Intrasonics we’ve been using FFmpeg and FFprobe as reliable tools to stream audio. These open-source libraries are great; however the sheer extent and complexity of the library means that some features are not well documented (This is not a slight against the FFmpeg community, it is simply that the task is pretty immense. Seriously look at the number of options available. This is without getting into codecs and audio/video formats and…).

This article is about the reconnect flag in FFmpeg. We’ve been using these commands in our operations involving continuous audio streaming and so spent a fair bit of time puzzling things out. Some of this was found out by some experimentation and some by puzzling out the behaviour of the source code. Many thanks to fellow devs Abbie and Alejandro for their help investigating, and Jose for proof reading and commentary.

This flag does as you might expect, it attempts to reconnect a lost connection — valuable when say constantly streaming audio/video from an online source and there’s a temporary network issue. While it sounds simple on paper, there’s little documentation about it and the answers that do exist on places like stack overflow are quite vague. It is simple to state `use -reconnect 1 -reconnect_streamed 1` to attempt a reconnect but if you want the precise details of what will happen, you’ve come to the right place.

A word about versions

Firstly, this is only available in FFmpeg 4.4.1 and above. If you’re using Ubuntu 16 or above, this won’t be available from the standard repos and you’ll need to add Rob Savoury’s[1] repos (ppa:savoury1/ffmpeg4). All comments that I’ve made work the same in the recently released FFmpeg 5.

What does reconnect actually do?

This seems like a silly thing to say, as the name seems self-explanatory but for the purposes of precision, the flag will make FFmpeg reconnect to a stream of data and attempt to append the packets upon successful reconnect directly to where it stopped. What this means is that, if for example you have a live broadcast that you’re capturing with FFmpeg, FFmpeg disconnects and then successfully reconnects after 7 seconds, you will likely notice a seven second jump in the output. This is important to remember.

Another point is that reconnection does not take away from the overall time of output. So, if you define FFmpeg to record a stream to produce a maximum of 20 minutes of data and a reconnect happens (or several reconnects even), you will in fact get 20 minutes of data. Just keep in mind, this might be 20 minutes with jumps in them. This means that if you allow your `reconnect_delay_max` value to be quite large, you can potentially end up with very large jumps in your end product.

How to use reconnect

FFmpeg’s documentation doesn’t seem to detail anything about the `-reconnect` flag, but lists these details about the other possible flags under the HTTP section of their protocol documentation.

reconnect_at_eof

If set then eof is treated like an error and causes reconnection, this is useful for live / endless streams.

reconnect_streamed

If set then even streamed/non seekable streams will be reconnected on errors.

reconnect_on_network_error

Reconnect automatically in case of TCP/TLS errors during connect.

reconnect_on_http_error

A comma separated list of HTTP status codes to reconnect on. The list can include specific status codes (e.g. ’503’) or the strings ’4xx’ / ’5xx’.

reconnect_delay_max

Sets the maximum delay in seconds after which to give up reconnecting

However to get any of these to work, you’re required to include the `-reconnect 1` flag. Note that simply using the reconnect flag on its own won’t do anything.

Additionally the positioning of these flags is important — they must go before the input flags (-i, -re).

An example

If you’re keen to try this, do the following:

ffmpeg -reconnect 1 -reconnect_on_network_error 1 -i http://badurl.bad ./output.mp3

And you’ll get the following:

As you can see it attempts to do staggered reconnections, eventually giving up. Note that this pattern of text is slightly misleading — it does not attempt to reconnect 63 seconds into the operation and then stop, but in fact 2 minutes in. Each `will reconnect` message is indicating how long FFmpeg will wait to try again. So it tries right away, then waits 1 second. Then after that another 3 seconds. Then after that another 7 seconds… and so on. I’ll get into the details of the precise pattern going on here and how this relates to `reconnect_delay_max` later (see the section on reconnection attempts if you just want that).

Definitions in more detail

To some degree these definitions are self-explanatory, although there are few points to note. Firstly `reconnect_on_network_error` and `reconnect_on_http_error` are not the same, as you might casually suppose. If the error in question is a HTTP Error, this won’t be counted under the `reconnect_on_network_error` flag.

This means that for instance `-reconnect 1 -reconnect_on_network_error 1` will exit out if for some reason it receives a 500 error from the server. To catch that you’d need `reconnect_on_http_error 500` or more generally `reconnect_on_http_error 5xx` which will deal with any 5XX error. If you do use these options note that the xx is case sensitive `-reconnect_on_http_error 4xx,5xx`, will work, but `-reconnect_on_http_error 4XX,5XX` will not.

Moreover, these reconnection commands only apply to the initialisation of the recording. If you want to protect yourself from interruptions partway through a stream, you’ll need at least `reconnect_streamed` and quite likely `reconnect_at_eof` (this is from experience, as some audio streams have EOF problems).

A caveat about reconnect_at_eof

One small thing about `reconnect_at_eof`. It does not work with .mpd streams. The mpeg-dash format requires you to first download a playlist file which then coordinates the audio streams. What happens then is that FFmpeg stumbles over this first step — once it is finished downloading the playlist file an End Of File error is raised, which causes a reconnect, which causes the playlist file to be redownloaded, which then raises an EOF error… As it is, the other part of the reconnect command works fine with .mpd streams, so your best advice is to leave it out.

How do the reconnection attempts work?

Lastly, we have `reconnect_delay_max`. How does this work? How does it relate pattern of output?

So firstly, the reconnect_delay_max is not a timeout setting for reconnections — that is if you set it to 150 seconds it will not wait to 150 seconds, making reconnections with that pattern detailed above and then disconnect at that point. What it in fact will do is wait to 247 seconds then disconnect. Why is that? Because 247 seconds is the cumulative delay of multiple attempts to reconnect, each taking longer and longer to happen, with 127 being the longest delay between attempts (after that it will stop trying and fail).

FFmpeg has a predefined pattern of delays between attempts. Using 150 in this case will make FFmpeg take as its maximum delay the value in its pattern closest beneath that value. In this case, that’s 127.

This sequence for the delays is determined by 2^n-1 from n >= 0. So we have a pattern that goes from 0, 1, 3, 7, 15, 31, 63, 127,… You can use this formula to work out what position in the sequence you’ll get from a given n. You can then use the summation formula 2^(n + 1)-n-2 to find out how much total clock time reconnections will take.

Note that this means there’s no value in setting reconnect_delay_max value any higher then the intended value in the sequence. 8, will always terminate after it reaches the 7 second delay.

About Intrasonics

Intrasonics is the leading global provider of audio recognition technology. Our audio watermarking and audio matching technology is widely used in audience research, content protection, synchronised apps and interactive toys. We are market leaders in providing the tools required for audio recognition and synchronised content response triggers.

Our technology is platform independent, inaudible, noise tolerant and fully protected by more than 20 patents. Integration possibilities include development kits and software/hardware encoders.

[1] Rob Savoury has been doing some excellent work providing software for Ubuntu releases. He’s currently looking for funding at https://www.paypal.com/paypalme/Savoury1 (alt links at https://launchpad.net/~savoury1)