Java TS Video Downloader

Did you come across a website hosting a video you wanted to download, but couldn’t because the video was formatted as a Transport Stream (TS) file?

Unlike MP4 or MOV multimedia formats, most TS file formats DO NOT incorporate a full video in a single file; rather, each individual TS file represents a fragment of a full video. In this article, I will describe in detail how to successfully download and assemble TS video files to produce one single, full video.

But if you’ll rather not go into detail, here is the direct GitHub project source code:

Jet-C/java-video-downloader-util

TS Files and Where To Find Them

MPEG Transport Stream (TS) is a type of formatted data file used for transmission and storage of audio and video. Most internet streaming and digital broadcasting is done via TS since it grants receiving clients the feasibility to read data right at the beginning of a transmission. Receiving clients are not compelled to wait for a full data file to download because fractional .ts files are continuously sent across the network.

Transmissions are built from multiple .ts file fragments. To piece together a full video, the individual pieces must be downloaded first and then assembled. The video’s .ts file fragments can be obtained by fetching the HTTP requests registered in the network logs.

Using the network logs, we can observe the website’s multiple HTTP requests for .ts files

How does your browser know how to appropriately assemble the numerous files requested? This is where M3U8 files come into play. Beside the .ts files’ HTTP requests, one or more requests for .m3u8 files should appear in the network logs.

What are MP3 URL (M3U8) and Master M3U8 files?

M3U8 files are simple, plain-text files containing metadata for .ts file fragments or .m3u8 files. The metadata includes several data directive tags and the mapping of the file locations. File locations are expressed as a full and relative path URLs or system directories.

A master M3U8 file specifically lists the locations of available .m3u8 file variants. These variants represent different video quality bit rates and map the .ts fragments associated with a particular rate.

Having different bit rates enables clients, like a browser, to select the rate yielding less buffering. Depending on your network connection, the browser can actively switch from a lower bit rate to a higher one and vise versa. For the most part, a higher rate results in a better quality video.

Note: M3U8 are M3U files that explicitly state UTF-8-character encoding, but M3U files may use other character encoding standards. Data described in M3U files may involve many types of multimedia, not just video.

The following master M3U8 file sample, lists the full path location URLs of available .m3u8 file variants. The different bit rates can be identified by the “BANDWIDTH” tags.

Master M3U8 files are distinguishable via their “EXT-X-STREAM-INF” directive tags

In the sample .m3u8 file variant below, the .ts fragment URLs are listed in the sequential order required for correct video assembly.

*.ts fragments are listed in the sequential order required for assembly

Click here and/or check out this wiki page, to access in-depth documentation of master playlists and M3U directive tag descriptions.

Video Downloader Utility

Let’s dive into how the downloader utility works.

We’ll be using the FFmpeg command line program to assemble all the .ts file fragments. The program must be installed on your machine prior to running the utility.

In order to capture the .ts file fragments we’ll use Selenium, a browser automation tool and the network utility, BrowserMob Proxy (BMP). Their dependencies are included within the project’s pom.xml file.

Project “java-video-downloader-util” Maven dependencies

Steps

1. Using Selenium’s Chrome WebDriver, automatically start a Chrome browser instance and direct it to the website hosting the TS video.
2. The BrowserMob Proxy intercepts all the network requests from the website and logs them to an HTTP archive (HAR) file.
3. Filter the HAR file for .m3u8 file requests and identify the master M3U8 file via its “EXT-X-STREAM-INF” tags.
4. From the master M3U8 file, retrieve the URL associated with the highest bit rate .m3u8 variant. Bit rates are denoted via the “BANDWIDTH” directive tags.
5. Download each individual .ts file fragment using the URLs listed in the .m3u8 file variant. The fragments’ bit rates will match the variant’s rate.
6. Once all .ts file fragments are downloaded, prepare them as command line arguments and pass them over to the FFmpeg program.
7. Using FFmpeg, concatenate all .ts fragments into a single, full video.

Requests are captured in a HAR file to facilitate filtering m3u8 files.

Concatenation of *.ts video files using FFmpeg command line

Utility Usage

You can download the utility project from Github.

README.md

Additional Considerations

The Browser Mob Proxy request configuration must include the necessary header parameters whenever the video’s hosting website involves authentication Cookies, JWT, or additional fields are required. A recommendation is to use the Selenium WebDriver to automate the website’s log-in and retrieve any required credentials.

If the website uses skewed, relative paths to identify the video’s resource files, specific modifications to automate the download of the video may be required.