Receiving ATSC digital television with an SDR
The popular RTL-SDR dongles commonly used for amateur software-defined radio were originally built for receiving DVB-T digital television, a broadcast standard used in Europe and elsewhere, where the American ATSC standard isn’t. But since presumably the radio is software-defined (SDR), a logical question to ask is, can these cheap RTL-SDR devices be used to watch ATSC as well?
Well it turns out, despite the common use case of these RTL2832U-based DVB-T receivers as SDRs, they don’t perform television signal demodulation in software. RTL2832 has a hardware demodulator for DVT-B (a 5 MHz — 7 MHz signal), in addition to the software-based mode, where the analog-to-digital converter passes raw IQ samples to the host computer for processing. This SDR mode only supports a bandwidth of 3 MHz, at best, not nearly enough for receiving DVB-T, nor ATSC (6 MHz) for that matter.
You could solder multiple RTL-SDRs together and use something like multi-rtl, in an attempt to attain the 6 MHz bandwidth required for receiving digital television signals. However, there are several mid-range SDR transceivers available with greater bandwidth encompassing ATSC.
In this post I’ll use the HackRF One (20 MHz bandwidth, plenty enough to cover ATSC, and then some) since that’s what I have readily available, but others have had success with the SDRplay, see: Watching ATSC HDTV on the SDRplay RSP
Virtual channels & subchannels
Before we begin to tune into the radio spectrum we need to know how the television “channels” correspond to frequencies. In the old days of NTSC analog television, largely phased out since 2009, the channel you see on your TV set would always map to a fixed frequency, e.g. 28 to 554–560 MHz.
With ATSC, this relationship is not so straightforward. The “channel number” for 554–560 MHz, if there is one, is chosen by the broadcaster, embedded within the transmission stream. Any 10-bit number can be transmitted (0–1023), arbitrarily. This is known as the “virtual channel number”. By contrast, the RF channel is always determined by the frequency (554–560 is still “RF channel 28”).
The FCC helpfully provides a website of DTV Reception Maps, where you can enter your address and see what channels you can expect to receive. The virtual channel is listed first, but you can click the callsign to see the RF channel.
But that’s not all. The virtual channel has two parts, a major and minor number. These may be separated by a dot, but that can be misleading since it is not a decimal number (unlike, say, FM frequencies, which are decimal numbers), or a dash, although that can be misleading too since it is not a range. The FCC website uses dashes so I’ll use that as well.
What is this minor number? The “subchannel”, another 10-bit number chosen by the broadcaster. Digital television is efficient enough to allow multiple simultaneous video/audio/data streams, so the subchannel identifies the specific stream in the broadcast. Some stations transmit up to a dozen or so subchannels.
Introducing GNU Radio’s file_atsc_rx
Back to the Watching ATSC HDTV on the SDRplay RSP tutorial. I found that article quite helpful, even without an SDRplay and Windows. Adapting it to HackRF on OS X. First find the GNU Radio ATSC receive example file:
gnuradio/gr-dtv/examples/file_atsc_rx.grc
Open this file in gnuradio-companion, the flowgraph is straightforward:
For reasons I am not entirely clear with, the File Source is read with Output Type of “Short”, and then converted to Complex with the “IShort to Complex” block. I deleted this block and changed it to read Complex:
Complex floating-point is the native format saved by GQRX, so if the flowgraph is adjusted to read in this format no conversion is needed.
To read/write data quickly, create a RAM disk:
hdiutil attach -nomount ram://4000000
diskutil eraseVolume HFS+ ramdisk /dev/disk4
The raw IQ captures grow large quickly, so having a fast RAM disk for saving and loading can mean the difference between overloading and not.
To capture the signal from the air to disk, try gqrx. Find a strong ATSC signal in your area, find the RF channel, then lookup the frequency ranges from North American television frequencies. Wikipedia gives a lower edge and upper edge, add +3 MHz to the lower edge to get the center. For example, RF channel 36 is 602–608 MHz, centered at 605 MHz.
Configure an 8 MHz bandwidth (enough to cover the 6 MHz of ATSC, and then some) in GQRX, and tune to the center frequency, 605 MHz in my case, and a strong ATSC signal should be visible:
Notice there is a spike on the left edge — this is the “pilot tone”, at precisely .309440559 (although it can be adjusted by an offset by several hertz by the broadcaster to reduce interference with analog stations). Listen to this tone as narrowband FM and it should sound like a solid tone, not too noisy for a clear signal.
To remove the spike at the center, choose “DC remove” in GQRX. There is a “DC Blocker” block in GNU Radio as well, but I just used DC remove in GQRX. Tools > I/Q recorder, then save to the ramdisk. Record for some time, as long as you can until the disk fills up.
Set the sample rate in file_atsc_rx to 8 MHz, set the input file name on the File Source to what you just recorded, then execute the flowgraph. If all goes well, it should output an mpeg.ts file you can watch in a player such as VLC.
Except it didn’t work so well for me. GNU Radio reported ATSC decoding errors:
!!! atsc_fs_checker: PN63 error count = 56
!!! atsc_fs_checker: PN63 error count = 56
!!! atsc_fs_checker: PN63 error count = 8
!!! atsc_fs_checker: PN63 error count = 53
and /Applications/VLC.app/Contents/MacOS/VLC reported MPEG transport stream errors:
ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 1, expected 2) for PID 18
To analyze what went wrong, lets compare to an ideal ATSC signal.
bladeRF’s ATSC-Transmitter
bladeRF ATSC-Transmitter demonstrates how to use the bladeRF SDR to transmit ATSC video, but it also helpfully includes a sample .ts file you can use to test with, ready to modulate into ATSC.
Open GNU Radio’s transmission example: file_atsc_tx.grc —change to read from the .ts, create an atsc.cfile, symbol_rate 10.7622M, match sample_rate in file_atsc_rx.grc, remove IShort to Complex, change File Source type from Short to Complex, then can decode .cfile to .ts. FFT of the generated signal:
This is a nice clean strong ATSC signal. Notice it looks better than the noisy ATSC signal I received over the air. We can use it to test GNU Radio’s ATSC decoding pipeline, to make sure it is working as expected.
Understanding the ATSC Receive Pipeline
ATSC Receive Pipeline is a hierarchical block, defined in Python atsc_rx.py. No corresponding .grc flowgraph ships with GNU Radio, however it is easy enough to construct one from its components, from this Tweet:
This allows each step of the pipeline to be analyzed individually:
My changes to this flowgraph are available here, “file_atsc_rx2” (be warned, many experimental changes for testing, your mileage may vary, edit it to suit your purposes as needed):
Let’s see how the generated ATSC signal is decoded.
Place an FFT after the file input, and then after the ATSC RX Filter block for comparison:
running the same flowgraph on my over-the-air received ATSC signal:
The edges around the generated signal appear to be more well-defined. Not the best signal, but it does decode into something. What is .ts, exactly?
Analyzing the MPEG Transport Stream
VLC can play MPEG Transport Stream (.ts) files, but to varying degrees of success depending on the corruption of the ATSC signal. Other tools to analyze .ts files:
ffmpeg -i mpeg.ts
Shows some general information. tsinfo is included in tstools (built from source from GitHub TODO: create a homebrew package):
tsinfo mpeg.ts
Tons of errors! tsreport, also from the tstools suite, shows more detail:
tsreport mpeg.ts -data -v
### PMT payload has zero length
### Error starting new PAT in TS packet at 1310172
### tsreport: Error reporting on input stream
Scanning a longer recording, “Found 24 PAT packets and 0 PMT packets”, but all said (e.g.) “Packet 2153 is PAT, but has no payload”. ~10 GB capture:
tsreport mpeg.ts -stdin -data -justpid 0 2>/dev/null
36209928: TS Packet 192607 PID 0000 [pusi]
Adapt (19 bytes): 48 fb ca ec 32 fa 49 06 9d 74 17 58 c0 36 4c 07 01 45 66
Payload (164 bytes): e9 ee 9e dc d6 f4 4b 92 ae e3 92 4d f7 d8 6f e3 0e da 15 52 64 7a 2d 4d c2 8a a0 bb ef 85 57 e9 09 55 46 0d 56 d1 16 5e 8c ea 60 b1 a8 ef bb 80 a7 eb 20 a8 c6 b3 22 76 94 a2 b1 46 66 80 c4 ff eb 55 94 ab 33 52 84 69 e8 08 03 da 38 50 86 89 c3 2a a2 63 43 62 10 25 5a 7d 83 5c bd c1 3b e5 48 c8 1e 3d 03 e1 d6 aa 3c 47 08 bc 7f 7c 52 e2 34 57 5c 6b 5a ab f9 16 91 6b 48 5e 1d a6 af 25 bb 16 96 d3 15 29 0e fa 03 6d 95 b4 c9 85 5d 5d d0 8b 3b 6c 36 dc 41 a1 76 2b 0b f1 7a 7f 81 d0 a2 92 df 41
tstools logs many errors about the “adaptation field”. This is a 2-bit field, specifying whether the adaptation or payload field or both are present, but 00 is reserved since having a packet with neither makes little sense:
### Packet PID 16c9 has adaptation field control = 0
which is a reserved value (no payload, no adaptation field)
Not clear why GNU Radio emits these invalid packets. The transport_error_indicator field is correctly set when the demodulator was unable to decode the packet, but it is not always set on these packets.
Using TSReader Lite (Windows program, in a VM), see some metadata:
So it is not all bad. The virtual channel table, including the channel short names (KICU-HD, etc.), are visible, although the video has many errors. On a real TV set, this would be enough to display the virtual channel and callsign, but not video.
A Word on Antennas
Adjust the ANT500 telescopic antenna to c/605 MHz, record about 5 GB, lots of errors but finally get some fragments of audio and video!
An HDTV antenna built for this purpose may further improve reception:
However, there are various types of RF connectors. The HackRF and other SDRs and the ANT500 antenna uses uses SMA (subminature type A), NooElec NESDR Mini 2+ uses a smaller MCX (micro-coaxial) type, HDTV antennas (including GE 10621 Ultra Edge Series Amplified Antenna) uses a larger type F F connector.
The RTL-SDR Blog adapters bundle could be helpful, as I couldn’t find a suitable adapter at my local electronics store. Using a real HDTV antenna will have to wait for some other time, I’ll have to see what I can do with what I have.
Gauging the reception by ear, tune to 602.309441 MHz, narrow FM demodulation, listen to the pilot tune with headphones and adjust the antenna to make the tone sound clearer and less noisy. Observe the waterfall and aim for a strong distinct signal. With some effort, was able to get a signal of sufficient quality to view the stream types:
VLC, Media Information…:
Matches what I see on my TV, including the subchannels. There is even ASCII text in the file, including the channel name:
strings -20 mpeg.ts
Demystifying the PSIP
Metadata packets transmit periodically in the MPEG TS (see MPEG Transport Stream documentation) stream:
Each packet has a 13-bit packet identifier (PID). PSIP is the standard for program information, most PSIP packets use PID 0x1ffb: master guide table (MGT), virtual channel table (VCT), and others. But event information and text (EIT and ETT) uses arbitrary PIDs set in the MGT, which lists table types and their associated PID where to find them:
To extract some of this data, wrote a quick and dirty Python script: psipdump.py — it finds a decent amount of channel metadata.
But there is already a PSIP parser in VLC, modules/demux/mpeg/ts_psip.c.
Not in VLC 2.2.4, to see the information, had to install a VLC 3.0.0 nightly build. This adds a --ts-standard atsc flag (if omitted, auto-detects), and then you can go to Window > Media Information to view the PSIP data:
As more of the EIT tables load, you can see further into the upcoming event schedule:
Even with a large amount of corruption from poor signal reception, much of the channel metadata (including the electronic program guide, in the PSIP) can be decoded successfully, given that is sent repeatedly.
This presents a possible application of SDR for ATSC. Scan for RF channels, show the virtual channels and subchannels, along with what shows are being shown on each of them. Even without working video/audio, perhaps this limited functionality would be useful for some purpose. However, I would really like to receive video, as television was intended.
With some further adjustments to the HackRF, enabling the amplifier (RF gain 14), tweaking the gain (IF gain 16, BB gain 20), moving the antenna, etc., I was able to get at least some fragments of choppy audio/video.
Live streaming with VLC
Rather than having GNU Radio read IQ samples from disk, and write decoded MPEG TS to disk, it can be modified to use the live samples over the air by replacing the File Source by an Osmocom Source. As for the output, the File Sink can be replaced with a File Descriptor sink, to be piped to VLC.
Create the GNU Radio File Descriptor block, enter 1 as the file descriptor, then run from command-line:
python file_atsc_rx2.py | /Applications/VLC.app/Contents/MacOS/VLC -
Source code for this flowgraph again, in case you missed it:
VLC will load fd://0 (standard input), if all goes well the video plays:
Conclusions
Receiving digital television broadcasts with a software-defined radio was met with limited success.
It works in principle, but with too many decoding errors, causing choppy playback, making it unwatchable in practice with my current setup. CPU power does not seem to be the issue, removing this variable from the equation by recording IQ samples to a ramdisk, no overflows observed, then decoding offline later, did not improve the decoded video output.
The limitation may be in the SDR transceiver or antenna hardware I am using. The HackRF One has an 8-bit ADC, compared to the 12-bit ADC of the SDRplay which reportedly could be used to successfully receive ATSC. Testing with a higher-end SDR should prove informative.
My antenna (using the introductory ANT500, a 75 MHz to 1 GHz telescoping antenna configurable from 20 cm to 88 cm, tested various lengths) is attached directly to the HackRF transceiver, which is plugged into my computer system sitting directly in front of it, indoors. A flat amplified HDTV antenna in a similar area can receive ATSC signals just fine, but I couldn’t test it with the HackRF One yet, without an SMA to F adapter.
Since the antenna is screwed directly onto the transceiver, which itself has a short USB cable to the host PC, there are limited options where I can move it to better ascertain signal reception quality. Running a coaxial cable between the antenna and transceiver would allow for more thorough testing. More radio accessories to acquire.
Long story short, as powerful as software-defined radio is, you cannot escape the hardware. If I in fact am running up against the limitations of the HackRF, it may be time to consider a Nuand bladeRF or LimeSDR. TBD.
Update 2016/07/07: See my tests with a bladeRF and a better antenna in Upgrading from HackRF One to bladeRF x40
