A new future for classic TV
Not too long ago (as a good Treker), I finished re-watching all seven seasons of Star Trek: Deep Space Nine (DS9), a series that ran from 1993–1999. Deep Space Nine was shot on film, but all the post production was done on video in Standard Definition (4:3 ratio, 480i). Deep Space Nine is great sci-fi, and is frequently considered a fan favorite.
Deep Space Nine, however looks dated. Between a central character being computer generated for a portion of many episodes, and the grand battles later in the series, the CGI has that unrefined look that is typical for television in the ‘90s.
As a Star Trek fan, it was a happy day when it was announced that both the original series and The Next Generation were being updated to HD. The original film was rescanned, and all of the special effects were updated. This required thousands of hours of effort, and millions of dollars of investment to update 10 seasons of classic television.
Wouldn’t it be nice if CBS spent the time and energy to update Deep Space Nine? The issues is that apparently CBS never made money on the HD conversions for TNG and the original series. Let’s be honest, Deep Space Nine is not particularly significant to people other than true Trek fans (even if it is, in many ways, a superior series), and will never see an HD conversion like the other two series saw.
The first HD broadcast of a popular TV show was the Tonight Show in 1999, so the thought of widescreen, high definition wasn’t even a thought for most scripted shows until the 2000s.
Two decades later, HD is the norm. Even the most basic smart phone shoots HD video. Special effects have improved significantly since DS9 aired. CGI is used extensively on TV on even mundane scenes to reduce production costs. Building worlds within a computer is cheaper and quicker than building the real thing, and 99% of the time it’s seamless.
The cost to update classic TV shows becomes cheaper each year as technology improves, but it is extremely unlikely that classic TV will be updated to HD in the way that TNG and the original series was. There is still a lot of manual effort that would need to go into updating any show, especially something like Deep Space Nine.
The Advent of Neural Networks and Machine Learning
In the last decade, there has been a lot of work in neural networks. In 2014, generative adversarial networks (GANs) were leveraged to produce some astounding results when set to a particular task.
For instance, when a properly trained neural network is given low resolution images of faces, it does an amazing job of creating something resembling a face.
A neural network generates new information based on how it was taught. An obvious use is to have it upscale video where possible. Upscaling video and sharp details is great, we’re not too far away from high quality up-scaled video from SD sources.
Upscaling shouldn’t be our goal though. If we (as fans) wanted the “real” treatment, we want the special effects to be updated too. Goals like re-rendered space shots with high resolution models, and more realistic ancillary effects. Updated live-action shots where there are CGI elements with proper lighting and reflection maps, etc.
Luckily neural networks are capable of much more than upscaling images. GAN trained neural networks can create music, generate artwork or images based on low resolution or low fidelity input.
Imagine updating all the ship CGI to be more realistic without actually building up-to-date 3D models, relighting, and re-rendering all the shots. What if you could just point our “smart” computer at a space battle and say “make it better”.
This is the power of neural networks. Instead of upscaling images, computers recreate details based on what they’ve been trained to do.
In fact, NVIDIA is pushing the boundaries in exactly that way. SPADE takes a rough sketch where you paint with “sky”, “dirt”, “etc”, and SPADE creates a realistic image (with reflections and everything).
SPADE begins to show the power of having the computer “create” instead of “upscale”. When it comes to DS9, increasing the resolution isn’t enough. SPADE was trained on thousands of images of real world rocks, etc to that it could learn what a rock looked like.
In Deep Space Nine, one of the primary characters can change shape, and the effect is akin to the T-1000 from Terminator 2 (the liquid metal terminator). Of course on TV in the ’90s the effect was low quality and can take today’s viewers out of the moment. Imagine feeding in images of metal objects to train a computer on how this character should look, and ask the computer to replace the subpar CGI metal in the show with high resolution images that look integrated into the scene.
Or, feed in ship battles and imagery from modern Trek (and other shows/movies) to train it on how ships should look, and how explosions, etc look in space and have it go to town. No need to spend human time modeling and thousands of hours of computer time rendering these images. Just use the existing footage as a roadmap and “add detail” and “improve the look” of these ships and battles.
Neural networks can even be used for sound. There was an example recently where a neural network was used to artificially add sound to Google street view images based on what the computer “saw”.
We’re not there yet, but I think there is an opportunity for “smart software” to computationally upscale video and sound for older TV shows into high definition. SRGAN upscaling and SPADE image generation are step in the right direction, but these are baby steps. This same technology could someday be used to what it took thousands of person hours and millions of dollars to do for TNG and TOS.
Currently a lot of this technology is used to upscale older video games with impressive results. There is a long tail to get from video games to full-motion, live images, but we’ll get there.
I wouldn’t be surprised if a company like Netflix is already invested in this technology. It benefits a streaming service to offer classic TV in HD. There are so many classic TV shows that could be updated from black and white to color, and of course standard definition to high definition.
There’s even the opportunity, much further in the future, for episodes to be created from scratch for classic TV. Those missing episodes of Doctor Who from the ’60s could be recreated from the original scripts, etc.
The question isn’t if there will be Deep Space Nine in HD (or 4k VR), but when will it happen?
