NTS — Aphex Twin Live Video Stream
Words on web technologies used around the Aphex Twin live video stream from Field Day.
Preparing for and live-streaming Aphex Twin from Field Day presented a series of interesting challenges and here I’ll be giving the technical insights into the project as the front-end and Node.js developer within the NTS tech team.
Source code for the project is available on NTS’s Aphexer and Afx GitHub repos.
Concept
An “Aphex Virus” took over the NTS website on the weeks preceding the live-stream, and there were 3 methods to creating it:
- Aphex Twin Face Replacement — Replacing faces within the host photos with Aphex Twin’s face.
- Font Switching — Switching between the regular NTS font and a bespoke Aphex Twin font.
- Aphex Twin Face Overlay — Flashing an overlay of Aphex Twin’s face.
A dedicated Aphex Twin page was also made for containing information on the project as it approached, finally becoming an embed for the live stream. Some user interaction was included on the page with the goal of providing data for usage within the live video stream itself.
Aphex Twin Face Replacement
Replacing NTS hosts’ faces with Aphex Twin’s required two key elements; locating faces within images and drawing Aphex Twin’s face onto those locations.
The Node.js Faced library (built upon OpenCV) provided the tools for face recognition. All that was required was to convert the original source image into a readable image buffer, easily achievable with Jimp thanks to it’s ability to read an image directly from a URL:
Jimp.read(url, function(err, sourceImage) {
sourceImage.getBuffer(Jimp.AUTO, function(err, sourceImageBuffer){
faced.detect(sourceImageBuffer, function(faces, matrix, file) {The next step was to draw a scaled version of an Aphex Twin face onto the correct location of the image. faced.detect’s callback returns faces , an array of Features providing information on the x, y coordinates of the detected faces. With these, we can determine the width, height and location of where to draw the Aphex Face.
After using Jimp to load the source PNG image for an Aphex Twin face, the scaleToFit function was used to scale it appropriately without stretching. Each Aphex face fitted slightly differently (some contained hair, more neck, wider section of face etc.), so a unique modifier sizeMultiplier was applied to the width and height to get a better fit:
let aphexFaceImageScaled = aphexFaceImage.scaleToFit(
face.width * aphexFace.sizeMultiplier,
face.height * aphexFace.sizeMultiplier
);Next, each Aphex face was rotated slightly. This helped remove some sense of uniformity of which was particularly obvious within images containing many faces. Calling the Jimp rotate method with a randomised rotation (within limits) applies the rotation wanted.
Note, Faced also returns the Feature object for the eyes on each face, which could be used to create a more accurate rotation.
Finally, the scaled and rotated Aphex Twin face is drawn onto the source image, achievable with Jimp’s composite function:
let aphexImage = sourceImage.composite(
aphexFaceImageScaledAndRotated,
x,
y
);
With multiple faces of Aphex Twin accessible, randomising the face used per detected face was possible, having some great results:
Client-Side Image Swapping
The logic behind swapping images client-side was simple and performed surprisingly well. The initial expectation was that the image-swapping would cause performance issues client-side however this was never apparent.
A basic jQuery selector was used to retrieve all the img elements to be switched. An img src such as “https://media2.ntslive.co.uk/resize/1600x1600/1dcbc68f-e76a-4ab9-bf47-881ebb18509f_1449754316.jpg” was requested with “resize” replaced with “afx”, routing the request to our aphexer service.
Each ‘Aphexed’ image was loaded with a preloader, and were swapped only upon their load to ensure a clean switch.
Infrastructure (words from Andi Studer)
Embedding the aphexer service within our existing media and backend AWS ECS cluster was chosen to keep infrastructure overheads at a minimum. An open source build script was used for including OpenCV (with Node.js) in a Docker container.
We hoped to keep the server load to a minimum by running the service behind a content delivery network (AWS CloudFront), however later learned this wasn’t enough to keep the servers stable (see more below in Lessons Learned).
The client-side image swapping was controlled by an object hosted on Google’s Firebase Realtime Database which was read on each page load. This allowed real-time control of the frequency, duration and probability of the face replacement and font switching.
Bespoke Aphex Twin Font
Aphex Twin’s logo was injected into any letters that had a shape to contain it. Not every letter was customised, helping reduce the intensity of the font switch.
Initially, the switching between the fonts was too subtle. The key to it feeling right was swapping the uppercase versions of the Aphex Twin letters with the lowercase versions. On the font switch, the Aphex Twin letters were always capitalised so as to put a focus on the them.
The Password Field
Archived page here.
In the 2 weeks running up to the live stream, next to the project’s information, an input box was used to collect ‘password attempts’ (via Firebase Realtime Database) for use within the live stream itself.
Excluding a brute-force hack attempt (of which there were later multiple), we had 62,653 genuine password attempts on day 1 only.
Within the password attempts, people expressed their frustration with the password, expressed their personal opinions (many “f*** donald trump”s), submitted their emails and names (didn’t even have to ask), pasted unused code from the NTS git history (!), quoted lyrics from the audio used on the page and other general gibberish. The largest percentage of passwords though were “Aphex Twin”, “Richard D James” and variations of those which proved to work very well for the video stream.
With the passwords collected (and personal emails filtered out), a simple web page was created to step through the passwords and display them within a terminal aesthetic (the scrambling effect achieved with a modified version of Justin Windle’s excellent Text Scrambler). The video team screencasted/recorded the web page in execution, with the green background allowing them to easily incorporate it into the live stream.
Full recording of live video stream available here.
Lessons Learned
Caching infrastructure (words from Andi Studer)
Using the content delivery network (CDN) infrastructure to cache the aphexed images proved insufficient and resulted in the NTS media service becoming overloaded unnecessarily. Switching off the aphexing periodically was therefore a necessity.
As it took the media service several seconds to respond to an aphex (face replacement) request, and the CDN had no cache available whilst it was being processed, multiple ‘aphexing’ requests for the same image on new media were being made. On pages like our Recently Added, this proved troublesome.
To address this, we pre-processed images as soon as they were uploaded to our file storage service (AWS S3), occurring long before the content became public. For this, we used an AWS Lambda function triggered by S3 Event Notifications to request and store an aphexed version of the new image.
Using Firebase Realtime Database (words from Andi Studer)
Using Firebase Realtime Database for the first time in a production environment worked surprisingly well but had a couple of gotchas:
- The Database Rules allow fine grained access control and validation, though took us a few attempts to really get them tight.
- Choosing the correct pricing plan before releasing would have been good. We started with the free plan but as this only allows 100 simultaneous connections, we quickly reached the limit. “Pay as you go” proved better suited to our needs. It allowed unlimited connections, thus handling the spikes in traffic whilst keeping cost minimal (< $10 for the campaign period).
As usual, a little more upfront reading would have saved us from some tricky moments… but then, how’d we learn ;)
