Machine Learning for Front-End Developers: An Interview with Charlie Gerard
From flying drones with her thoughts to fashioning interactive
clothing, creative technologist Charlie Gerard gives us an insight
into her many side projects and explains how front-end devs can get started with machine learning
Charlie Gerard is fascinated with human-computer interaction, in
particular how you can use web technologies in unusual ways outside of the browser. She realised that if there’s an API or JavaScript framework, you don’t actually have to use devices as they were intended but instead you can hack them and build your own prototypes. For her first side project, she controlled a Sphero robotic ball with a Leap Motion and a sprinkle of Node.js in order to make it move with hand gestures. It eventually led her to wow conference audiences around the world with a talk about controlling things with the brain using JS.
“I was researching other devices I could buy and came across brain sensors,” Charlie remembers. “I became interested in neurotechnology and it really opened up a world for me. I had absolutely no idea that you could just buy a brain sensor and that as a dev I could build something with it. As soon as I discovered that, I had to tell people.”
Experimenting with brain-computer interfaces in JavaScript
Charlie learned that some sensors like the NeuroSky give you access to raw data, which can be used for machine learning. Others give you access to mental commands or facial expressions, making it possible to interact with a web interface through eye movements. At the time there wasn’t a JavaScript framework to use with the Emotiv Epoc sensor, so to build an interface, Charlie wrote her own open- source JavaScript framework, based on the sensor’s C++ SDK, as well as a Node.js add-on.
It enabled other developers to play around with this technology without having to learn C++ or Java. The experiments she built with it include a brain keyboard (look right or left to highlight a letter, blink to select and display it in the input field), pushing a three.js 3D cube and even flying a mini drone with mental commands.
Charlie became more interested in machine learning and found it wasn’t as scary as it seems. “At first I thought only data scientists would understand it,” she explains. “But understanding the basics doesn’t actually take that long. It’s about coming up with an idea and starting with a small goal, then diving a bit deeper.”
Front-end developers can try out machine learning through open-source
JavaScript libraries such as ml5.js and TensorFlow.js, developed by Google. The latter currently offers three features: using a pre-trained model, transfer learning and defining, training and running a model entirely in the browser, all in JavaScript (for a more detailed explanation, it’s worth having a read of Charlie’s Smashing Magazine article).
“Go in steps, use a pre-trained model first and then combine it with custom training data before doing everything in the browser,” Charlie advises. “It requires a bit of knowledge and the training can take a lot of time. Doing it all in JavaScript is interesting if you wanted to visualise the training process and how the accuracy of the predictions change over time but you probably would fall back to Python if you were building something for production because it’s faster. So it’s still very early stages for web developers. Most
of them are building experiments to learn more about machine learning but now more and more people are joining because they realise it’s doable in JavaScript.”
For one of her creative coding experiments, Charlie added detection
model PoseNet, which recognises body shapes in images and videos, to TensorFlow.js so that she could play VR game Beat Saber in the browser just with her hand movements — without the need for expensive equipment.
She also prototyped a gesture recognition system with an Arduino microcontroller and TensorFlow.js to play Street Fighter with
body movements. The latter combines machine learning with another one of Charlie’s passions: hardware. And this features quite heavily in her projects, for example when she built wearable interfaces in JavaScript to use clothes as an input device.
“I came across conductive thread, which is embedded with stainless steel,” Charlie explains. “If you attach a microcontroller, you can get data from it just as if you plugged in a wire. So when your body comes in contact with the thread, you can use that as an input. It’s really flexible and easy to work with. You can sew it into clothes, a pillow, a towel or a couch — anywhere that you have fabric — and make them interactive. You could even embed it all around your house and create your own way of interacting with interfaces or devices without having to pick up your phone.”
Side projects at the intersection of art, science and technology
Charlie builds her prototypes in her personal time to learn technologies she doesn’t get to use at work. A senior front-end developer by day, she’s had roles at ThoughtWorks, the New York Times and most recently Atlassian, where she focused on improving the user experience of project management software Jira. Now she’s about to join Netlify in Amsterdam.
Astonishingly, Charlie — also a Google developer expert — only moved into web development five years ago. Having started out in advertising as a digital producer (a type of project manager) at a few agencies in Paris and Sydney, she worked with developers at a time when 3D came to the browser via WebGL. She realised she was more excited about the interfaces they were building than her own job, so she decided to quit and enrol in a 12-week General Assembly bootcamp.
It was one of the best decisions of her life. “Initially, I tried to learn to code by myself after work,” she remembers. “But when you don’t know anything about an area, you don’t even know what to Google. I knew what HTML was but I didn’t know how to add JavaScript to a website. I thought if the bootcamp didn’t work out, I could always go back to my job as a project manager. At least I would better understand what my developers were doing. But I really liked that feeling of writing a piece of code and it does something in front of you. I still feel that with hardware now. I loved it and never went back.”
Creating new, smarter interactions
While the problems Charlie solves in her side projects are often harder than the ones at work, whenever she writes a blog post or speaks at conferences, her goal is to demonstrate that nothing is inaccessible. “I’m really excited about that,” she explains. “The research papers I read to inspire me to build something are really hard to understand. They’re written in a very scientific way but they’re needlessly complicated. You could break it down and make it so much easier. Anybody can come up with a good idea and build something interesting if they know that a certain technology is accessible. Otherwise you just feel blocked by thinking it’s too hard and you’re not even going to try.”
And so there’s seemingly no limit to Charlie’s side projects. There’s always something new to discover. She maintains a list of potential prototypes she wants to work on (for example, interacting with the electrical signals in plants) and researches a wide range of topics.
“I like to mix more than just tech, so I don’t only read news about front-end development and JavaScript. I follow what research centres like the MIT Media Lab or Disney’s labs do. I spend a lot of time exploring their projects and if they’re not open source, they at least give me an idea of what researchers are looking into. I then think about a way to make it happen with web technologies. If there’s a project about interactive textiles, I think about how to build something that’s a bit more helpful to people with motion impairments, for example. I always try to find a different angle and that usually leads me to the next thing.”
When Charlie bought her second brain sensor, it was entirely open source, exposing her to a lot of raw data. She realised she couldn’t really do anything with it until she knew more about machine learning and so that has become her focus over the last year. She’s currently looking into different kinds of data that can be used for machine learning. For her Beat Saber experiment, she used camera data, while her Street Fighter demo used accelerometer and gyroscope data. Now she’s exploring sound.
“If you build a spectrogram out of sound data, you can see the difference in patterns between when you’re speaking, knocking on a door or chopping carrots. As microphones are all around us, in laptops, phones and smartwatches, you can build software to recognise these patterns without changing your hardware and buying expensive smart devices for your home. So you could automatically pause your TV when your phone is ringing or pause a video on your iPad when you’re cooking and the system hears the sound of chopping.”
As usual when tinkering with new technology, Charlie not only notes the
opportunities but also the limitations. For example, to work with machine learning data, you need lots of samples and it’s important that you check their quality.
“When you train your algorithm, remember it’s just a piece of code,” Charlie
advises. “It’s not actually smart. It’s going to find patterns but you might not always understand why it’s predicted something. There is a lot of trial and error. Sometimes you may have to go back and, for example, check your data is labelled correctly to reguide the algorithm.”
The more Charlie looks into new types of human-computer interaction, the more she finds keyboards and phones to be quite restrictive. “There are so many ways that you could make your house adapt more to the way you live your life,” she points out. “You could make your environment a lot more playful and a lot smarter than it is now. I’m endlessly fascinated by coming up with creative ways of doing things that we didn’t know we could do. Making a device work for you is so much more exciting than just learning how to use it.”
This article originally appeared in issue 329 of net magazine in 2020 and has been reviewed by Charlie Gerard prior to republication.
