Building interactive textile-based interfaces with Arduino and JavaScript

9 min readJan 3, 2019

A few years ago, I came across Project Jacquard, a project by Google ATAP to create an interactive denim jacket to control apps on your phone, such as picking up or declining calls, changing music, etc…

I thought the idea was super interesting and wanted to replicate it to understand how it works but never really found the time until recently!

I really love exploring innovative technologies in a way that’s accessible to a lot of people so I spent a few days over the Christmas holidays building a quick prototype using some Arduino components and JavaScript. Here’s the end result so far:

Rough prototype of interactive textile-base interface

It’s a very rough 1st version, it’s extremely hacky on both hardware and code side, so I’m sharing this post only to explain the overall concept, so anybody can reproduce it if you’re interested :). It’s definitely not production ready and there are a lot of improvements to be made!

Concept

Before starting the prototype, I spent some time trying to figure out how the Project Jacquard jacket works, to have a better idea of how to go about building it.

I knew it used conductive thread, which is a thread that looks and behaves like normal sewing thread but is also conductive. It allows it to be used to replace wires, but can also be turned into a capacitive sensor, so it can sense if you’re touching it or not. In “classic” capacitive sensing, when you’re reading the capacitance value, you get 0 if a component is not touched and 1 if touched. In this project, I used a more advanced type of capacitive sensing where we can get data in a wider range.

If we think about a basic “swipe” gesture, we could start with trying to allow a user to perform a “swipe right”, by sewing 2–3 threads in a column-style layout, connect them to pin 7–9 on the Arduino and detect if we’re touching them from the one connected to pin 7, to the one connected to pin 9, for example.

Let’s get into a bit more details.

Material

To build this, on the hardware side, I used:

And on the software side:

Arduino
Node.js
Three.js

Steps

Instead of working on the “swipe right” example I mentioned above, I was intrigued about how they built this:

As mentioned above, touching threads in a certain order from left to right would have been quite easy, I imagine I would have had to check if the data I get from touching the thread went from 0 to 1 on pin 7, then 8, then 9; but in the 3D animation above, they were able to have a more interesting interaction.

After doing some research and reading their paper, I realised they sewed the threads as a grid, to be able to translate each “point” (where horizontal and vertical threads cross) into x and y coordinates. These can then be used the same way you can use the mouse x and y coordinates to animate things in the browser.

1) Sewing the threads

I have never sewn anything in my life.

It was my first time and that in itself was a challenge… I‘d advise you to buy a embroidery hoop to keep the fabric from moving, it makes everything so much easier!

To make sure I was sewing straight lines, I drew them on the fabric before starting so I could just follow them as I was sewing.

Also, make sure each piece of thread is not in contact with another one otherwise your reading will be wrong.

In the end, after a few attempts, I ended up with a grid like this:

As you can see, I have 3 rows and 3 columns, so a total of 9 possible points I can touch.

As these “points” will be translated into (x,y) coordinates, the more columns and rows, the better, because the more points you have, the more you can divide your screen and interface to have a smoother animation.

Now that I sewed the threads, let’s move on to assembling the hardware.

2) Hooking up the Arduino to the fabric

The basic setup is the following:

A 10MO (MegaOhms) resistor is connected to pin 4 and pin 8, and an alligator clip does the link between pin 4 and whatever you decide to attach to the other hand (conductive thread, wire, coin, banana, etc…).

The 10MO resistor is necessary for the type of capacitive sensing we’re doing here, if you use a lower resistor, it might not work as well.

This works for 1 piece of thread, but in my case, I have 6, so I had to move away from connecting to the Arduino directly, and instead, use a breadboard.

Here’s an example for 2 pieces of thread but if you need to connect more, just repeat the same setup for each thread.

In reality, this is what it looks like… 😱

Very hacky setup of alligator clips with Arduino

It’s currently super messy but it can be made smaller if you want to move away from prototyping.

Each alligator clip is connected to one hand of each thread. Now, let’s move on to the code.

3) Getting the capacitance in Arduino

Below is a few code samples from my setup, but you can start with this script.

I used the CapacitiveSense library. Below is the initial setup of the different variables.

We indicate which pins are used for the capacitive sensing, we declare integers x and y that are gonna hold the coordinates we’re gonna send to Node.js, we also create integers to hold past coordinates sent so we don’t send unnecessary data to Node.js, and finally we create strings for the result sent over. I used strings because Arduino doesn’t have arrays or hash maps built-in so for now I just used strings and will find a way to refactor later.

Then, in the setup function, we start by resetting the value on all pins, initiate our result variables that we’ll send later over serial, and begin the serial communication.

In the loop function, we read the value coming from all capacitive sensor pins, check if they reach a certain threshold (if the thread connected to these pins is touched), and if we haven’t sent these (x,y) coordinates already, we send them over serial.

That’s basically it! As I said before, it’s a very quick and dirty solution that needs to be improved but it works! :D

4) The JavaScript back-end

In Node.js, my original thought was to use the Serialport module to get the data and use web sockets to send it from the server to the browser to change the animation. I have worked with things like this before without any issue, but for some reason, with this setup, the serialport communication was fine at first, but as soon as I tried to hook up web sockets as well, everything was extremely slow and a lot of the data was lost, so I had to find another way.

I decided to use child processes to have one that handles the communication with the Arduino, and one that sets up the web server and sends the data to the browser.

I don’t think it’s ideal and I haven’t refactored it yet, but i’m sure there’s a better way to go about this.

The main Node.js file looks like this:

The arduinoData.js file looks like this:

In this file, we require the modules, open a serial communication on the same port as the Arduino with the same baud rate and when we receive data, we send it back so the arduino.on('message') function in the previous file is triggered and then sends the data to the webServer.js file that looks like this:

In this file, we create a simple express server and when we receive data from our process, send it via web sockets to the front-end.

5) Animating the front-end

In our front-end file, to receive the data coming from the server, we write something like this:

var socket = new WebSocket('ws://localhost:8000');socket.onmessage = function(e){  var msg = e.data;}

Now, the whole communication between the sensors and the front-end should be set up so we can do whatever we want with it! I decided to work on an animation that looks like the one Google made but you could do a lot of other stuff! :)

It would be too long to explain in details how the 3D animation works, but I basically created a wireframes plane using Three.js, divided it into 9 parts and animated whichever part is activated from touching the thread.

I hope to be able to get into more details in a future post! :)

Opportunities

To be honest, I haven’t put that much thought into the different kinds of opportunities that could come from such technology. I was more excited by the challenge of recreating it!

However, as Google used it in a jacket, one of the most obvious opportunities is to have that kind of technology in clothes, but not only! It could also be embedded into couches, blankets, etc… in situations where you could interact with some interfaces or devices without having to pick up your phone.

Limits

As you can see, the whole thing is pretty hacky and bulky at the moment so the main limit is that it’s not wearable or portable in an easy way. However, with time, that could be fixed as you could replace the Arduino with something way smaller like an ATTiny85 for example.

Another limit is the amount of thread I could sew by hand. If I could use a sewing machine, that would probably allow me to use more threads and have a nicer and more fluid interaction in the front-end.

Also, at the moment, it works with an Arduino UNO so it has to be plugged into my computer. Ideally, I’d like to change that to a wifi or bluetooth enabled micro-controller so it could be fully portable.

Finally, I feel like the rate of data transmission between Arduino and JavaScript is a bit slow compared to other projects I’ve built before, so optimising the code would be necessary to improve it.

That’s pretty much it! I wrote this blog post pretty fast so it is probably incomplete but I wanted to make sure I share what I learnt so far in case it’s helpful to other people.

I’ll probably write a few others once I improve the prototype and build others! :)