Visualizing & Augmenting Jacquard’s Gesture Catalog
Who says you can’t teach an E-textile new tricks?
Our fundamental goal is to make Jacquard’s capabilities widely accessible to the global community of developers to encourage a hacker culture around it. In doing so, we have also managed to demonstrate a handful of its untapped capabilities/applications that show exciting potential for the future of e-textiles. Today we will be demonstrating the usage of our data collection, visualization and annotation tooling which altogether enabled a new gesture that leverages Jacquard’s ability to discern varying levels of applied pressure per thread as a second dimension for user input!
Data Visualization for Existing Gestures
We created an iOS data collection application (linked below) and corresponding python visualization utilities to record and play back existing gestures without having a Jacquard jacket. The technique of visualization is largely inspired by the popular arcade game, Dance Dance Revolution, where the timing is communicated by a scrolling sequence on the y axis. In this case, the numbers on the y axis indicate how many time steps into the future a pressure reading is, thereby communicating that the top of the visualization is the current reading and the bottom of the visualization is 45 time steps ahead. Below are the four gestures officially supported by Levi’s and Google:
We see the level of capacitance on each of the 15 threads on the horizontal x axis colorized to form a heat-map, where brighter colors indicate a higher capacitance and therefore a higher amount of applied pressure while darker colors indicate a lower capacitance and therefore a lower amount of applied pressure.
Prior to this semester, last year’s MUC Team discovered a fifth gesture encoded into the firmware of the jacquard tag, which they dubbed “Scratch”. By moving your hand back and forth on the jacket, this gesture’s status code is broadcasted by the jacket over bluetooth but the official app does not acknowledge it at all.
Creating Custom Gestures
The default, supported Jacquard gestures make effective use of the relative arrangement of the parallel threads along a single axis, thereby enabling only single dimensional gestures. We have managed to leverage the parallel arrangement of threads in the Jacquard Jacket’s cuff along with the variable capacitance that each thread measures, enabling us to discern levels of pressure applied on each thread.
Using both of these dimensions of user input allows us to introduce two dimensional gestures, such as pressing lightly on the whole cuff and, without lifting your hand, pressing harder on the whole cuff. We have dubbed this as “Force Touch”, heavily inspired by Apple Inc.’s “Force Touch” technology which enables trackpads and touchscreens to distinguish between various levels of force being applied to their surfaces.
Data Preprocessing and Segmentation
We used segmentation and machine learning to train the jacket to recognize new gestures. We take a constant stream of capacitance readings from each of the 15 threads and flatten the 45 most recent readings into a 1 x 675 input vector, diagrammed below:
Neural Network Configuration and Performance
We trained our model with a neural network of the following configuration.
model_m = Sequential()model_m.add(Reshape((SEGMENT_LENGTH, N_FEATURES), input_shape =(input_shape,)))model_m.add(Dense(125, activation='relu', kernel_regularizer = keras.regularizers.l2(0.01)))model_m.add(Dense(25, activation = 'relu'))model_m.add(Dense(5, activation = 'relu'))model_m.add(Flatten())model_m.add(Dense(M_CLASSES, activation = 'softmax'))
The above neural network configuration yielded an f1-scores of 0.97 for negative training examples, 0.91 for positive training examples, 0.91 for negative held-out-testing samples, 0.77 for positive held-out-testing samples.
More importantly, our model was able to predictably output sequentially increasing probabilities of a ForceTouch reading as a positive event approaches and then similarly output sequentially decreasing probabilities after the event has completely transpired. We therefore take an informed approach to registering Force Touch gestures, where we wait for n consecutive high probability outputs for Force Touch from streaming live data before officially registering the gesture as detected (warm up phase). After this, we wait for n consecutive low probability outputs before acknowledging that the aforementioned gesture has completely transpired and thus another gesture is ready to be detected (cool down phase). So without further ado…
…Jacquard, meet “Force Touch”
Challenges and Future Work
Perhaps our most challenging obstacle with the introduction of this gesture was how frequently the model confused it with the “Cover” gesture. However, after providing it further labeled training data that distinguishes the two gestures, the model performs impressively well. This warm up and cool down approach also accounts for accidental duplicate readings of Force Touch along with random, fleeting spikes in Force Touch probability. We anticipate these type of mistaken similarities to arise when more custom gestures are introduced to Jacquard’s gesture catalog. Such a situation will inherently bring up the question of whether it is better to create a single model that outputs multi-label confidences for all custom gestures, or to have a binary classification model allocated for each gesture to use One-vs-All approach.
Links to Source Code
(Python) Data Processing, Visualization, Annotation Tooling + Custom Gesture Training Utilities
(iOS) CocoaPod with Force Touch
(iOS) Data Collection App
Georgia Tech’s E-Textile Hacking Team is always looking to collaborate with engineers and researchers across industry and academia. If you are enthusiastic about new forms of human computer interactions like this one or interested in our work on the Jacquard E-textile, please reach out!
