Cooking up an alternative: making a ‘knocking gesture’ controlled app
Posted on July 22, 2016 by Zoltan Kolozsvari
Have you ever seen a seismometer-application running on your smartphone? It’s incredible. The refinement and responsiveness means that it picks up on even the smallest vibrations. So, what if you could use these vibrations in the way we use touch screens: to control your phone?
This was the question posed by a small team in our Budapest office, who launched ‘Project Woodpecker’ to create a knock (vibration) recognition component embedded within an application. Our aim was to see how viable and useful ‘knocking gestures’ could really be.
Smartphones can be made capable of interpreting physical actions as abstract interactions, the same physical actions that basically all non-smart objects receive. Our devices are equipped with highly-sophisticated sensors, which provide clear data of all the mechanical forces affecting them. Forces such as knocking the side, back or vicinity of a phone can be recognised and turned into actions. All that’s needed is an application featuring a knock recognition component. Users can interact with their phones and control them with simple knocks.
Knocking gestures = splatter-free cooking
So how might this work in day-to-day life? Well, ‘knocking gestures’ work as a kind of remote control, where knocks can happen directly on the device, or on a transmitting substance — such as a table — if the gadget lies on one. So, if you’re strolling down the street and don’t have a remote built into your earphones, you could control your music simply by knocking a pattern on the phone which is in your pocket.
Here’s another example. There’s a glut of cooking apps out there, which brilliantly offer step-by-step recipes…but wouldn’t it be much easier if you could navigate the instructions for the perfect Victoria sponge without getting floury fingerprints on your screen as you scroll through the instructions? An app that could listen and respond to you knocking on the counter top would allow you to keep your beloved smartphone at a safe distance from all the mess of cooking, while still enabling you to walk through each step of the recipe — without even needing to touch it. Remember too, that ‘knocks’ are unambiguous; and people knock in the same way regardless of where they live or what language they speak. A knocking gesture is simple and universal.
Cooking up a solution
Convinced, we decided to create an imaginary cooking application that offers a step-by-step preparation function, utilising knocking gestures for navigating between the steps.
Basic patterns were used here for navigation purposes: two knocks, meaning ‘next step’, and three knocks, meaning ‘previous step’. One knock, or knocks faster or slower than a specific comfortable speed were ignored — similar to how double clicks work on a mouse. The UI responded to all sensed activities, so the user could understand if their knocks were recognised or not, and could react accordingly.
The knock engine
For knock detection we use aggregated data from the device’s accelerometer and gyroscope. The device continuously reports the forces affecting it in the three dimensional space. Each knock causes a big peak on the z-axis. We discovered two possible ways to recognise knocking gestures for this project. One would be based on crossing a pre-set threshold, while the other builds on pattern recognition.
However, using pattern recognition, the app could save the state of the device and continuously run a matching algorithm on this saved state, though this would be very time consuming to implement. Although this method would give us more precise results and random noise wouldn’t affect knock detection, we really wanted to see and test the concept in a live environment in the shortest time possible, so we decided to advance with the threshold method. This would be based on some basic rules: if the amplitude of the knocking’s resonance crosses a predefined limit, a trigger goes off and a command is executed. Though implementing the threshold method is much quicker and easier, the increase in noise levels in the motion sensors added another challenge.
Crossing the threshold
Using the threshold method also meant that detecting knocks while the user is walking or making any movement while the device is being held becomes very difficult, due to even the most minor movement of the hands. Such signals almost always crossed the threshold, so we agreed to limit knock detection to when the device is placed on a table or other flat surface, which we were able to detect at the start of the project, via a device’s motion sensor. When the signals were lower than a specific threshold value in each direction, the app could recognise that the device was not in motion, and enable the knock recognition engine.
We quickly realised that we were now introducing a new challenge — the table itself. Fine-tuning of the engine was necessary here to provide the required responsiveness. Every surface is made of a different material and transmits signals in different ways, which can impact the sensing mechanism enormously. We had to set thresholds, delay times and signal-filtering methods very carefully to ensure our knocks were recognised correctly.
Turning knocks into math
In the very first iteration we checked only whether the signal received on the z-axis was over a predefined threshold or not. However every knock produces not only an initial reading, but also resonating, or echoing readings. This means that this version could detect a single knock as multiple knocks. In order to eliminate this issue we started to collect and store the sensor’s signals from the previous few milliseconds and we detected a knock only when the signal was over the threshold and over all the previous signal levels. This simple solution meant we could detect knocks precisely when testing in a calm environment.
To detect multiple knocks, following knocks should be detected only within a certain time-frame. A knock gesture will be completed if no knocks are detected within this time period. After a multiple knock gesture had been recognized, a cool-down timer would start. The next separate knock gesture could be detected only after this timer has been stopped. The cool-down timer made sure that gestures were well separated and continuous knocking on the table (for example when chopping vegetables) would not create additional unwanted recognitions.
Knock by knock to the final demo
For the continuous development of our knocking framework, we created several sample applications.
At first we created an experimental app that only collected the raw accelerometer data. We captured simple knocking patterns (2–4 consecutive knocks with varying gaps between them), and visualised them as signals to see what the capabilities of the device were.
In the next step we moved on with a very simple app that could display the first results of our knocking framework. Here, if a double knock or a triple knock was detected, we represented them with small flashing labels. However we realised that without useful feedback, it was really difficult to learn the proper gestures. Instead, we presented every single knock event along with the specific number of detected knocks, clearly visible on the UI. This second approach was more effective, as the user could learn how to control the device much faster when receiving useful feedback.
Our third sample app was a UI prototype that used knocking gestures for navigating through pages. This gave us our first impression as to see what it felt like to control a device by knocking. To provide clear feedback for the user about the knocks detected, we represented the detected knocks as a series of dots, which then turned into a message bubble when a valid gesture was recognised. We found that with this control feedback we could also rely on the user’s perceptions to achieve good interaction, along with refinements to our knocking framework.
The fourth and final demo app iterated as a step-by-step cooking guide that fully integrated the knocking framework and unified all our findings from the entire process, including the navigation and the feedback systems.
This blogpost is part of our apps series on Codevoyagers. Check out our previous posts:
Learn with us
Take a look at our current job roles available across our 10 global offices.