Touch base dialogues with things: Responsible IoT & tangible interfaces
By Iskander Smit
The ThingsCon report The State of Responsible IoT is a collection of essays by experts from the inter-disciplinary ThingsCon community of #IoT practitioners. It explores the challenges, opportunities and questions surrounding the creation of a responsible & human-centric Internet of Things (IoT). For your convenience you can read it on Medium or download a PDF.
End of 2017 Google & Levi’s will introduce a jacket on the market that is a result of a long research program Project Jacquard by Google ATAP. It may be the first big introduction of ‘smart garment’ on the market, at least one by big players. In the fashion tech scene a lot of experiments are done over the last years, often focusing on the identity of the person wearing the fashion, changing behaviour based on the wearer’s feelings. Other garments actively react to the outside world, like the BB.Suit by ByBorre that creates a personal bubble of clean air if the wearer is in a polluted city like Beijing. All of these are still rather conceptual though. We also see near-skin devices that merge the physical and digital: Digital and physical experiences are not that different anymore. Embodiment of the digital is a concept that is emerging and moving into the mainstream. At its core this is an iteration of tangible user interfaces (TUI). Where TUI is a new form of interacting with digital information, embodiment makes your physical behaviour and feeling part of the digital environment.
One of my favourite explorations into this new marriage of body and bits is the project Surveillance Spaulder of James Bridle where he puts some muscle contracting pads on his shoulder, connect these to a infrared sensor that is triggered by surveillance cameras in London. Every time he passes a camera walking down the street he literally feels he is watched as his shoulder contracts. This is direct link between impulse and action. (In that sense it is not Internet of Things as there is no internet involved, but data-driven nonetheless.) A jacket like the one from ByBorre can let you feel invisible threats, like air pollution. An interesting design question here is how pollution feels: Does a polluted area feel more heavy, does it feel restless by vibrating, or warm? As your phone gets warm because it is inefficiently processing apps, that warmth is a sign of fast drainage of the battery. These kind of direct relations are interesting and can lead to a new language with our things.
In 2014 Fabian Hemmert published his PhD-research about new forms of interaction with things, and he looked to phones in one of his research through design cycles. Could a phone that has more data grow, or become heavier? Do we need these kind of clues to be more attached to the invisible aspects of data? Even before that, BERG and Timo Arnall did research into making wireless connections visible with the Immaterials project through several experiments with light and smart photo tricks as time-captures. Much later, the Architecture of Radio app tried to do a similar thing. The quest is to make the invisible visible.
Touch plays an important role in social relations, as we learn from Gijs Huisman. He did his PhD on ‘Social Touch Technology’, or touch on a distance. I was involved in a valorisation project from this research and did some explorations the last years in the role touch and especially feeling touch can play as interface to others and to our world. Haptics is the science applying tactile sensation to interact with computer applications, and is the way this mediated touch is made accessible for interfaces. One of the research artifacts was a sleeve that is put on the under arm. It had 12 sensors and 12 vibrating engines. By touching on person’s sleeve, the other wearer can feel that touch remotely, be it a stroke or grabbing. The research found that the reception and understanding of stroking patterns resembles the human touch and people do feel a person’s touch at a distance. The use case for the sleeve was in the research context for death-blind people. Especially people born with this handicap have no other means of communication than touch; it is the only way to have a dialogue. There is a language of touch, especially done on the lower arm, a relatively sensitive part of the body.
Just like other languages, touch needs to be learned and is depending on mutual understanding of ‘words’. Just like in voice based communication, you can distinguish tone and meaning. If we don’t know each other’s language we can communicate by basic sounds and interpretations. The feel of a tone can be linked to an emotion: The same goes for haptic feedback. We can feel danger, love, and other extreme feelings — more subtle one are harder to understand. There is a cultural and gender bias too: Men understand certain signals less clear from women than from men and vice versa. My conclusion after a couple of years of exploring haptics: Touch is very powerful to stress emotions, and to notify with a meaning. But it’s rare to have an instant understanding. You can link a couple of different signals to concepts, but you need to learn these by doing. The Apple navigation app is embedded in the Apple Watch system and is pre-programmed with different haptic patterns to signal if you need to go to right or left. I need to learn this over and over as I do not use it every day.
In tangible interactions and haptic feedback the dialogue with the thing is key. We need a mutual understanding of the technology and the relation with the thing. Holly Robbins looks1 into to the relation we have with technology and sketches different levels of mutual praxis with the technology. With smart things, this mutual praxis with technology becomes a more important part of our life. The dialogue we will have is increasingly defining our relation with things. With new interactions like touch and speak we will have these dialogues, and they are necessary. The interactions will be more interesting as the intelligence of the things grows with the development of AI. Things will start to understand us and react to our behaviour, but we will also start to understand technology better — and be able to have a different conversation based on this newly developed understanding.
A question around intelligent things that have dialogues with us is to which degree the intelligence will live in the cloud, or in the things itself. The current architecture is often cloud based: Just think of Amazon Echo or Google Home, and also open source systems as ReSpeaker, all of whom use the cloud as the backbone for their conversations. This has limitations, though. It’s not that we need to be afraid for offline things, but rather the amount for data transfer that is needed as the conversations become more natural. We will see that things themselves will need to house part of the intelligence and use only impulses to trigger conversations and embed contextual information.
The separation of conversational intelligence and contextual savviness is an important architecture for the coming generation of things. Apple’s Airpods are an interesting example in this: You could have thought that they wouldn’t need a lot of computer power just to connect and send Siri instructions to the phone, but the devices do more than gesture & movement based interactions. Airpods are prepared for a much bigger role in the new digital services ecosystem.
In a model of the trigger-based interactions that emerge from wearable devices like Google Glass, Android Wear, and others, we see how information is served in little context-driven chunks. What and where and in what form is influenced by a mix of profile information of the user, and knowledge the system has on more static characteristics related to the service being used. Second, the system gets sensor-based impulses it can work with. Micro interactions, actions in the moment, together with these known characteristics allow it to calculate what to present at any given time. It’s a continuous dialogue.
The tangible user interface will play an important role in the digital near future. In the model of Heroshii Ishii2 we see that this type of interface requires a different approach as the mental model of the digital information needs to be made tangible too. Haptic feeling needs to come above the surface as we need a tangible representation of the digital model in the tangible space.
In an essay Tom Coates wrote on the shape of things in this tangible world, he made a clear distinction between embodied interactions and the Internet of Things. Embodied interactions are starting with digital interactions made into tangible experiences. “The internet of things, however, is much more about enhancing the physical with the digital, making the objects make more sense at a distance, or drawing out information from them and bringing it into a virtual space where we can do stuff with it.” He thinks the service layer is key to the experience of the things. In a research by Nazli Cila3 the role of things and data is divided in three levels. Collectors are the things that collect data by use, and present this data to the users as input for new decisions. We have a lot of these products in the Internet of Things domain now. Sensors are added to all kind of products, and the collected data is presented in dashboards as nice visualisations. The next level is Things as Actors: Products that take the decisions on their own. The data is used for intelligent computation. Like the June oven that measures weight and pictures the form of the food to decide upon the cooking scheme. Third level is Things as Creators. Not only are the products intelligent, and use data for decision-making based on designed rules. Rather, things start to design their own rules, and create new things.
What does this mean for ethics in IoT? Both Cila and Robbins are researching the role of integrating between computer and human. They assume that this will be a very close relationship. And with tangible and wearable haptic devices this relation will be felt, literally. In a research program we are setting up at TU Delft now, we are looking into the role intelligent things might play in the city of the future. Are things also citizens in the smart city? And what kind of agency would these things have? Do we trust things to become full citizens? Thinking of the closing words of Bruce Sterling at this year’s SXSW festival he made a clear case that ultimately AI will rule out humans as disturbing part of an efficient society.
We are not at that moment. The relation we have with intelligent things will bring new design challenges. A student from Eindhoven — Felix Ros — made an interesting device for autonomously driving cars. You can put your hand on the device that gives feedback on the ‘thoughts’ of the car, what decision the car is making.
It is also a mean to intervene with these decisions. The human driver is still in charge, for now. But will this be the case if we decide to give more agency to the car and not the driver? At LABS we designed a next iteration of an intelligent barbecue that addresses these kind of questions helping the incidental grill cook with indicators on the process based on inputs, like thickness of the food and temperature of the barbecue. A system that is automated but leaves space for the cook to make their own decisions. The barbecue shares its insights (for example on timing), but when and how to act is a decision the human makes. This makes the barbecue more personal and adaptable, and the next generation of barbecues can take into account these learnings.
In designing the next generation of intelligent things we are also designing a system of interactions between the physical device and a rule-based computation model. We use a mix of physical and haptic touch points, and decide where the intelligence of the dialogue is put: Does it go into the thing or into the system? As designers, we need to take a stand where the agency of the machine might interfere with that of the human.
Iskander Smit is working at agency Info.nl in Amsterdam, that designs and develops digital services. Iskander is responsible for research and development and leading labs.info.nl. This R&D department dedicated to new developments, and emerging connected technologies. Labs conducts research and experiments on the internet that enters our daily life and how we interact beyond the screen. Among others, Labs is partner in a research project together with the University of Applied Science Amsterdam on the role of vibro-stimuli as digital coaching and mediated social interactions (the internet of touch) for health related topics.
Iskander has a background as Industrial Design Engineer and have been working as creative and strategist in digital services. He is member of Council Internet of Things and co-founder of the Behavior Design AMS meetup. He initiated and co-organises the Amsterdam edition of ThingsCon.
ThingsCon is a global community of IoT practitioners dedicated to fostering the creation of a human-centric & responsible Internet of Things. Learn more on ThingsCon.com, join an event near you, and follow us on Twitter.
This article is licensed under a Creative Commons (CC by-nc-sa) license. Please reference the author by name. To link to this page, you can also use the shortlink bit.ly/riot-report.
- Robbins, H. (2015). Disrupting the device paradigm: designing for mutual praxis in connected objects. ↩︎
- Ishii, H. (2007). Tangible user interfaces. CRC Press. ↩︎
- Cila, N. Products as Agents: Metaphors for designing the products of the IoT age. ↩︎
