The screen is dead: Augmented spaces are the future of interaction design
by Sjors Timmer
In the past 50 years, computers have infiltrated the work environment to the point where there’s hardly a job left where they’re not in use. And in the process, all the ways we understand the world — through sound, touch, and spatial interactions — have been flattened into pictures behind glass. These days we mainly interact with the world via mouse taps and finger swipes.
But new technical developments such as augmented reality, object recognition, and spatial computing give us the opportunity to incorporate space as a medium for interaction design. How could we make computers disappear as devices, and just become part of the environment? We have to let go of design concepts that have been developed for a world of flat screens and start over with designing for spatial interactions.
Thinking with space
For his 1990’s paper The Intelligent Use of Space, researcher David Kirsh watched many hours of Parisian pastry chefs going about their business and saw how important space was to their thinking.
How we manage the spatial arrangement of items around us is an integral part of the way we think, plan and behave — David Kirsh
Just look at carpenter Paul Seller in his workshop, where he’s using these structural arrangements. If you watch the video, you’ll hear him explain that he’s going to make a join between two pieces of wood. But you might have already guessed that just by looking at this photo.
If you follow the sequence of tools on the workbench from left to right (his right to left), you can predict he will start with a line drawing, pick up a hammer, and then use one of the chisels.
There’s more — something that comes so naturally that I didn’t notice it at first. All the chisels point with the sharp edge away from him, and so does the saw, and of course the hammer points with the handle towards him. When we arrange things, we’re constantly lowering the cognitive burden for our future selves.
Working with information
Now it’s all well and good if you’re a craftsperson working with objects, but what if your material is information? Well, have you ever placed an envelope next to the door to remind yourself to take it to the mailbox the next time you go out? Or used a whiteboard to keep track of a meeting’s progress? We already naturally use space to help us think. Sure, we could probably remember and think through these ideas without the help of spatial artifacts, but we would be slower and more likely to forget things.
Another benefit of making information spatial is that we can more easily collaborate as a group. We see this idea in games such as Monopoly or The Game of Life, but perhaps most obviously in workshops.
A format that takes information spatialization to the next level is Business Origami. Originally developed by Hitachi’s Design Center, it uses paper cut-outs as representations of people, groups, channels and environments and it invites service workers to create miniature models of the systems they work in.
How might we fuse tangible and digital information?
But what about when we bring in computers? How can we interact with digital information beyond a screen? As I mentioned, augmented reality, object recognition and spatial computing give us exciting new opportunities to use space as a medium of interaction design. Already we’re seeing examples of whole new ways of interacting that combine tangible and digital information.
Amazon Go
Take Amazon Go — a platform so well designed that the strangeness of its existence has barely been examined. The computer has disappeared so far into the environment that all that remains of it is the QR code that you need to scan when you enter the store.
Amazon Go challenges our notion of what a computer can be. We no longer perform actions on the computer — it quietly takes our whole being as input. Has the computer disappeared, or has the computer become everything?
CityScope Boston
One of my favorite examples is CityScope Boston, an interactive model created by MIT’s City Science lab. It was developed in collaboration with Boston’s public transport service to provide citizens with tools to explore various possibilities for local streets and immediately see the impact of their choices.
This installation could be placed in local community centers to allow citizens to explore the impact of different traffic solutions on the roads they live on. You don’t have to be an expert in government policy or computing to understand the impact of different choices. And when different stakeholders can see how their choices impact each other, it opens up the floor to more meaningful and thoughtful discussions.
Dynamicland
Dynamicland is a non-profit research lab building a new computing medium where people can work together with real items instead of screens. Combining cameras with image recognition software and projectors, it provides an open programming language. You can assign capabilities to cards, posters, and pens — any object that can be recognized by the cameras — and it uses these objects to create incredible interactive environments.
Four rules for the future of interaction design
So how we can use this deeply ingrained understanding of space to work with computers better? What’s the best way to exploit the power of computers, and use the best of our human abilities?
1. Start with the physical world
Interacting with the world was easy in the pre-computer era. If you want to take photos you’d pick up a camera. To look at photos, just open a photo book.
To invent a new relationship between physical and digital, we have to stop reusing our current metaphors developed for screens and start from scratch. Forget the last 40 years of graphical user interface design, and start with objects and actions in the physical world.
2. Design for action
Unlike what’s often been said, we don’t examine first then take action second. In reality, we try things out, and in doing so learn and expand our knowledge.
So instead of letting users yell voice commands into the void, guide them with the surrounding space. Use objects to give people hints of what might be possible.
3. Design for collaboration
When ideas are laid out in space, it’s easier to understand what’s going on, because everyone sees and can interact with the same physical artifacts. Like the example of Dynamicland, everyone shares the same space, and all the underlying actions that have made this happen.
Make the most of this opportunity, and design interactions for multiple people to collaborate on. Instead of single user, single device, think community computing.
4. Design for exploration
Perhaps the greatest strength of computers is their ability to rapidly model all kinds of different scenarios. When computers can help us visualize and understand the impact of our choices, we quickly learn and make the best choices. Use physical space to help people explore complex digital information.
By redesigning the way we interact with computers, we can reawaken our dormant social, spatial and physical skills to help us tackle complex problems — from improving the way we collaborate in offices, to opening up city planning and democratizing decision making.
