Why Screens Suck: Interaction Design in the Physical World

A version of this article appears on Purple, Rock, Scissors.

How great was Minority Report? Since 2002, I wanted to be Tom Cruise, conducting information with intentional gestures and arm swings via floating dashboards of blue light. Galleries of these mental GIFs form our visions for where digital is headed.

In 2013, minds were blown again. We met Joaquin Phoenix in Her, voice commanding his way through an invisible workflow. We had to reconcile our comfortable mental models of what UI could be with the idea that there could be no UI. It’s an appealing hypothetical: we still admire sunsets and smile at strangers, rather than being immersed in an endless interface.

This article is an attempt to internalize a few contradictions:

• I agree with both movies.
• I disagree with both movies.
• I love physical objects. Stuff like pencils and panic buttons.
• I’ve got carpal tunnel, eye strain, and backaches from a life chained to screens.
• I’ve dedicated my life to digital strategy.

Pop open the carton of milk that arrived after your smart fridge sent a notification all the way up the supply chain, ordered your favorite brand, automatically charged your supermarket account, then retargeted you with ads for competitor milk — and have a seat. Here’s why screens suck, why they aren’t necessarily the future, and how to handle it.

Why Screens Suck for Users

• You spend two-thirds of the meeting answering Slack messages and one-third pretending to be listening.
• Your toddler’s pacifier is an iPad game.
• Everyone at the dinner table is on their phone, so you take out your phone. You make a joke about it. No one laughs.
• Your husband swings his wrist to read a text on his Apple Watch, knocking his beer onto his Surface Pro. You look up from your MacBook and use your Android phone to pause the TV.
• The word unplug is now synonymous with relaxation.

Why It Matters

Avoid entertaining visions of increasing and impending isolation from others. We’ll probably always crave real hugs and high fives with our friends and family. We’ll continue to conspire in basketball courts and shared work spaces.

Actually, we like people so much that we scramble to straddle multiple channels for human connection at once. Attention makes us feel good. Say your girl Kim is in the middle of that great story, and you feel the vibration of your phone. Kim’s attention is probably guaranteed for the next few minutes; answer that phone. That text, like, heart, retweet, snap, or email in the form of a mysterious, anxiety-inducing blip of a notification screams, “someone else is offering attention.”

You swear you’re multitasking. It’s a myth. You’ve got a human brain. We only rapidly switch attention between competing things, creating the illusion of multitasking while actually neglecting each task.

Because a screen demands complete visual commitment, the more connected we are, the more we subtly but consistently offend each other.

Then wearables showed up. Though less invasive, checking your smartwatch still demands enough visual attention to cause a car crash. These are the words I’ll probably eat in five years, but here it goes: mediocre adoption rates for marginally easier user experiences will further prove what Google Glass already found out — a smaller screen is still a screen.

In other failed attempts to screen the world, no one’s raving about Chili’s tablet ordering. In one brand’s attempt to insert themselves into the user’s space, they fall short at the edge of yet another screen, solving a problem that didn’t exist, while creating new stress over established social norms, like whether the server still deserves 20%. (Chili’s, by the way, didn’t place plastic barriers on 45,000 tables for better UX. By cutting out trivial human interaction time, check averages increased.)

There has to be a limit to the tolerable number of devices demanding our attention. We have trouble being present at family dinners, work meetings, and birthday parties. Even when we’re here, we’re not.

Why Screens Suck for Designers

Hold on now. If making stuff for screens is your job, you might not feel like there’s a problem here. Hey, some experiences are better off confined to screens. Web and desktop applications are utilities, built to offer tools and steps to get things done in one focused space. With Oculus and Myo, I’d happily go Minority Report on a virtual wall of post-its. TV, films, and videogames are a distinctly separate escape from the often lame physical world. Such experiences are hereby quarantined from this infectious screen hate.

Three real problems include:

1. Screens have an edge.
2. They’re two-dimensional.
3. They are an infinitely variable box.

We spend lots of energy optimizing for the infinite resolutions, aspect ratios, and device settings on which our work appears — time we could be spending solving bigger problems.

One abstract problem is industry sustainability. As designers, strategists, and builders, what if we ran out of new ways to organize information and create experiences inside of rectangles? Imagining the end of innovating screen-based communication literally keeps me up at night. In my nightmares, we’re recycling UI patterns and channeling retro-inspired aesthetics, because like in all established fields, there is eventually no such thing as an original thought. Then what? Digital has felt new, for a long while, but how many more years can we spend tweaking conversion funnels before AI makes us obsolete? If I sound dramatic, it’s because I go a little crazy when I’m getting bored.

Every medium gets a makeover. Print had paper, then came embossing, laser cutting, and 3D printing. Paintings had canvasses, then came street art. Digital had screens, then came things. This is the part where pencils and panic buttons come back in style.

Designing Screenless Experiences

I won’t cram in all the exciting possibilities of IoT. I’ll stick to what I “know.” How could interaction design evolve from screens? Surprise: the answer is still user-centered.

Designers and agencies have to adapt to new environments and circumstances, much like backpackers, for example. It’s why we pack light, keep our eyes open, and explore. Think like a local. Understand the user’s entire world. Deconstruct problems through first-hand market research, immersive surveys and interviews, personas you would bet money on, and confessional-style user testing. Achieve creeper status until you breathe the contexts in which users live.

First Principles of Physical Interaction Design

Then comes making stuff with things. We’ll need a few new rules. This is a modest draft of recommended mindsets which, in the current accelerating climate, will be outdated by the end of this sentence. Still, here’s a start:

1. Any surface or object is an input device. The world is the experience.
2. Design for voice.
3. Sensors should help automate tasks. (But don’t overdo it. People lose trust in systems with little visibility.)
4. People want to focus on people. Eye contact, body language, and being present are making a comeback.
5. Work should be built in. There’s value and satisfaction in rituals like cooking, gardening, chopping wood, and using a pen. Completely frictionless experiences actually devalue the user’s perception of a task.

Real Problems with Screenless Solutions

Note that none of the following are one product or application. They’re tapestries that thread themselves into old fabric. They don’t demand attention. They stay out of the way.

• Users need milk. Don’t just blast the user with invasively targeted ads for milk. Design a user experience that makes shopping for milk obsolete. (This is how that carton arrived on your doorstep.)
• Users need to get healthy. Don’t just content farm endless workout videos, healthy eating articles, or celebrity endorsements for athletic wear. Build systems that reinforce a positive lifestyle. (Blue Apron, FitBit, and “smart apparel” companies are already on it.)
• Users need affordable healthcare. Don’t just roll out fancy features to make insurance easier to buy. What about an omniscient hub that moderates patterns and inefficiencies in the status of all hospital beds in the world, the current effectiveness of all prescription drugs, the rotation schedule of all medical residents, and the vitals from all patient wearables? (Too Big Brother for you? The data exists now.)
• Users need good education. Stop burying teachers in state-mandated curricula and tests. Hack a few Wii controllers, run interactive lessons, and literally gamify learning. Let teachers monitor and sort engagement data, graph improvement, diagnose what frustrates kids, and make data-driven decisions — then get back to teaching. (Yeah, this example still involves a screen or two, but for a fraction of their planning period.)
• Users need a healthy planet. Stop wasting energy on campaigns designed to guilt trip a society that celebrates excess. Every Earthly system can be more convenient, more efficient, more sustainable, more rewarding, more automated, more fun, cheaper, and easier — but only by design.

“Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke

Real Human Roles in Screenless Design

If I have a goal here, it’s to subtly persuade you all to consider exploring this world with me. Our collective minds are more valuable than our individual minds. It won’t be easy. We might even have to reinvent ourselves. It might look something like this:

• UX: We’ll keep breaking out our already ambiguous field by degrees of physicality. “Retail Consultants” will map cross-channel, cross-device, and brick-and-mortar touchpoints; “Gamification Strategists” will tap concepts like progress, incentives, and flow; “Service Designers” will know SaaS.
• IA: Information architects might cross-train as actual architects, using environmental psychology to resolve usability issues in interior design, lighting, and signage. We’ll model taxonomies and schemas that wrangle physical and spatial concepts. I can’t even.
• Engineering: Developers will rock more than code. Truly full-stack engineers might break out Arduino, Raspberry Pi, wifi triangulation, location services, RFID, NFC, BLE, and every sensor, button, dial and switch in Home Depot. It’s like electric legos.
• Project management: Think wizard Mickey in Fantasia.
• Business Development: You’re our evangelists. You’ll spark the client’s imagination by turning “I need a website” into “I need panoramic digital strategies that break traditional concepts of time and space.”

We’ll have to construct bridges to the physical world with concrete, not just code. Interaction design shouldn’t be confined to artboards and canvasses. Be a sculptor. Think holistically about digital to solve hard problems. Don’t just placate them with products trapped behind glass.

Because the possibilities are so darn endless, there’s this looming sense of dramatic irony whenever anyone claims to know what’s coming. Maybe this article will join Her, Minority Report, Space Odyssey 2001, Blade Runner, and countless other groundbreaking sci-fi movies in a time capsule of blind visions for the future.

Maybe I’ll just patent “the pencil with a panic button,” then go back to watching too much TV.

If you enjoyed this article written for Orlando digital creative agency Purple, Rock, Scissors, please hit the heart below to recommend it to others!