Ecosystem of modes

Screens have dissolved into the fabric of everyday culture. Welcome to the ecosystem of modes — a new material for designers to explore.

While preparing another interaction design course I’m teaching this winter, I keep coming back to the Patterns for Multiscreen Strategies we published in 2011. The slide deck has since been viewed 70k+ times, downloaded 3k+ and liked 600+. The patterns have been cited in all kinds of publications, from hands-on online articles to scientific books.

Those six patterns — Coherence, Synchronization, Screen Sharing, Device Shifting, Complementarity, and Simultaneity — helped us navigate smartphones, tablets, and desktops working together as an ecosystem of screens. Today, Netflix syncs your watchlist, Kindle remembers your page, AirPlay shifts content between devices and so on. These patterns have dissolved into the fabric of everyday culture.

But something fundamental has shifted. Today’s systems don’t just respond — they orchestrate across voice, vision, gesture, ambient sensors, and adaptive interfaces. We’re no longer designing for screens. We’re designing for an ecosystem of modes. This orchestration itself has become a new material for designers to explore.

What follows is how I’m thinking about this evolution. I wrote down six patterns we’ll explore and experiment with throughout the coming semester.

The Building Blocks: Modes of Interaction

Before we can orchestrate multimodal experiences, we need to understand some fundamental modalities. The list might not be complete, but helps as a starting point:

Tactile — Physical touch, haptic feedback, texture, pressure
Gesture — Body movement, hand signals, spatial manipulation
Visual — What we see: displays, light, color, form
Voice — Spoken language, conversation, verbal commands
Audio — Non-speech sound, music, sonic feedback, acoustic cues
Spatial — Location, environment, position, proximity, ambient sensing

Multimodal Interaction Patterns

Each mode — whether it’s a sensory modality or a situational state of human or machine — has unique affordances and contexts where it excels. The interesting questions emerge when they work together, creating choreographies of interaction where timing, rhythm, and transitions matter as much as the modes themselves.

1. Coherence

An experience that feels unified across all interaction modes, with each mode adapted to its specific strengths and context.

Example: Your smart home responds coherently whether you adjust lighting through voice (“Alexa, dim lights”), a tactile wall switch, or a spatial gesture. Each interaction feels part of the same system while being suited to its mode.

Open question: When does pursuing coherence actually limit each mode’s potential?

2. Adaptivity

Systems that intelligently adjust their modal presentation based on user capabilities, contexts, and literacy levels.

Example: An interface that simplifies itself for novice users or adjusts between modes based on environmental factors (like automatically switching to voice when detecting driving).

Open question: How do we make adaptation transparent without requiring users to explicitly manage their preferences?

3. Synchronization

All modes stay in sync, maintaining a unified state across the entire system.

Example: Stopping the playback of a music track or movie on your smartphone automatically stops it on other devices.

Open question: What’s the cost of real-time synchronization? When is asynchrony actually better?

4. Mode Shifting

Actively transitioning from one primary interaction mode to another as context changes.

Example: You start a recipe search by voice in your kitchen (“show me pasta recipes”), shift to visual browsing on your tablet while shopping, then switch to audio-only instructions (hands-free) while cooking.

Open question: How do we make mode transitions legible without making them laborious?

5. Complementarity

Different modes serve distinct, complementary roles in a unified experience.

Example: A sign language learning app uses visual demonstration, tactile feedback for hand position correction, audio pronunciation guides, and gesture tracking to verify accuracy. Each mode contributing its unique strength.

Open question: When do complementary modes start competing for attention instead?

6. Simultaneity

Multiple modes active at once, creating richer, more nuanced interactions.

Example: During a video call, you see participants (visual), hear their voices (audio), receive tactile alerts for raised hands, while AI provides captions and spatial awareness adjusts lighting to your focus level.

Open question: How much simultaneous input can humans actually process meaningfully?

Questions we’re asking

As we work through the semester, we’ll be looking for:

Real-world examples that demonstrate these patterns (or break them entirely). Where do modes complement vs. compete? When do transitions feel effortless vs. jarring?

Counter-examples that challenge these patterns. What successful multimodal experiences don’t fit this taxonomy at all?

Ethical considerations that emerge. When modes transition, when systems adapt, when AI orchestrates — these moments need to be legible, not mystifying. Each mode has its own character. The challenge isn’t to hide this complexity but to make it understandable.

The fundamental question: Before adding another mode, another sensor, another interaction layer — does this add value to community, society, humanity? Or are we creating complexity for its own sake?

An Invitation

If you’re researching or designing multimodal systems, I’d love to hear what you’re seeing:

• What patterns are you noticing in your work?
• What examples would help students understand these concepts better?
• Where do these patterns fall short?
• What am I missing entirely?

Feel free to get in touch via Linkedin or send me an email — I cannot wait to hear from you. I‘d also be happy to discuss and explore these patterns in a workshop with you and your team. I promise to bring music.