Mar 18, 2015
Beyond Language
Scrying the Horizon of Interactive Design
We are at a unique juncture in the long-term progression of technology and human interaction. While automated systems and the mythical AI continue their exponential trajectory toward ubiquity in our environments, we find ourselves inextricably linked to technology in such a way that we must now fully assume that the human-computer interactive experience can and will become synonymous with the human experience.
Beyond the craft of software design and development itself, the interactive medium has reached a platform of maturity which clearly demonstrates new paradigms in the experience of reality— interacting directly with a touch screen is ready-to-hand to the same degree as Heidegger’s hammer. We use it without theorizing. Children today, even infants and animals(!), are discovering infinite ways to manipulate their technological world despite having minimal exposure to the keyboard and mouse which strictly dictated the constraints of human-computer interaction for a generation. So where do we go from here?
Learning from the Future
Science fiction has always provided the raw material from which we are able to envision the next step, but visionary projection is only part of the equation. Media like Star Trek: The Next Generation went so far as to foresee our phasing of disparate manual devices into a glass interface as well as interaction through flat colored tiles as the minimalist and purely-functional interpretation of how computers (still bearing the influence of the Victorian typewriter) could manifest in a distant future. From there, they took it a step further with the ability to interface directly with a central computer through natural language commands — another growing trend we see in consumer electronics today — but that warrants a separate conversation.
Recent flat design trends are a first leap toward a functional and aesthetic standard derived organically from actual human-computer interaction, fundamentally unbound from the analog world. Whereas the height of skeumorphism’s popularity represented an attempt to bring the physical into the digital, mimicking material interaction on a flat screen with keyboard/mouse or finger-taps, our current trajectory is clearly aimed at the third dimension.
Minority Report was one of the first films to realign the possible with holographic interaction suspended in space. More recently, Ender’s Game and Prometheus have shown how we can expect technological advancements (no more gloves!) to allow for this new level of digital-spatial manipulation to seamlessly merge our virtual and physical realities. Until then, we are operating in a transitional phase, taking advantage of various gadgets to integrate human movement and translate it into functional commands on a variety of devices and platforms.
These examples are an early expression of an emergent language, visual and symbolic at a high level but capable of including and conveying all existing vehicles of thought. This post-linguistic communication is a circular return to our roots: pictorial and iconic/symbolic expression was invented and leveraged for idea-transmission long before modern alphabetic languages like English established themselves as useful tools for commerce and widespread dissemination of words on paper. It appears that we are settling into a new equilibrium, balancing genuine human language with interactive visual display.
Learning from the Past
What fascinates most about our heritage of artistic communication is its dynamism and nuance relative to the static and immutably-linear expressions of letters-on-page. As you read through this sentence, each word is essentially a container of sound pointers. These phonemes are dependent on their pairings and relative order to create sequences of context, which are interpreted through memorization to represent bits of information which can be assembled into larger strings of meaning.
Such rigid communicative tools — bound by explicit definition and complex syntactic requirements — present a significant challenge for anyone looking to communicate nuanced and abstract ideas. This is especially true if you want to share information between linguistic frameworks. If your sound-symbol library or internal lexicon of meaning is not aligned with mine to a certain tolerance of similarity, little can be done to bridge the gap.
A visual language, such as the ancient carvings of Göbekli Tepe, maintains meaning and communicative power through a different mechanism. By constructing image through icon and artistic expression, complex information is conveyed holistically. This creates an opportunity for new and evolving interpretations as one is able to leverage their own rich visual-symbolic memory to achieve unique understanding. Of course we cannot know the creator’s full intent, but one can recognize familiar animals and shapes to instantly derive meaning in a way that an unfamiliar alphabet cannot provide.
Effort toward a visual, memetic form of communication is the key — and will be a driving force behind the evolution of purely accessible, universal interactions on and beyond our physical screens. Designers, developers and technologists with an eye toward the future (and a willingness to embrace a constantly-changing medium) will be uniquely positioned to influence a future in which there is no boundary between the digital and physical worlds. A survey of our current opportunities reveals a clear trajectory with many unanswered questions.
Today and Beyond
Pioneers of virtual reality (VR) are in the process of integrating our minds directly with the interactive experience. Immersive worlds like XING: The Land Beyond are poised to embrace the awe and wonder of a new frontier, just as the dreamscapes of Myst provided the perfect canvas for emerging 3D graphics technology decades ago. This begins the long and inevitable march toward absolute visual and haptic realism, the pinnacle of which will allow humans to escape physical existence for a virtual, lucid dream much like The Matrix itself. Until then, we have an interesting set of challenges to consider.
A visually-immersive VR experience will go far in convincing the mind that it has entered a new world, but the need to move freely and feel in any meaningful way is going to drive industries and innovations that have yet to be seen. How do we simulate the resistance between two lightsabers, and the dynamic environment of a duel? Magnetic-haptic sword peripherals, combined with visual trickery to keep players moving within an enclosed space (like the famous Star Trek holodeck) might be a reasonable start—but the infrastructure and enclosure required may prove too much for some.
So, as much as Morpheus, Oculus and SteamVR are an evolution of virtual immersion, I’m not looking to leave reality behind completely. Google Glass establishes wearable tech with a computer and lens overlay, but this is functional computing as an intermediate on-par with the (soon to be) pervasive interactive smart-watch. What I’m anticipating is augmented reality — a layer of visual technology woven into the data stream already being processed and rendered real-time by the mind. A seamless overlay of fully-controlled computational muscle that enhances my existence at a fundamental, ambient level.
Iron Man and others have driven the modern interpretation of ‘Heads-Up Display’ (HUD) such that we have a rough idea of where this ship is heading. Concentric circles with isometric and radial linework mirror the shape and focus of the eye, providing a middleware layer between our perception and the outside world. Status messaging, dashboards, video feeds and ambient environmental visualizations extend information into physical space through lightweight wearables and ethereal holography.
What was once an existence driven by the need to extract knowledge and understanding from extremely limited resources now requires the ability to process, filter, manage and apply an overwhelming torrent of information that bombards every moment of our waking lives. As we design and build interactive systems, the focus must be on minimizing noise and maximizing the specific information needed within a given context—despite the inherent paradox of doing so with even more embedded technology.
As such, the challenge is to usher it all into reality through a fusion of software and hardware design. The end result: a functional symbiosis between human and machine which can ultimately become our shared destiny. Display of key information, biometrics, identification of unknown opportunities and access to the infinite knowledge of the internet will be the nexus through which humanity elevates itself into the next stage of technological evolution (no nano-implants required).
Taken one step further, into your living room, Microsoft’s HoloLens declares that “the era of holographic computing is here” with interactive floating globes, tabletop Minecraft and seemingly infinite ways of turning your daily life into a digital wonderland. As we explore this new interactive frontier, designers and developers can expect a period of explosive innovation unlike anything seen since the emergence of touchscreens. That is, until the size and limits of primitive first-generation hardware force us to evolve yet again.
The next step is to blend this technology seamlessly into our environment, and facilitate ubiquity such that individual holographic experiences become shared interpretations of reality for all who choose to enter the realm of the previously-unseen.
Shared Vision
Given that the input for our eyes is nothing more than a stream of photons and interference patterns, contribution to this prima-materia opens wide the confining spectrum of our biological limits. Just as corrective lenses fundamentally change what the mind can process from the perceived world, so too can evolving technologies facilitate dynamic experience by managing the filters of our default reality.
Magic Leap challenges us to “imagine being able to generate images indistinguishable from real objects and then being able to place those images seamlessly into the real world.” If their “lightweight wearable” can evolve into something as seamless as a contact lens (or conventional glasses at least) we may be looking at the ultimate trajectory of technology augmenting the natural experience— and a legitimate platform to “bring magic back into the world.” At scale, the shared spectacle of hundreds watching a whale float over the beach or thousands experiencing a holographically-enhanced football game could fundamentally change the scope of communal social interaction.
Outside any specifics of their technology, the perspective espoused by Magic Leap CEO Rony Abovitz seems uniquely aware of the overlooked truth that “nothing that will ever match the grandeur of what our own brains can create in terms of visual experience.” If everyone involved in the ongoing eruption of these new interactive experiences manages to keep this in mind, and maintain a healthy respect for the infinite majesty of the high-def holographic supercomputer that is the human brain, it is easy to see how the 21st century can and will provide us new perspective on how to use it.
