Sensual Computing
The great project of the Web has succeeded beyond anyone’s fantasies. Unthinkable quantities of data lie no further away than a few clicks away. Yet the potential of this new resource remains almost wholly untapped. Data is not knowledge, and data — on its own — does not lend itself to understanding.
This problem is not new. Long before the Web, engineers in the aerospace industries studied how to make a supersonic jet operable by a pilot who might be in the middle of a multiple-G banking curve, requiring full situational awareness with as little movement as possible.
Most of the modern work in augmented and mixed reality systems comes from these earliest experiments in flight systems instrumentation and displays. The modern jet fighter pilot uses a helmet with an augmented reality ‘heads up’ see-through display that overlays important flight, control, weapons and combat data across the pilot’s visual field:
For a pilot, less is more — presenting too much data tends to lower pilot performance, because each bit of data comes with its own ‘cognitive load’ — our finite capacity per unit of time to read, interpret and understand data. Put too much data on the display and the pilot can’t concentrate on flying. Development of the heads-up display focused on finding the right mix of data, presented in the most apprehensible form, allowing the pilot to make the best possible decisions.
Coming forward forty years — on the far side of a data explosion that made it not just easy to gather data, but even easier to publish and share data — we find ourselves confronted with the same class of problem. Our lives may not depend on making the best possible decision within a split-second, but nevertheless we need to be able to mitigate the cognitive load of all of this data.
At the moment, avoidance typifies our strategy for managing that cognitive load. We simply pretend the data does not exist. Almost any organisation of any scale anywhere in the world has collected a lot of data and uses nearly none of it, because the additional cognitive load to the organisation far outweighs the perceived value of using that data to improve organisational capacity.
This imbalance lies at the heart of one paradox of ‘big data’: we have so much of it that it’s become impossible to get our heads around. Yet lessons learned at the beginning of the journey into augmented and mixed reality computing could usefully be applied to the problems produced by the Web’s success. We might imagine ourselves as near kin to those pilots strapped into supersonic aircraft: flying through a data-rich environment, always trying to make the best possible decisions.
A reimagining of the human-computer relationship can rebalance the scales between the weight of data and our capacity to make decisions. All data must be weighed against its cognitive load — not solely, but absolutely. Data that carries with it a high cognitive load — because there is simply so much of it, or it is obscure, or for whatever other reason — must pass through a lens of apprehensibility that humanises its focus, shaping it for understanding.
“A picture’s worth a thousand words.” This antique insight provides justification for a push into a new era of ‘sensual computing’, where all of the ‘words’ of data find their way into ‘pictures’.
Sensual computing starts with the proposition that data is not particularly well formed for human understanding. The qualities which make data easy to collect and share are not those which make it easy to apprehend and interpret. Given the fundamental ‘unfriendliness’ of data, sensual computing defines a methodology to ‘humanise’ data, making data ‘subject’ to the human. At present, humans bear the full cognitive load of that data, and are ‘subject’ to it. This reversal, to a more natural order, serves as the foundation for sensual computing.
Sensual Computing as Decision Support Framework
Sensual computing centers on making the individual smarter. That begins with an inquiry establishing a framework for outcomes, centring on improving the capacity of the individual to make data-driven decisions through the application of sensual computing techniques:
- What decisions need to be made, or made better?
- What are the capacities of the individuals who make these decisions?
- What data is available to guide the decision-making process?
- Can improvements in decision-making be quantified?
What decisions need to be made, or made better?
Any organisation working within sensual computing methodology must begin with a strong sense of desired outcomes, while simultaneously acknowledging that anything learned in the pursuit of these outcomes might transform both expectations and scope of outcomes. Sensual computing is constructed as an ‘lean’ methodology, seeking both to reach its goals and to learn from its process — within its process — to refine those goals.
What are the capacities of the individuals who make these decisions?
Every individual has a unique set of innate and learned capacities that fundamentally shape the character of a sensual computing experience. These capacities determine the effectiveness of a particular sensualisation technique. For example, some people have innate capacities to interpret spatial relations; others have greater capacities in aural or tactile senses.
A ‘one-size-fits-all’ approach to sensual computing will inevitably fail to engage the capacities of some, possibly even the majority of users. Individuals immersed in sensual computing tasks must inventory both their capacities (innate and learned) and weaknesses; with this information, selections of technique can be made that favour and are conformant to an individual’s capacities.
What data is available to guide the decision-making process?
Decisions drive the choice of data in sensual computing. Rather than letting a particular data set drive the range of possible decisions, sensual computing seeks to identify all relevant data sets, delivering them to the individual in a way that allows the individual to explore, correlate and develop these data sets.
Data is more than simply a resource in a sensual computing framework; it is an exploratory medium. Within an agile methodology, insights gained from this exploration can be fed back into both the choice of data sets (widening or narrowing) and into the scope of possible decisions.
Can improvements in decision-making be quantified?
What can not be measured can not be improved. Sensual computing thrives on ‘moving the needle’, increasing human capacity for decision-making. Without a clear sense of what those outcomes mean, this methodology has no way to target results. In most cases this improvement will be easy to measure. Because so much organisational data simply lies dormant, any improvement in the capacity to make data-driven decisions will mark a zero-to-one step change.
In all cases this improvement (or lack thereof) must be measured; first because this is the essential ingredient in a lean ‘build-release-measure-refine’ process; second, because this helps the enterprise come to a deeper understanding the benefits of making a sustained investment in sensual computing.
Sensual Computing Methodology
The following chart provides a high-level overview of the different disciplines that together comprise sensual computing: Decisions, Data, Visualisations and Interactions:
Decisions drive the choice of data. Data determines the scope of visualisation techniques. Visualisations frame the range of possible interactions. Interactions define the framework within which decisions can be made, tested, and applied. Through a process of feedbacks — and further interactions — this will lead to new decisions being identified, new data scoped, new visualisation techniques developed, and new interactions defined, in an lean methodology that embeds test-points with the outputs of this methodology, making it easy to measure and continuously improve decision-making capacity.
Decisions
Sensual computing meshes the computer’s mastery of data with the human capacity for understanding. The touch points — where data meets capacity — define the regions where decisions can be substantially informed and influenced by a sensual computing methodology.
Sensual computing requires effective, up-front modelling of the decision to be made. While many decisions can be improved through sensual computing, it may not be a good fit for decision support where there factors affecting that decision have not been fully identified.
Data
After decision outcomes have been clearly defined, data that can usefully support decision-making must be identified. This data might already exist in abundance, but big data tends to drown important and ‘weak’ signals in an overwhelming avalanche of ‘data noise’. Data scientists serve as key contributors during this phase of sensual computing, working to locate and provide a path into the most relevant data.
A data scientist may find that even with copious amounts of data collection, the data available does not facilitate decision support. In this circumstance, the data scientist may recommend the collection of new data set, or alternately suggest revising the set of decisions and outcomes.
Visualisations
Sensual computing encompasses a range of senses. Although work most often begins with visualisations, humans are multi-modal, tuned to multiple senses simultaneously. Sensual computing means sensualisation: an approach that acknowledges not just what the eye sees, but also what the ear hears and the fingers touch. Visual, aural and tactile dimensions of sensation work together to produce an experience greater than the sum of the parts.
Every individual has different capacities to apprehend visual, aural and tactile information. There is no ‘one-size-fits-all’ approach to sensual computing. Instead, sensual computing offers a deep ‘toolbox’ of sensualisation techniques. As part of an agile methodology of testing and improvement during this phase of the sensual computing methodology, these visualisations should be easily modifiable, until the right mix of sensual components has been identified for an individual and decision task.
Interactions
The feedback between visualisation and interaction — manipulating something and observing its effects — is among the most effective forms of human learning. Jean Piaget’s theories of Constructivism show from the time we’re small children, we are intent experimentalists, learning about the world by playing with it.
Bringing this same capacity to sensual computing, we can get our hands stuck into the data: pulling it apart, examining it, putting it back together or changing it, to see what else it changes. Where done well, the interface disappears — for example, Google’s Tilt Brush — because the sensual computing environment fully engages the creative capacities of the individual.
Within interactions, an experience of the relation between data and decisions emerges. This results in real understanding, transcending the sterility of seeing but not being able to touch.
Sensual Computing Project Teams
Effective project teams for sensual computing draw personnel and capacities from each of the four identified disciplines.
Decisions: Each team will need deep participation by stakeholders and analysts, driving the selection of decisions and determination of successful outcomes.
Data: In order to succeed, a sensual computing project team needs data scientists who understand both the decisions that need to be made, and the data resources on hand to make those decisions.
Visualisation: At the core of the sensual computing team are team members with both depth and breadth in a broad range of sensualisation techniques. Team members will bring their own ‘toolboxes’ of favoured techniques, and — in the most successful outcomes — add to their toolboxes with bespoke tools created to meet the needs of a particular constellation of data, decisions and interactions.
Interactions: Interaction design is fundamental to sensual computing, tieing data into decisions via visualisation interactions, using interactions to amplify and reify an individual’s capacities within the sensual computing environment.
We are at the beginning of understanding the subtleties of interactivity in sensual computing; most of the talent pool available today has learned their skills either in web design or gaming. The intersection of these two disciplines is likely to be the most fertile ground to locate capable sensual computing interaction designers.
Sensual Computing in the Enterprise
The business case for sensual computing is straightforward: decisions that have been made poorly, or at great time/expense, can be made more quickly, accurately and much more economically.
In every enterprise there are a class of key decision-makers who are informationally overloaded — swamped with too much data presented too incoherently for effective decision making.
Identifying these individuals is a necessary first step toward ‘moving the needle’ within the enterprise. Look for the most informationally overloaded, offering them the opportunity to employ sensual computing to increase their decision-making capacity:
- Where is data too hard to use?
- Where are important or weak signals drown out by ‘data noise’?
- Which individuals can most benefit from ‘humanised’ data?
The sensual computing project team needs to carefully analyse every aspect of the identified individual’s decision requirements, data resources, and native capacities, working to craft a solution that uniquely meets their needs. Built within an lean methodology, that solution provides test points into all four aspects of sensual computing (decisions, data, visualisations and interactions) so each can be measured and modified by the team throughout the process.
Continuous improvement should lead rapidly to measurable increases in performance — both in the time it takes to make a decision, and in the quality of the decisions being made. Use sensual computing to ‘move the needle’ with data-overloaded individuals — delivering the best return on investment — and those successes will springboard sensual computing across the enterprise.
Conclusions
Graphical computing has been with us for almost two generations, but we still struggle to use it at anywhere near its full potential. The Web has offered rich media connections to data for twenty of those years, yet has only resulted in increasing our cognitive load. Sensual computing offers us the opportunity to have our immense computing and data resources conform to our requirements, rather than forcing us into theirs.
Sensual computing is not new. The core technologies of virtual/augmented/mixed reality are not new. Visualisation is not new. What is new is our understanding of what is possible when we make the data conform to the person. With sensual computing, we have a key to unlock human capacity and potential. We simply need to change our point of view from data-centric to human-centric: from this, all else follows. Sensual computing can not solve every problem, but it will help us to understand and make decisions about the world we have created.
(In a post to come soon after the New Year, sensual computing methodology will be used to analyse a handful of current- and first-generation immersive projects.)
