Decoding field of TUI

Based on technical report on “Tangible user interfaces for peripheral interaction” by Darren Edge.

Technologies of Interaction: WIMP to Calm Technology

Conventional WIMP interaction- Windows, Icons, Menu and Pointer 6 fundamental actions associated with it- select, position, orient, path, quantify, and text (Foley et al., 1984).

Graspable User Interface: Properties of graspable user interfaces.
1. space-multiplexed, each virtual function having a single dedicated handle;
2. strongly specialized, each handle tailored to suit its single virtual function;
3. concurrent, multiple handles acting as parallel points of user control;
4. spatially aware, handles having positions within a spatial frame of reference;
5. spatially re-configurable, handles being free to move within that reference frame.

Whilst these properties still hold in terms of current understanding of TUIs, the idea of graspable function handles is fundamentally ’pre-tangible’; it is concerned with physical extensions of graphical user interface elements, rather than predominantly physical interfaces consisting of meaningful physical representations.

TUI: TUIs enable users to exploit and augment their spatial and social contexts, through the meaningful coupling of physical and digital media. TUIs in a way try to achieve Mark Weiser’s vision of “calm computing” by introducing “graspable media” on the one hand – located in the foreground of activity and at the focus of users’ attention – and “ambient media” on the other,existing in the background of activity and at the periphery of users’ attention (“tangible bits”).

Peripheral Interaction: The term “peripheral interaction” is used to describe this new class of TUI: ’peripheral interaction’ is about episodic engagement with tangibles, in which users perform fast, frequent interactions with physical objects on the periphery of their workspace, to create, inspect and update digital information which otherwise resides on the periphery of their attention. This style of interaction achieves a balance between the “calm technology” of Weiser and the “engaging user experiences” of Rogers (2006), in which “people rather than computers […] take the initiative to be constructive, creative and, ultimately, in control of their interactions with the world – in novel and extensive ways”.

“A new way of thinking about computers in the world,one that takes into account the natural human environment and allows the computers themselves to vanish into the background”.

Evolution of TUI: Origin, Emerging and Embodied

Origin: Their aim was to “make computing truly ubiquitous and invisible” by taking advantage of “natural physical affordances (Norman, 1988) to achieve a heightened legibility and seamlessness of interaction between people and information”; this would be accomplished through TUIs that “augment the real physical world by coupling digital information to everyday physical objects and environments”.

Emerging: By reference to the Model-View-Controller (MVC) structure of GUIs, in which there is a clear demarcation between input and output, the structure of TUIs is presented as Model-Control-Representation (physical + digital) or MCRpd. The key characteristics of this interaction model are as follows (Ullmer and Ishii, 2001):

1. Physical representations (rep-p) are computationally coupled to underlying digital information(model);
2. Physical representations (rep-p) embody mechanisms for interactive control (control);
3. Physical representations (rep-p) are perceptually coupled to actively mediated digital representations (rep-d).
4. The physical state of the interface artifacts partially embodies the digital state of the system.

Embodied: Embodied interaction, or the “creation, manipulation, and sharing of meaning through engaged interaction with artefacts” (ibid.):

1. computation is a medium;
2. meaning arises on multiple levels;
3. users, not designers, create and communicate meaning;
4. users, not designers, manage coupling;
5. embodied technologies participate in the world they represent;
6. embodied interaction turns action into meaning.

The first pair of design principles relate to the fact that interactive technologies have meaning not just through their functionality, but through their symbolic social value and role within systems of practice.

How does tangible interaction realization of ubiquitous vision?Whilst ubiquitous computing is about the perception of computer-mediated interaction fading into the background, it is not about “ambient” background computing; rather, it is about a blurring of the divide between the physical and the digital, such that we no longer notice the difference. Tangible computing is therefore just one realization of the ubiquitous computing vision, making computation more real by augmenting the physical objects in the environment with digital information, and vice-versa, until familiarity with such augmentations render them as the new reality in our subconsciousness.

Types of Representation

Holmquist et al. (1999) refer to three main types of representation in TUIs: tokens are “objects that physically resemble the information they represent in some way”; tools are “representations of computational functions”; and containers are “generic objects for the transient storage, manipulation and distribution of digital information.”

Themes

The framework itself consists of four themes, each elaborated by a set of concepts that aid in the understanding and application of those themes in TUI design; it presents a “deliberately non-restrictive view” of tangible interaction that enables “systematic shifts of focus”, rather than strict design prescription. The four themes and associated concepts are as follows (Hornecker and Buur, 2006):

1. Tangible Manipulation refers to “the reliance on material representations typical for tangible interaction”, through the concepts of haptic direct manipulation, lightweight interaction and isomorph effects.
2. Spatial Interaction refers to “how tangible interaction is embedded within space and occurs in space”, through the concepts of inhabited space, configurable materials, nonfragmented visibility, full-body interaction and performative action.
3. Embodied Facilitation refers to “how configurations of objects and space affect social interaction by subtly directing behaviour”, through the concepts of embodied constraints, multiple access points and tailored representations.
4. Expressive Representation refers to “the legibility and significance of material and digital representations”, through the concepts of representational significance, externalization and perceived coupling.

TUI Design

Motivate: The two primary sources of opportunity are the realisation of a technology and the identification of a user problem – these are mutually beneficial, in that technologies provide the basis for currently feasible solutions, and the search for problems suggests new ideas for future technologies.

Prototype: A valuable tool in the HCI designer’s toolkit, capable of addressing both dead metaphors and technology-driven design, is low-fidelity prototyping. Since most potential users of TUIs are unlikely to be familiar with the concept of ’tangible’ interfaces and post-WIMP interfaces in general, the use of tangible prototyping as a participatory design technique for envisioning future technologies is a valuable tool in tangible interaction design.

Analyse: The “TAC paradigm” is a way of analysing TUIs by systematically describing all of the relations between their physical and digital parts. In the resulting description language, the term “pyfo” is used to denote a physical object that takes part in a tangible interface, with each pyfo being a token, a constraint, or both. A token is then a pyfo that represents a variable, be it some digital information or a computational function, and a constraint is a pyfo that limits the behaviour of the token with which it is associated. The five key properties of TUIs from the perspective of the TAC paradigm are as follows:

1. Couple: a pyfo must be coupled with a variable in order to be considered a token.
2. Relative definition: each pyfo may be defined as a token, a constraint, or both.
3. Association: a new TAC is created when a token is physically associated with a constraint.
4. Computational interpretation: the physical manipulation of a TAC has computational interpretation.
5. Manipulation: each TAC can be manipulated discretely, continuously, or in both ways. The physical manipulation of a token is afforded by the physical properties of its constraints.

A more abstract classification of tangible interfaces is presented by Dourish (2001), segmenting the design space along two dimensions: actions–objects, i.e. to what extent the elements of the interface represent actions or objects; and iconic–symbolic, i.e. to what extent the physical representation has similarity with the digital thing it represents.

2.3 TUI System

The primary distinction is made based on the overall purpose of the TUI: as a direct-manipulation model, in which the effects of actions are experienced locally in both time and space; as a behavioral specification, in which the effects of actions are experienced at some future time; or as an communication channel, in which the effects of actions are experienced immediately at some remote location.

2.3.1 Tangible as direct-manipulation
Iconic: TUIs based on iconic models mimic physical situations for a variety of purposes.

Symbolic: Model-based TUIs need not be limited to existing physical scenarios, however: symbolism can be used to represent abstract concepts and the relationships between them. There are three general themes within this category: objects used to support the convenient organisation and access of information; objects used for real-time performance due to their interaction directness; and objects used for joint problem solving due to their visibility and malleability.

Symbolic objects for convenience: media management
Symbolic objects for convenience: control management
Symbolic objects for real-time performance: musical improvisation
Symbolic objects for real-time performance: narrative
Symbolic objects for real-time performance: other tasks
Symbolic objects for problem solving in abstract domains

2.3.2 Tangibles as Behavioral Specifications
“A further alternative is for the user to specify the structure of the required behavior, rather than directly specifying the required actions”. Such interfaces no longer provide direct manipulation, since the effects of changes are only visible once the ’specification’ is activated, and even then only to the partial extent that the prevailing input conditions determine one set of behaviors out of potentially many.

Iconic: The iconic specification of behaviour is akin to programming by demonstration, in that the resulting behaviours (the output effects) resemble their specification (the input actions), albeit at some future point in time.

Symbolic:
Symbolic specification in space
Symbolic specification over time

2.3.3 Tangibles as Communication Channels
The TUIs discussed up to this point have all been concerned with the production of effects local to that instance of the TUI, and either immediately in time after the interface action (in what I have called direct-manipulation models), or at some future point in time depending on ’program’ activation and input parameters (in what I have called behavioural specifications). A third alternative is for program effects to be observable immediately but at some other location, with TUIs being used as communication channels.

Iconic Channels: An iconic channel as one in which locally performed actions are mirrored in remotely executed effects.

Symbolic: Output effects need not always mirror input actions. In symbolic communication channels, the input actions (at one or more TUIs) are interpreted and executed as symbolic effects (at one or more TUIs).

TUI Evaluation

The ways in which TUIs are evaluated roughly fall into six categories – in terms of increasing formality, these are: casual observation by colleagues or visitors; presentation as interactive exhibits at shows or museums; observation-based user studies with mock tasks; observation-based user studies with real tasks; extended contextual deployment; and controlled experiment.

2.4.1 Controlled Experiments: The conclusion is that for collaborative optimisation tasks, actuated tangibles can encourage behaviour that is both more exploratory and efficiently goal-directed compared to tangibles alone, which in turn outperform screen and mouse solutions.

2.4.2 Extended Contextual Deployment: on observation and semi-structured ethnographic interview of the ten people – managers of a multi-disciplinary information technology research institute – to whom Nimio devices were deployed.

2.4.3 Studies Involving Real Tasks

2.4.4 Studies Involving Mock Tasks

2.4.5 Interactive Exhibits

2.4.6 Casual Observation/No Comment

3. The Analytic Design of TUIs

What is often missing from presentations of TUI systems is a description of how the researchers moved from the initial problem to the final design. In many cases, insight may be cited as the origin of design, or intuition as the method of selection between competing designs.

Thus, there is a need for a comprehensive yet concise formulation of the different aspects of design, along with the most appropriate conceptual and methodological tools that can be applied to the development of TUIs.

The author introduces an Analytic design methodology, a design process for developing any TUI system.

Overview of the Analytic Design Process for TUIs

1. Context analysis identifies the activities in a context that could benefit from TUI support– it refines a design context into a design opportunity;

2. Activity analysis describes the properties of a TUI that would appropriately support these activities – it refines a design opportunity into a design space;

3. Mapping analysis generates the physical-digital mappings of a TUI structure with these properties – it refines a design space into a structural design;

4. Meaning analysis provides these mappings with meaning that users can understand and adapt – it refines a structural design into a surface design;

5. Appropriation analysis considers the consequences of users adapting the intended meaning – it refines a surface design into a contextual design.

Context Analysis:

The application of context analysis to an activity domain results in a better understanding of how appropriate different forms of interface and interaction might be for supporting the activities of that domain.

Context analysis could be done through ethnography. Some of the “discount ethnography” techniques are: cultural probes, technology probes, and contextual design. Amongst these, Contextual design might appear to be the best compromise between in-depth ethnographic inquiry and design-oriented investigation, but it is still a relatively heavyweight method that requires many hours in the field.

a) Applications of Context Analysis:

The approach to context analysis is focused on user activities and a high-level decomposition of context into four distinct aspects that impact upon the situated accomplishment of these activities:
structural aspects, procedural aspects, cognitive aspects, and social aspects.

Structural Context
How are activities distributed across people, artifacts, and space?

Procedural Context
How are activities initiated, co-ordinated, and completed over time?

Cognitive Context
How do the cognitive demands of activities compare to the means of cognitive support?

Social Context
How do the social demands of activities compare to the means of social interaction?

b) Influence of Context Analysis

Directly, it determines the activities whose desired usability profiles will be specified by activity analysis.

Indirectly, it determines the mappings that will be judged as appropriate during mapping analysis. In addition, context analysis also determines what things are to be represented in meaning and appropriation analysis (which respectively determine how these things are or could be represented).

Activity Analysis

The purpose of activity analysis – the next stage of the analytic design process – is to describe the abstract properties of interfaces in a manner that allows them to be compared against both one another and the requirements of the context in which they would be deployed.