Charting past, present, and future research in ubiquitous computing

Twenty years ago, the idea of carrying the computing devices wherever we go might have been a crazy (imagine a desktop strapped to your back and a keyboard strapped to your gut). But today, a world without handheld devices is unimaginable. The paper I am blogging about, starts with ubiquitous computing ( the whole idea of having computing power whenever we want, wherever we want), proceeds to cover prior research conducted in this field, their pros and cons, and finally introduces everyday computing covering scalability via the dimension of time. It concludes by talking about social implications and methods of evaluation for ubiquitous computing in today’s world.

:::::::::::::::::::::::::::::::::::: INTRODUCTION ::::::::::::::::::::::::::::::::::::::
Weiser’s (the man who coined the term ubiquitous computing a.k.a ubicomp ) vision was to empower ubicomp devices with the implicit goal of assisting everyday life and not overwhelming it. He also envisioned new applications that would leverage off such devices and infrastructure. These incipient ideas have directed the research in ubicomp into three broad themes, (i) devices that use natural interfaces that support common forms of human expression (voice, gestures etc), (ii) context-aware devices that can adapt to changing physical environment & (iii) automated applications that can capture live moments and make them accessible anytime.

Orthogonal to these themes is scalability, which is inseparable from ubicomp. Weiser again categorizes scale w.r.t distribution in the physical world (portability), number of people (broad impact .. not the one we wrote in our project proposal) and time. This paper introduces a new term called everyday computing to address scale issues w.r.t time. We will cover the three major themes first and then move to everyday computing.

COMPUTING WITH NATURAL DEVICES

For the most part voice, hand gestures and writing were the primary streams of research in this theme. Though a novel idea, there still a need for improvement to the devices. A multitude of challenges arise when we try to design devices that can take such a free-form input.

First-Class data types: These free forms of data bring a new challenge — How do you take them as input? Standardizing support for free-form is largely unexplored. In case of pen-based interaction, we generally assume pen-ink, converted to text, is the data, but class-notes taken by a student doesn’t necessarily need text conversion. Instead of blind text conversion of notes, we can try to link annotations written with a pen with the audio/video of what was said or seen at that same time during a lecture.

Error-prone interaction: Inputs from recognition-based interfaces, such as voice, pen-ink, etc. are prone to a variety of errors. Three primary research areas have emerged to address Error handling of recognition-based. (i) Research aimed at improving recognition technology in order to eliminate or reduce errors. (ii) Informing error through explicit user input, and can help the user to discover errors through techniques like confidence measurement, statistics, and rule-specification. (iii) Create a toolkit that presents a library of error-handling techniques of recognition-based input.

CONTEXT-AWARE COMPUTING

Our ubicomp devices must leverage a simple piece of context, user location, and provide apt services. Location is used as context in applications such as GPS-based car navigation, handheld “tour guide” systems. Recognizing Individual objects (like barcodes) is another piece of context. To explain what is “context” in a deeper level, the paper gives a definition using five W’s: Who (user, spectators ), What (device, associated objects), When (time of use), Where ( location) and Why (purpose of device). Not only this context must be identified, but they must also be represented clearly to enable hassle-free usage.

Context Fusion: Having a certain context, is useful. However, there are few truly ubiquitous context services. For instance, positioning services like GPS may not work indoors or can be ambiguous, even in urban regions. The solution is to assemble context information from a combination of related context services. Possible challenges include a seamless transfer or sensing/info between borders of different context services.

Context Awareness + Natural interaction — Augmented Reality: Many context-aware applications try to give the user, real-time information based on actions in the physical world. A tour guide systems use the user’s movements in an exhibit, to display more context-sensitive information. By incorporating augmented vision and augmented hearing displays, we will more closely
integrate context-aware interaction with the physical world.

AUTOMATED CAPTURE AND ACCESS FOR LIFE EXPERIENCES

Tools to support automated capture of and access to live experiences can remove the burden of doing something humans are not good at (i.e., recording) so that they can focus attention on activities they are good at (i.e., indicating relationships, summarizing, and interpreting) — particularly for meetings/classroom environments. Prior works in this area include — Capturing phone conversations and then providing access to the content of the conversations at a later time; Time stamping artifacts produced by free form ink on electronic whiteboards in classrooms. This time-stamped data segments, are then connected with the corresponding audio/video snippet. (For the sake of brevity I am skipping a few) Emphasis on Ubiquity is clearly seen in all these methods, with their separate capture and access phases. Electronic capture is moved away from traditional devices, like the keyboard, and brought closer to the user in the form of natural interfaces.
This theme is so unconventionally elegant, an entire section in the paper is dedicated to explaining potential to challenges in this field. First, let’s see the challenges associated with Capture and then the issues faced with Access.

Challenges in Capture : The above example of classroom notes shows how access can be a vital component. But its also hard to assure everlasting availability. At times devices may not be within reach to capture an ephemeral moment. Like fumbling to find your phone to capture a shooting star.

Technical exchanges often flow freely in opportunistic encounters. Even in formal meetings, exchange of information spans across multiple artifacts like sticky notes, whiteboards, sketch pads, etc and are rarely captured or documented in its full essence.

The whole activity can sometimes be recorded and used as an archive to refer at a later point in future — but it’s hard and time-consuming. Take constructions for example. Assume we have a recording of the actual construction of the building — in all sense this is a standing archive. In case of a technical failure, the appropriate technician could “replay” the construction as see where things went wrong.

Challenges in Access: In the access phase, we must ensure multiple forms of
playback capabilities. Playback in real time, not always efficient — are often situations in which this is inappropriate or overly inefficient. A student
reviewing a lecture for an exam, wouldn’t want to see the entire lecture again. He/She would just need a specific part of the lecture or an overall summarization.

Synchronization of multiple captured streams during playback is vital. In the above example of the meeting room, it would be quite confusing if all the recorded info, from sticky notes, sketches and free form ink are just dumped at the user’s face without a coherent connection between them.

In many cases annotating or revising the captured material is appropriate. Multiple revisions when stacked properly can show the user a chronological evolution of the ideas. Challenge here is to create such an interface that can clearly show multiple versions. Especially tough when the data is already timestamped(like in the previous case of time-stamped Lecture notes).

TOWARDS EVERYDAY COMPUTING

Everyday computing refers to extending computing services to day-to-day activities that are not only ubiquitous w.r.t time but also loosely defined in terms of how and why they are done — interacting with family, managing information and other activities that are unstructured and informal. Designing for everyday computing requires addressing these features of informal, daily activities. They rarely have a clear beginning or end ; Interruption is expected ; Multiple activities happen concurrently ; Since these events interpret and act of real-world events , time is an important discriminator ; Associating and context-rich models work well with these tasks rather than hierarchical models — this is analogous to saying “have a collage-based structure with interconnecting threads rather than a folder based structure, creating folders within folder to denote hierarchy.

Much prior work has been done to ensure synergy among the three themes of ubicomp when it comes to everyday computing. Though developed independently they tackle the challenges faced in individual themes and how to make devices robust in integrating all three of them

Research directions in Everyday Computing

Continuously present computer interface: Concepts like information appliance (passive devices used only for information storage) are being developed into more interactive and human-centered ideas to develop anthropomorphized agents that interact, understand and personalize. But we are yet to reach satisfactory levels.

Present information for different levels of periphery of human attention: A fancy way of saying “make my smartwatch to give a beep for every 100 calories I burn”. In many cases, information from peripheral devices lack the means to be moved to foreground attention. This needs to be addressed, giving either the user or environment the ability to push info to the foreground.

Connecting events in the physical and virtual worlds: My emails and social media is a different world from my breakfast joint and gym. Despite efforts as early as 1993 there is much work left to be done to understand how to combine information between info from virtual world with preferences in the physical world

Modify methods to support informal and peripheral context-dependent opportunistic behavior: Combining information from methods as different as laboratory experiments and ethnographic observations are far from simple. Each method has its own evaluation strategy, and the primary challenges are — learn how these methods inform each other, how their results can be combined and how to use it iteratively to enhance the development of the device.

ADDITIONAL CHALLENGES FOR UBICOMP

Evaluation of ubicomp systems

Except for the Tivoli system, not much work has been done in this area. ( I strongly suggest to read this paper, especially the “Design Features and Issues” section) The primary issue is that traditional evaluation methods like Formative and Summative evaluations don’t work with Ubicomp. Even with Tivoli, the use case is very specific to meetings and may not be applicable to other real-life scenarios.

Finding a User need: We need to know the user’s needs before we commit to building the ubicomp system. For the most part, this processing of knowing the user’s needs is a gradual process that evolves. Hence development must be closely tracked with requirement gathering. We look into two specific case studies where attempts were made to determine how a system is being used, what kinds of activities they are used for.

Xerox PARC Flatland: The designers used ethnographic observations of whiteboard use, coupled with questionnaires and interviews. The qualitative data gathered this way helped in design — for instance, active areas of use on the board (aka “hotspots”) move around and the pattern of movement was inferred.

Audio Aura: Sometimes the affordances and usability may not be easy to infer or difficult to use. In this case, the developers tried to explore how peripheral awareness of activities be enhanced through the use of ambient sound. The combination of active badges, wireless headphones was too clunky and the headphones appearance was less appealing. Audio experience was hampered due to the rudimentary nature of the language used.

Evaluating in context of authentic use: The scaling dimensions of ubicomp systems — device, space, people, or time, make it impossible to use usability labs. Evaluation must be done on realistic deployment into an environment of expected use.

Classroom 2000: A project aimed at making a system that can capture most of the classroom experience. Its suppose to make the capture better so that students can experience the lecture in addition to taking notes. They tried to reintroduce student note-taking units that can personalize the capture experience and also encourage better note-taking practices. — After 6 months of close study on 100 courses with 30 different instructors.

Avoid Task-centric evaluation techniques: The majority of usability techniques are task-centric. If the user’s tasks are known, then evaluation can be performed to determine the fitness of the system and its interface. But in most of these ubicomp settings, tasks are loosely defined hence traditional task-centric measures that use usability testing, fail to provide a valid evaluation.

Social Issues of Ubiquitous Computing

A basic concern about any information stored in a computer is knowing
who can access and modify the contents? Where are the bits ?. Wearable
computing partly counter this — providing security by keeping
the bits local (on the body) and removing the risks of transporting them
over a public network.

Another fear of users is the lack of knowledge of what computing
system is doing, or that something is being done “behind their backs.”
To counter this design systems can be put in place that makes access information transparent.

It is also important to allow those being sensed or recorded to have control to
either stop the recording or to at least control the distribution of pertinent
information. This arises in the design of collaborative environments where the actions and roles of collaborators are fluid and difficult to articulate in a single snapshot.

Although issues surrounding the appropriate use and dissemination of
information is quite old, specific concerns stem from ubicomp making personal and sensitive information more generally available. The fact that computers can easily track our daily activities — a feat that previously required a large amount of human effort — is disconcerting at the least. Hence partly these concerns exist in the minds of the users as well.

In general, social and legal practices continue to evolve in concert with
technological and design innovations. In each situation, people will compare
the perceived benefits and costs of the uses of ubicomp technologies. The paper ends this discussion in a positive note saying better methods are being worked upon

:::::::::::::::::::::::::::::::::::::: CONCLUSION ::::::::::::::::::::::::::::::::::::::::
Weiser claims that the whole point of ubiquitous computing was
to create compelling applications that would drive the development of
devices and infrastructure. But the author campaigns for a broader view that promotes the general-purpose utility of ubiquitous interaction with computational resources. Although application-centric or task-centric focus works for individual devices and unilateral interactions, it fails for a system distributed over time and place. The real goal for ubicomp is to provide many single-activity interactions that together promote a unified and continuous interaction between humans and computational services.

::::::::::::::::::::::::::::::: PERSONAL REFLECTIONS ::::::::::::::::::::::::::::::::
A six-page long Blog post is overkill for any assignment. But this reading made me do it, due to it’s novel applicability to Human-Centered design. Though the paper deals mainly with Computing devices, themes such as natural language and context awareness are very much applicable for any usable device. It makes so much sense to think of devices and games in this way — as entities that interact with humans. My project is not necessarily related to any user devices, but still, I enjoyed spending time on reading this paper and Tivoli system out of sheer interest. Apart from devices, I do see scope for including these ideas in Software and applications as well. Especially context awareness can make software very robust and tend to user’s changing needs.

::::::::::::::::::::::: CONTEXTUALIZE AUTHORS AND TIME ::::::::::::::::::::::
The authors of this paper Abowd and Elizabeth D. Mynatt were great authors. But my attention was caught by the person who coined the term ubicomp. Like many other big names covered in class, Mark Weiser also worked at PARC (by worked I mean he was a chief scientist there). A better-known fact is that the Mark Weiser Award (given by ACM SIGOPS) is named after him. Another memorable name is the Tivoli system, which again was developed by PARC. This is an excellent collection of ideas for the design of an electronic whiteboard. I have added the link for the paper in the blog for interested readers. I didn’t find any significant context on time though since these were relatively new compared to Norman’s works but significantly predates context aware systems like smart phones and iPads.

:::::::::::::::::::::::::::::::: INSIGHTFUL SYNTHESIS ::::::::::::::::::::::::::::::::
Context-Awareness is one of the most interesting topics covered in this paper. Not surprisingly much prior research have been done in this field. Dey and Abowd 2001, talks extensively about context-awareness among devices. Section 3 in their paper deals specifically about requirements and frameworks for handling context, which gives some interesting facts on how existing systems need to adapted to store context information. An interesting correlation is how much of the content in this paper is inline with our understanding of Location awareness from Eija Kaasinen’s work. In both works we see emphasis is placed on reliability and how well they correspond to locations/events in real world.