A Designer’s Checklist for Working with Ubiquitous Computing

Technology is always present in our lives.

The amount of technology we use in our everyday lives has been growing fast, and nowadays technology is present in all aspects of our lives. Although one would assume that technology is created to make our lives easier and thus us happier, the average life satisfaction has remained more or less at the same level for over 50 years. So, technology isn’t making us happier. At least not directly, and why is that?

The effect of ubiquitous technology and social media on our everyday lives has been actively discussed in recent years. Having our mobile phones always with us allows us to connect with others and build relationships, but studies have also shown that extensive use of mobile phones and social media have negative effects on mental and physical health.

Not only academics but also other influencers in the tech scene, like the Center for Humane Technology, have been trying to raise awareness of how technology steals our attention. Recently, also Silicon Valley has started addressing the issues of ubiquitous technology and social media. This summer, Facebook and Instagram rolled out features to control the time users spend using Facebook and Instagram apps. Apple released their Screen Time app with the iOS 12 release in September, while Google announced their “digital well-being” features in this year’s Google I/O.

How do we avoid problems with ubiquitous technology?

Ubiquitous computing is not inherently bad, in fact it opens doors to more intuitive and calm products and services. One of the reasons why ubiquitous computing has caused problems is the designers’ insufficient understanding of the influence of the ever-present technology on human mind and behaviour. For example, the use of graphical user interfaces may not always be the best solution, such as is discussed by Golden Krishna in his book “No Interface is the Best Interface”.

To avoid creating ubiquitous products and services that cause problems, we need to follow a user-centered design process and test our products with real users in real contexts. It seems likely that insufficient knowledge of the effects of some traits of ubiquitous products and services, like their omnipresence or invisibility, is causing challenges. The deeper technology will be embedded into our lives, the more prominent these challenges will be.

If we could avoid the typical challenges with ubiquitous products and services, we could create better products with a lower investment, as fewer iterations would be needed during the design process. Therefore, I have created a set of design principles that help us avoid these problems and create ubiquitous products and services with a better user experience. Specifically, the design principles are focused on user interfaces. Following these principles will not only create a better user experience, but will also guide us towards a healthier relationship with technology. With the speed that new technology is being embedded in our surroundings, it will be more and more important to follow these principles.

Defining a Common Goal

Before we take a look at the design principles, let’s define the type of user interfaces that are appropriate for the new ubiquitous products and services. Since GUIs are not always the optimal UIs for ubiquitous products that could for example leverage several modalities and new sensor technologies for invisible and implicit UIs, we need a better UI standard to aim for. I suggest so-called natural user interfaces (NUIs) to be the next standard of UIs that we as designers should pursue.

Natural user interfaces form a design philosophy and source of metrics that enable iterative product development. One of the first companies to define and put the term into use was Microsoft.

The word natural does not describe the properties of the product, but instead the behaviour and feeling of the user while using a product or service. Using a product with a NUI is effortless and graceful, which in the end results in happy users. In general, a NUI leverages input and output technologies to mirror human capabilities, optimize progress from novice to expert, work with given contexts and tasks, and fulfil user needs.

NUIs can be seen as the next evolution of user interfaces, as they

  • leverage new technology to further reduce the load that graphical user interfaces create on users
  • support user tasks even better
  • are not bound to the traditional WIMP interaction style, for example.

The Design Principles

I created a set of 35 design principles based on the state of art literature to help us design ubiquitous products or services with better UIs. With the help of the design principles, designers can acquire a quick overview on what is important when designing novel ubiquitous computing applications that feel as natural as possible. For easier readability, the 35 principles are divided into topic areas, to each of which I’ll give a short introduction and an example. The examples are merely to help you to grasp the idea of each principle, and thus they ignore many important design aspects. Therefore, please do not consider the technical feasibility of the examples.

General Interaction Characteristics of Natural User Interfaces

The first six principles describe general features that the interaction with products or services should have. They are not specific to ubiquitous products or services, but are a foundation for any good user experience.

Designers should design interactions that…
1. fulfil user needs.
2. are close to natural human behaviour.
3. are adapted to human capabilities.
4. optimise progress from novice to expert.
5. are easy to learn, but not necessarily guessable.
6. are standardised, if possible.
In the future the surgeon will be able to use several modalities to control the patient monitoring system.

Let’s imagine an emergency room with a patient monitoring system that can be controlled by the surgeons by hand gestures and natural language, without having to touch any surfaces.

In this example, a surgeon needs to successfully interact with the system during the various phases of an operation to get the information needed and to control the system as desired. The system takes into account the user’s contextual capabilities by allowing commands with natural language, if hands can’t be used. Similarly, when it is not natural or easy to describe the desired functions with natural language, hand gestures can be used. For example, hand gestures could be used when zooming into a specific part of an image.

For the hand gestures to work in a stressful environment, they need to be easy to learn, and preferably known from other systems, such as pinch-to-zoom gestures. Simultaneously, the system can guide the users on their way to becoming an expert user, by for example suggesting hand gestures to be used for various purposes, when the user is hesitating to give a command.

Understanding Context and Designing for Implicit Interaction

In order not to overload users, ubiquitous services and products need to be designed for the context of use to automatically support the natural behaviour of the users, and not only to execute individual tasks. This requires a deep understanding of the users, which can be acquired through methods such as ethnographic studies and observations. Being embedded into our everyday surroundings, ubiquitous computing also has social implications.

Designers should understand the users and context of use so that the product or service…
7. supports activities instead of tasks.
8. can use implicit interaction to automate or simplify tasks, while simultaneously preserving transparency without risking ethics or privacy.
9. is designed for the users and tested in that context.

Think of a service for elderly people diagnosed with Alzheimer’s or dementia that monitors their physical activity and mental state through sensors in clothing and through analysis of their speech and tone of voice. The service could automatically create reports of the state of the patient to help a doctor. In addition, the service could predict when the user is about to get a seizure to alarm caregivers.

In this example the service reduces the amount of regular check up tasks a doctor needs to perform and thus gives more time to focus on the health of the patient. The patient does not have to do anything but (s)he implicitly interacts with the system. However, it’s important that all stakeholders know what data is collected and who has access to it, and that nothing is done without the consent of the patient or his/her trustee. Naturally, the service is designed for a very specific user group, namely dementia patients, and thus needs to be designed for them and tested in their living environment.

Metaphors and Mental Models

Products or services leveraging ubiquitous computing can suffer from invisibility, which is why metaphors should be used to describe the product or service to the users and to create a correct mental model for the users. The metaphors should preferably be well-known to the users from other products, services or contexts.

Designers should use suitable metaphors…
10. to address problems related to complexity and non-transparency of the product or service.
11. that users are familiar with, if possible.
12. to maintain simplicity and controllability.
13. that are in line with how the functionality of the product or service is distributed between different touchpoints, with the value proposition of the product or service, and with the users’ mental model of the product or service.
A digital referee system could be completely hidden from the users, which makes it harder to understand how the system works.

Think about playing on a basketball court that automatically monitors the game with cameras and other sensors on and around the court and in the ball. The system will make all the calls that a referee normally does, from simple throw-ins to fouls and announces them through loudspeakers and on a display.

Calling this system a “digital referee” makes it easy to grasp what the system is doing without having to explain what is technically needed to make the system functional. This metaphor also creates a mental model for the users, who will expect that the system works like a human referee. This way technical complexity is hidden and the users get a simplified view of how the system works.

Visibility and User Guidance

To feel natural, a product or service should have a user interface that gives proper feedback to the users and guides them with scaffolding. As ubiquitous computing applications embedded in the everyday surroundings may be hard for users to see and understand, discoverability of the product or service is important. Especially gesture-based interfaces require proper feedback and scaffolding, and would benefit from standardisation.

Designers should improve understandability by…
14. giving appropriate and immediate feedback to each action.
15. using a distinct response to every source of error, so that the user understands and sees the connection between cause and effect.
16. breaking down challenges to smaller ones.
17. communicating possible actions and how to invoke them.
18. providing support if something goes wrong.
19. providing the users with the possibility to explore the functionality of the product or service.
Lip reading will improve the speech recognition in at a noisy train station.

Let’s think about a speaking robot at a train station that leverages several modalities, such as speech recognition and lip reading with camera.

To give feedback, the robot should always tell how it understood the input from the user, so that the user knows if the system did not understand something correctly. In case the robot doesn’t understand the user, an error with an explanation should be displayed to the user. To break down the challenges, the user can first find a suitable train connection, then confirm the connection, and finally buy a ticket.

As the user can’t know how much natural language the robot understands, clear instructions of commands that can be used should be provided. In addition, instructions should provide an overview of the functionality of the robot, otherwise it may be hard to discover what the robot is capable of doing. Finally, if something goes wrong, the robot can call for a real person for help.

Calm Technology

Due to the physical presence of the product or service, it is important that designers design calm products or services that do not unnecessarily draw the attention of the users, and remain otherwise in the periphery of the attention.

Designers should reduce the required attention…
20. by requiring little cognitive effort once usage is learned.
21. by leveraging right modalities for intuitive and calm interaction.
22. by pursuing a design that is glanceable and requires, as far as possible, the periphery of user’s attention only.
23. does not interrupt social interactions.
24. and provide a clear overall picture of the product or service to the user.

If not already, your home will soon be full of technology. You will have smart lighting, heating and security systems and various connected appliances, such as kitchen equipment, entertainment systems, or even connected beds that measure your sleep.

Due to the amount of different products, only little cognitive effort should be required by each of them, or your daily chores using the devices will be very exhausting. If all the products would use sound to notify you, it would be annoying and hard to distinguish between the devices. Therefore the devices should use different modalities that do not draw the user’s full attention, such as haptic feedback. Similarly, to make technology work for humans, and not the opposite, the products should never call for the user’s attention or interrupt social interactions, unless absolutely required. For example, a kettle only needs to notify the user when the water is boiling. All connected products should blend in the background and provide a clear overall picture of the system to the user.

Designing with Data

Ubiquitous computing applications have typically access to vast amounts of data. It is, however, important to show only actionable and meaningful data to the users so that the product or service remains calm.

Designers should display data so that…
25. only meaningful data is presented.
26. it is actionable to the users.

In the future, a service on your mobile phone will act as your personal doctor and life coach. It will monitor many factors of your life, such as sleep, stress, nutrition, physical activity, or even emotional states through speech analysis.

The amount of data collected will be huge, but only a handful of it will be interesting to you. An understanding of the users’ motivations and goals are needed to form the right insights based on the data, for example if the user is trying to lose weight. Also, right context should be available with the data and insights to make the information personally meaningful. For example, personal data can be compared with average data of the same demographic group. Actionable data in case weight watching, for example, could include various lifestyle suggestions regarding nutrition and sleeping, for example.

Physical Appearance of Products and Services

It’s important to consider the physical appearance of the product or service leveraging ubiquitous computing, as interactions will increasingly be focused on physical objects and environments.

Designers should create smart objects that…
27. communicate their functionality through their appearance and encourage natural behaviour.
28. have enhanced affordance through technology.
29. are still usable if technology fails.
30. are interoperable with other products or services.
31. have direct user interfaces embedded on the product, instead of using another device, such as a mobile phone, to control the product.
32. are produced in a sustainable and ecological manner.

Think about smart sunglasses that display relevant information depending on your activity, such as route instructions and tourist information when walking in a new city.

The physical appearance of the sunglasses encourages the user to wear them like regular sunglasses, and a simple animation on the lenses could tell the user that the the glasses are capable of more than regular sunglasses. Naturally, if the battery dies out the sunglasses still function like regular sunglasses.

Through programming interfaces the sunglasses can communicate with other systems, such as a visitor system in a museum visited. The sunglasses can be switched on from a button on the frame and all other required settings can also be available directly from a menu that the sunglasses display for the user.

Privacy and Ethics

Ubiquitous computing is embedded into our everyday lives, which means that the product or service often has access to personal data. Generally, if users do not fathom, there is a privacy problem. Users should have control over the product or service to manage privacy related aspects, but it is also the designers’ responsibility to take privacy into account, by for example deciding how user data is handled. Users should also be made aware of the presence of products or services leveraging ubiquitous computing.

Designers should create ethical products and services by…
33. protecting the privacy of users.
34. informing users which parties can access user-specific data and how that data is processed.
35. giving users the freedom to manipulate the product or service and to influence the level of privacy.

The privacy of the user should always be guarded and no data should be shared anywhere without a consent. In addition to informing the user who can access their data and how it is processed, every system should give the users the possibility to decide what data is shared and with whom. Generally, following the GDPR guidelines is a good starting point.


A Checklist for Designers

I wanted to keep the principles as a concise checklist, as a so-called cheat sheet that we can quickly check to get an initial understanding of the topic. Naturally, if you find a certain topic area especially interesting, there is a lot more to read about each of them. Even though the design principles focus on areas that are challenging in case of ubiquitous computing, they are not only applicable to products or services leveraging ubiquitous computing, but also to many products leveraging emerging technologies as well as more traditional technologies.

I’m continuing my work to improve the design principles, so if you have any questions or improvement ideas, please let me know!