Designing Smart Experiences
The smart and invisible future of interactions and services.
A new generation of products and services based on smart technologies is about to change the world around us. Connected objects, smart assistants, sensors, beacons, big data and artificial intelligence are revolutionising not only many industries and business sectors but also the way we will interact with an emerging world of smart devices and services.
This new generation of services, products, applications and tools will deliver to the user a new kind of experience: a “smart user experience”.
What is a “smart user experience”? Briefly, it is a new class of experiences based on smart technologies: ”technologies with the ability to sense changes in their circumstances and execute measure to enhance their functionality under new circumstances.” (1)
A smart user experience is enhanced by sensors, connected things, AI algorithms and any other technology, that is able to listen, learn and adapt automatically in real time following the users behaviour, their context and any other specific condition and rule, that influence the service delivery and outcome and, eventually, the experience itself.
When products and services get digital, get smart.
As soon as a product or a service combines digital technologies, sensors and network connection capabilities, is ready to get “smart”. As Michael Porter writes, this is the “third wave of the IT-Driven competition”, when IT and connectivity becomes integral part of the product itself (2), expanding opportunities for new functionality and capabilities that cut across and transcend traditional product boundaries. The ongoing rapid decline in the cost of sensors, combined with the similar decline of the cost of computing power, opens up many opportunities for new consumer applications of the Internet of things, changing drastically many product experiences. Almost any object could embed and integrate various sensors and get connected.
The more this technology is embedded everywhere in consumer products, mobile devices and common things, the more it becomes invisibile and deeply intertwined with our everyday experience, breaking any barrier between digital and physical domains. In the words of Eric Schmidt (3):
“The internet will disappear to become part of our presence all the time.”
Actually Schmidt recalls Mark Weiser (4): as technology matures, getting pervasive and connecting everything, we will experience an highly personalised and interactive world.
Converting the massive volume of data collected by sensors and mobile devices into useful results and applications is still a challenge. This is dealt with in recent developments in artificial intelligence and remote computing. Rendered by data and sophisticated software algorithms, many working “Narrow Artificial Intelligence” (also called “Weak AI”, a very specialised set of instructions focused on specific tasks) are already part of a wide array of consumer and industrial services and applications, enabling a new generation of tailored and improved services and experiences.
We are just at the beginning, most of the first commercial applications are still limited and partial, but the list of services and products enhanced by some sort of software intelligence is long and is growing every day: from Nest to Google Now and Waze. Smart healthcare systems that track and assist patients in managing their therapy and new testing systems that help doctors to decide whether to perform surgery on patients. Automatic personal financial advisors, “robo-advisors” that manage investment portfolios with a little human supervision. Agriculture tools that analyse soil hydration and environmental factors, revealing trouble spots or opportunities for better crop management. Self-driving cars, connected vehicles, and smart energy management systems. Besides many mobile smart apps and personal assistants that sort and prioritise automatically email messages, appointments and suggest what you might want to buy or the movie you may want to watch this evening.
There are massive potential innovation opportunities for many different industries and business sectors: transportation, healthcare, financial services, education, retail, entertainment, consumer products, public services and so on. A wide range of human activities and tasks will be innovated and automated by intelligent systems capable to collect, process and organise data and information better and faster.
Next generation of services will be smart.
The impact of the smart technologies innovation is twofold: they change the way services and products are conceived and delivered, as well as the way users interact with those services and products.
The bar of the relationship with customers is about to rise again. If the current challenge for many companies is to set up an integrated and consistent digital-physical platform for their customers across channels and touchpoints, the next challenge will be delivering a smart user experience.
To remain competitive, companies will have to rethink the customer relationship building an interactive platform able to change and adapt itself continuously, in real-time, following and engaging the customer in new ways.
When everything is connected and interactive, a service can be everywhere, anytime, always ready and available for the customer, regardless of channels, physical locations and time of the day.
Digital banking and ecommerce are excellent examples of this transformation and of its consequences on the service experience. Users simply don’t think about channels, locations and time of the day: they just access the service opportunistically, when they want, anywhere, with the most convenient devices available in that moment. For a service being online and accessible by a mobile device is just the first step, a natural and basic functional condition. Now users ask for a simpler, more relevant and personalised service. A smart service.
Many traditional objects and spaces, like retail and work spaces, are about to get alive and interactive, capable to detect, identify and react to the user presence, delivering contextual and personalised content and services on smartphones and other connected devices. In this hyperconnected scenario, user centrality is more important than ever: being able to follow and support people everywhere continuously and consistently will be a crucial success factor for the next generation of service experiences.
Towards new UX challenges
What are the key characteristics of a smart experience? Most of them are inherited from the technologies and the platforms used. Some points are not completely new for designers but get new implications in the smart services scenario. New opportunities and challenges are unfolding for user experience designers, calling into question priorities and interaction logics far from conventional screen-based interactions and graphic user interfaces. The following is an intial and brief overview and analysis.
The platform is the experience.
The more objects are connected the more the platform and the applications available on the platform become the center of the user experience. In the “internet of everything” scenario any individual device, sensor and piece of hardware is just a resource of a distributed technical platform, often invisible (and sometimes negligible) for the user. Each connected device and every microinteraction it enables, is part of a technical ecosystem that provides the hardware platform for a software platform, which in turn enables many service solutions. Think about a connected light bulb: it is just a component of a service that is the only thing that matters for the user. The value is not delivered by the single connected object but by the software platform that combines everything (usally elsewhere, in the cloud) and that provides the user an appropriate interface to interact with the platform consistently across touchpoints and moments in time.
The ecosystem configuration can be very open and variable: some tools collect the data, some process the data and other devices provide the interface for the user. Any data collected at any point of the system becomes useful only when it is combined and processed by the software platform to deliver a specific result. The same connected gadgets and data can be used by different platforms with different purposes to deliver different solutions. The user value lies in the distributed smart service and its outcomes, not in the single smart devices.
Physical and digital don’t matter.
More than ever, a smart experience combines digital and physical worlds seamlessly. The underlying interactive ecosystem is an intertwined multi-device systems by definition, that combines different interaction models and inputs devices. Beyond traditional devices, the “internet of everything” includes potentially every connected device, sensor and thing (wearables, connected cars, implantable devices, accessories and so on). As a consequence, the distinction between digital services and real world touchpoints becomes less and less significant.
Ensuring the appropriate service outcome everywhere, anytime and its relevance for the context of the interaction, is critical for the relation with the user and one of the key problem for UX designers. Also a physical connected gadget and its features should be designed considering the goals and the features of the service platform it is supposed to belong, not the other way round.
Invisible interactions.
The more smart connected objects and services blend into the background of our every day life and in the world around us, the more interactions are going to be hidden, pervasive, implicit and automatic. Eventually out of the user control. In a world where sensors, interactive screens and devices are literally everywhere, most of the critical data exchange and interaction with the system that delivers the smart service is going to be less and less visible and direct. With ambient intelligence common behaviours and actions are tracked by multiple gadgets and sensors and become a source of data and triggers of many possible interactions. Crossing a doorway, for example, or walking near an object can activate various reactions, processes and feedback. Moreover the same action in different moments of the day or of the week, can activate different processes and deliver different outcomes.
The interaction with a system can be triggered by many different events, microinteractions, data and hidden controls, even without notice or a visibile feedback for the user. Multiple types of input devices can be distributed everywhere and combined together, creating thousands of interfaces of any sort, very different from the usual computer-based interaction paradigms. The triggering events of a smart experience are potentially unlimited. This is one of the key difference between traditional interaction design and designing for smart services: “Without a mouse and keyboard the new input is the user’s behavior itself” (5). Designers will have to rethink the human-system interaction beyond traditional screens, graphical interfaces and common input controls. Also in terms of usability, it’s not important to consider just the usability of individual UIs and connected objects but interusability: the distributed user experience across multiple devices (6).
Highly personalised, nearly predictive experiences.
A smart application delivers only what the user needs in a specific situation, following and anticipating tasks and expectations. Personalisation is not a new topic for designers and UX strategists, but in this case the experience goes beyond content and data to touch features and processes. Regardless the device or the output method, similar user actions and interactions can deliver different dynamic “smart selections” or “reactions” depending on various “smart conditions” managed by the system. Location, time of the day, user preferences, interaction history, weather and traffic condition, social connections and any other data and rules can affect the outcome of a smart service and the way it is delivered to the user.
Less, smaller and simpler interfaces.
This point is perfectly described by Rehabstudio as “more learning, less interface”: the more a system can learn from the user behaviour and adapt accordingly, the less an interface is necessary or a UI can be simpler and smaller (7). Making the most of cloud services, deep learning and remote intelligent algorithms, a smart service combines data and reduces automatically the possible options, showing to the user only the content and the features that are useful or relevant for that specific context and situation (for example a route to follow, a number to call, a set of content, a notification, a set of features and so on).
With smart services, live notifications, smart suggestions and dynamic recommendations are key moments of the user experience.
Users do not have to search or browse an application menu to find what they need, but they simply choose within a smart selection of the best combination of content and features related to their specific situation, prepared automatically by the intelligent system. In many situations the experience will be based on live and dynamic notifications, suggestions and recommendations delivered to the user on the spot at the right time. This is the use case of smart watches, for example, and of other small devices with small screens or limited interaction capabilities (wearables, smart home connected objects and so on).
The interaction is a conversation.
Sometimes, the interaction with the user often is initiated automatically by the service itself, with a notification or an highly personalised screen. This is more than natural interaction, gesture or voice interfaces: it’s a new set of possibilities, logic, constraints, affordances and issues for designers. If a system or a device is given the possibility to make choices, take autonomous decisions and be active part of the interaction with the user, UX designers will have to deal with new challenges (8), both for the user interface as well as for the overall user experience strategy, content distribution and service organisation.
Remote user interfaces.
In the everything connected scenario, user interfaces and smart devices are often decoupled and the experience is based on remote interactions, or on a combination of many types of interactions. Different devices and screens get combined and the experience is distributed: a device is used to control another and viceversa. A smart bulb is again a good example of this situation: a remote control application can manage different connected devices contemporarily often applying conditional rules. These kind of interactions can be based on proximity and near-field communication (for example, interacting with a product while standing in front of the product itself), or remote interactions (for example interacting with a system over the internet).
Many smartphone apps are the main interface for a wide array of connected objects and devices that have no UI at all, making our smartphone into a remote control for pretty much everything. Actually this is also one of the main issue of many smart platforms: if you need a smartphone (and an app) everytime you have to turn off the lights or change the temperature of the room, the experience can get cumbersome pretty quickly. As mentioned earlier, simple tasks are better managed with small and simple interfaces.
The result is provided combining the features of many different devices.
The user interface of smart service usually does not provide access to every single connected thing or sensor. The service result is a smart combination of the features of different devices. For example, many smart-home applications are based on “scenarios”: the user manages various scenarios (a combination of light condition, temperature and music — just to give a very basic example) and a software from an hub manages the network of connected devices to provide the desired result, dimming down the light, changing the temperature, managing music or controlling any other domestic appliances accordingly.
The “scenario” concept is pretty common in smart services: it includes conditions, rules and variables that are managed dynamically by the system itself, learning and adapting continuosly the outcomes following contextual situations and user behaviours. Scenarios (recipes, combinations and configurations) can be easily shared with other users of the system making the overall experience more engaging and social.
Provide a smart starting point.
In a smart service users don’t start the experience from a blank page. They start their experience from a predefined set of scenarios which evolve with the user interaction. Actually also this feature seem not very new: providing a set of predefined templates is a common solution of many digital applications and services. The difference lies in the continuous automatic adaptation of the scenarios: any predefined configuration provided by the system is not supposed to be fixed and closed, it should be a dynamic and smart configuration, capable to adapt and update itself automatically and continuously by following the interaction with the user.
The more the user interacts with the smart system, the more the system learns and adapts its configuration accordingly.
A good example is ‘The Grid’, an AI-driven system to generate websites, where the layout and the look and feel of the page adapts automatically to the content inserted by the user. In general, a smart system should be able to make the most user data and contextual information to make the first use of the service even more simple, personalised and relevant for the user.
Keep the user always in control.
Data, settings and preferences management is a central part of a smart experience. How user data and preferences are collected and used, and how the user is able to control and manage these data and processes? This is one of the most important and controversial point of a smart user experience. One implication of a smart services is the lack of direct control: outcomes and interactions are managed automatically by a system, the user can change the inputs, but the relationship between inputs and outputs is non-linear, complex to learn and the user may not be able to repeat the same conditions twice. This goes completely against any traditional HCI approach that requires a simple mental model that enables a user to predict the reaction of a system to his actions and to be able to repeat it consistently.
If data and preferences are crucial for smart service outcomes, the user should have the control on the inputs and the outputs of the system, and be able to manage data, settings, preferences and rules that feed the system. Today most of these features are managed with settings, control panels, on-boarding processes, preferences, questionnaires and many other similar features that usually are pretty hidden, boring and secondary for the user experience of many applications and digital services. For example, we all know very well how is complicated to manage the privacy settings in Facebook or the notification system of an internet banking.
These features are crucial not only for transparency and for the sake of personal data and privacy, but also to give the user a real control on the service and its outcomes and to make it more personalised, useful and relevant. The user should able to fine tune and update the system configuration continuously, making key settings and preferences easily accessible almost at every step of the experience. The user should be able to see what is behind every automatic decision of the system (“you see this because…” - “this happens because…”), in order to learn it and manage it and change the service result at will. Finally, the user should be able to reset completely the settings and program again the system. Configuration features and control panels should be designed properly in order to be more accessible, comprehensible and usable. To provide the user with a good overview of the potential consequences of different settings and choices, the effect of a configuration change on the service outcome should be visible and understandable, also in advance, possibily with graphic simulations and visualisations.
Conclusions
Today many smart systems and experiments are still driven mainly by technology. This is a natural step for any emerging technology before getting mainstream. But the “smartification” of objects is not enough to engage the user and make successful commercial applications. Beyond single connected gadgets and the technical framework, the user value lies in the consistency of the smart platform and in its capability to deliver useful services grounded on actual needs and goals.
A well designed user experience is the key for the success of any hyperconnected smart technology. The more connected devices and the technology blend in around us the more a good smart user experience is crucial for service adoption and efficacy. As Mike Kuniavsky writes, we need a new integrated design approach for platforms and services that combines interaction design, industrial design and service design to create smart experiences centred on the users and their real needs and desires.
A special thanks to Thomas Sutton.
References. This post is the result of many readings, conversations and discussions (and experiments with smart gadgets) occured over the last two years. The following references are just the most relevant and the most recent, but surely I am missing many others.
(1) K. Worden, W.A. Bullough, J. Haywood, Smart Technologies, 2003, World Scientific Publishing Co.
(2) Michael E. Porter, James E. Heppelmann, How Smart, Connected Products Are Transforming Competition, HBR, 2014 https://hbr.org/2014/11/how-smart-connected-products-are-transforming-competition
(3) Tim Worstall, Eric Schmidt’s Quite Right The Internet Will Disappear; All Technologies Do As They Mature, Forbes, 2015 http://www.forbes.com/sites/timworstall/2015/01/24/eric-schmidts-quite-right-the-internet-will-disappear-all-technologies-do-as-they-mature/
(4) Mark Weiser, The Computer for the 21st Century, 1991
(5) Justin Zalewski, Interaction Design within the Internet of Things, 2014 http://www.kaplusa.com/interaction-design-within-the-internet-of-things/
(6) Mary Treseler, How is UX for IoT different?, O’Reilly Radar, 2014 http://radar.oreilly.com/2014/11/how-is-ux-for-iot-different.html
(7) + Rehabstudio http://www.internetofuselessthings.io
(8) Azure Yang, Simone Rebaudengo, When Objects Talk Back, Design Mind, 2014 http://designmind.frogdesign.com/2014/10/when-objects-talk-back/
Other references:
Steven Hoober and Mudassir Azeemi, People, Ubiquity, and the Internet of Things: The New Mobile Context, UX Matters 2014 http://www.uxmatters.com/mt/archives/2014/01/people-ubiquity-and-the-internet-of-things-the-new-mobile-context.php
Claro Partners, The Internet of Things calls for a new approach to design, 2014 http://www.claropartners.com/the-internet-of-things-calls-for-a-new-approach-to-design-and-why-sticking-a-sensor-on-it-isnt-enough/
Antonio Regalado, Business Adapts to a New Style of Computer, Technology Review, 2014 http://www.technologyreview.com/news/527356/business-adapts-to-a-new-style-of-computer/
Dave Grey, Everything is a service, 2011 https://www.sprinklr.com/social-scale-blog/everything-is-a-service/
Michael Kuniavsky, Smart Things: Ubiquitous Computing User Experience Design, 2010.
Michael Kuniavsky, A pattern for user experience design in predictive analytics http://www.wired.com/campaigns/theconnectivehub/?p=390
Steve Denning, CES: A User’s Guide To The New Economy, Forbes, 2015 http://www.forbes.com/sites/stevedenning/2015/01/14/why-the-consumer-electronics-show-is-dying/
Tom Simonite, 2014 in Computing: Breakthroughs in Artificial Intelligence, Technology Review, 2014 http://www.technologyreview.com/news/533686/2014-in-computing-breakthroughs-in-artificial-intelligence/
Lelylan Blog, We need to improve the Internet of Things User Experience, 2014 https://medium.com/@lelylan/web-apps-for-the-internet-of-things-329fa683303
Thomas Sutton, Invisible Applications, 2015 https://medium.com/@thomas_thinks/invisible-applications-10501f6cfa5
