End-User Programming for the Smart Home

As you read this, there are more “things” connected to the Internet than people. That number is slated to rise to 50 billion by 2020, and the Internet of Things industry will have generated over 19 trillion dollars in profits over the next decade.

This inundation of connected devices or “objects” has found its way into the inevitable Jetsons-style home of the future. As engineering and scientific limitations have withered away, there has been an explosion of platforms, devices, services, products and appliances that are transitioning to an era of internet-connected intelligence.

The Design Problem

As it stands, the emergence of the Smart Home presents many interesting interaction design challenges. The concept of the Home of the Future is segmented by companies that are attempting to disrupt and dominate the industry. Broadly speaking, there are three models of interaction that a Smart Home can offer users:

• remote: users can control devices using remotely, presumably through applications
• machine learning and AI: users interact with devices with minimum effort and devices become intelligent enough to deduce, predict and anticipate user behavior
• programmatic control: users are able to program individual as well as collections of smart a more suitable devices to specify inter and intra-device behavior as they please

Of these, the last model of interaction, i.e. programmatic control of smart and connected devices, presents particularly interesting design challenges. We were particularly interested in tackling the design question: how can we best allow end-users to program a smart home?

Findings

Attempting to find a solution for the design question we posed (and justify the widely accepted cardinal sin in pairing “programming” with “end-users”) involved two key challenges.

Firstly, we needed to devise programmatic constructs that were extremely usable and simple. All of the archetypes we created as we envisioned our end-user population had a common denominator if we considered the edge-case: nobody would have any programming knowledge or experience, or even worse, any technical experience at all. Our commonest users would be homeowners and homemakers, and thus, we had to design with this consideration in mind.

Secondly, we needed to ensure that we do not make a trade-off between simplicity and power, a 2. engaging and interesting and therefore, devise an interface that is sufficiently powerful to represent a majority of the users’ desired behaviors while being simple enough for the aforementioned user populations to use.

An obvious metric for designing the interface was to ensure a degree congruence between the mental models of our users and the logical model of our programmatic constructs. We went through recent as well as landmark literature in end-user programming and the Internet of Things.

Of particular note was the paper “Practical Trigger-Action Programming in the Smart Home” (CHI 2014, Ur, McManus, Ho and Littman). The paper had three parts: 1. an analysis and aggregation of desired smart home behaviors, 2. an analysis of “recipes” created by trigger-action programming model for web-automation by users of ifttt.com, and 3. a usability test of an analogous interface applied in the domain of smart device programming.

The data and findings in the paper, as well as our own analysis, led us to the following design conclusions:

1. A Trigger-Action model of programming is a reasonable approximation of the users’ mental models for smart home programming: The paper asked 318 MTurk users from the US for five things that they would want a smart home with internet-connected devices to do. Out of the numerous behaviors collected, over half of the behaviors directly fit into the trigger-action, or if-<this-happens>-then-<do-that>.
2. A complex model comprising multiple triggers and multiple actions are important extensions to the simple trigger-action model. 77.9% of the desired behaviors expressed by the users in the paper were programmable by using an interface that affords if-<single-trigger>-then-<single action>. However, the remaining 22.1% required the usage of either: multiple triggers and single action, single trigger and multiple actions, or multiple triggers and multiple actions.
3. The complex model was expressive, learnable and usable. Data aggregated and analyzed from the paper showed that most of the desired user behaviors could be expressed using the complex model of multiple triggers and multiple actions. Further, over 80 percent of the participants were able to successfully write working end-user programs using the complex model. In addition to this, a positive learning effect was observed and users got better with each programming task completed. Lastly, over 75% of the participants agreed that the interface was: easy to use and intuitive, as well as engaging and interesting.
4. When-This-Then-That, along with a scheduled task creation model, is a more suitable programming model than If-This-Then-That. In our analysis, we did a mini-user study to ask a handful of participants as well as ourselves to list use-cases for a smart home. In particular, we oriented the listing process to find edge cases that do not fit the logical construct of If-This-Then-That. In our investigation, we realized that the latter model relies on creating actions upon singular trigger events. Therefore, in the cases where it is unclear what a qualifying trigger event is, or when there can be multiple qualifying trigger events, the interface falls short. An example of this is: If-<time is between 7 and 8 pm>-Then-<action>. However, in this case, there are a number of qualifying conditions that can trigger the action (7.01 pm, 7.30 pm, 7.45 pm, etc). Hence, we chose a variant of the trigger-action model, When-This-Then-That, where the usage of “When” instead of “If” mandates that we use changes in state instead of singular events as a trigger. Further, we also came up with a missing affordance, that of scheduling device behavior. To accommodate this, we included a schedule-based interface to create programs for planned device behavior in the future.

With our analysis out of the way, we got our hands dirty!

Our Solution

Wireframes

High-Res Mockup

Design Decisions

We made several design decisions for the usability and aesthetics of our interface. One of them was the idea of a “forced” user flow, where we allow users to only make a single decision at each step in the program creation process. Thus, this reduces the possibility of errors. Further, we used thumbnail-ed layouts with sufficiently large active areas (Fitts Law) but ensured there was enough space between adjacent icons. This made the interface graphically communicative and it was easy for users to pick their choices in a large grid. Additionally, we created a vertical user flow to create the effect of incremental progress as the end-user creates the programs. Additionally, the vertical user flow was an intuitive model of flow in a desktop form factor; it is also an efficient usage of screen space. We also chose to go with a Material Design inspired user interface particularly because of the compatibility with the thumbnail-ed layout, visual consistency and playful transitions we want to build this with. Lastly, we went with Roboto as the typeface of our choice as it seemed to impart a feeling of friendliness and its open curves made for a more natural reading rhythm.

We further included other screens such as: 1. a screen that allows users to browse through all created programs, 2. an error or conflict resolution screen, which is prompted to the user in the instance the user creates a recipe that conflicts with another existing one. This screen presents alternative solutions to the user. 3. a time-based program screen. Here we allowed users to pick a date and time from a calendar OR input it in another way. Time-based program creation also followed a vertical, forced and screen-by-screen user flow concept.