Formalizing Concepts and Planning for a Design Handoff

After testing a few prototypes in Sprint 8, the team continued iterating on the designs and arrived at two final prototypes that were then tested with real users. We also delved deeper into technical feasibility and identified key considerations that our client will have to make to implement the products.

Creating an ecosystem of prototypes

For the final design solution, we are proposing an ecosystem of products, each with a different modality. First, we have a desktop app, which is intended to be used for virtual troubleshooting in an office setting. And another product is a mobile app, which is for locating equipment and troubleshooting on-site.

Desktop key features

1. Spatially mapped alerts — provide an overview of what the campus is like at the moment, which zones and buildings are receiving how many alerts/service requests and require more immediate attention
2. Potential root causes — list out all potential causes to a specific alert based on historical data to provide users with a high level understanding of what the potential causes might be
3. Equipment and sensor relationship maps — visualize how different sensors and equipment are connected at a building level and a floor level
4. Directory/Quick shortcuts to other platforms — allow users to jump from our product to other resources such as vendor websites, building automation systems, or data historians
5. View recent work — display the latest work that was done to the equipment in focus to provide more context for troubleshooting

Spatially mapped alerts

Equipment and sensor relationship map — building level

Equipment and sensor relationship map — floor level

Mobile key features

1. Spatially mapped work orders — enable users to quickly see which buildings contain more work orders and hence require more immediate attention and check how to get to a building
2. Potential root causes — call users’ attention to the more probable causes and provide readings that are retrievable from building automation systems and data historian platforms. This feature is not meant to create false confidence. The suggestions will remain high level.
3. Locate devices through floor plans — provide a familiar visual that illustrates where different devices are. We plan to have an option to overlay other technical drawings like mechanical, electrical, and plumbing to provide another layer of information.
4. Locate devices through AR — mechanical systems can get so complex that even technicians with two years of experience at a campus still have a hard time locating certain devices. This feature can become so advanced that it will have the complex parts (valves in a tunnel) mapped out. For simpler equipment like air handler units, AR can still be beneficial to personnel who are not familiar with the space.
5. View recent work — display the latest work that was done to the equipment in focus to provide more context for troubleshooting

In general, the mobile app is most beneficial to technicians who are new to the space, either because they’re a new hire or they’re just substituting someone else in case of an emergency.

Desirability, feasibility, and usability

For a new product to be successful, there are three factors that need to be considered: desirability (whether users want the product), feasibility (whether the product is technically feasible to implement), and usability (whether users can use the product with ease). They cascade down in a way that desirability is the first determining factor, then feasibility, then usability. Our faculty advisor drew out a decision tree diagram that effectively conveyed how these three factors impact one another.

Desirability can be derived from previous research data, and we were able to confirm it again through a list of questions at the end of our latest user testing sessions. More details to come in a minute. We confirmed feasibility through desk research as well as consultations with our engineering faculty advisor and a PhD student at CMU’s Augmented Perception Lab. Based on this quick investigation, we learned that the ideas of integrating existing APIs of work order software to populate our products and experimental concepts around AR are both very technically feasible. Successfully validating the first two criteria, we then shifted our focus to testing usability.

Testing with a realistic facilities problem

After the initial round of testing with low-fidelity prototypes, we learned that it is imperative to the success of testing sessions that we have a realistic problem and content in our prototypes for users to troubleshoot. As a part of this latest prototype refinement, we did desk research, consulted a retro-commissioning engineer at CMU, and iterated through multiple versions until we reached one that felt like the right amount of complexity.

First version of a sample workflow

Final version of a sample workflow

Putting our prototypes out to test

After spending over two weeks building the prototypes, we were ready to put them to test. We tested with 8 participants from 4 organizations: 3 universities and 1 governmental entity. Some were virtual due to geographical constraints, but for schools in Pittsburgh, we visited them in person and turned the sessions into fun lunch events.

All participants provided a lot of useful feedback on aspects of the prototypes that were successful and areas of improvement. We are going through one more iteration after this where we will incorporate this feedback, refine the interactions and visual design, and will test the new prototypes through unmoderated testing over the next two weeks.

Feedback from the 8 testing sessions

In addition to usability, we were able to further validate desirability through a list of specific questions geared to quantify how much users are interested in our products. After running a quick statistical analysis, we learned that there is a significant increase in confidence level after users test our prototypes. The results also confirmed that users thought the prototypes are likely going to reduce machine downtime and that there is little concern over using a new software in their workflow.

Planning ways to commission the AR feature

Another important thing that we learned from the testing sessions was that facilities managers and technicians had concerns around implementing the AR feature on the mobile app; more specifically, they were worried that the implementation will take a lot of time and effort, a concern that we had already anticipated. And so the team created a systematic plan for different strategies that streamline the process of tagging anchors in the space. There are two main approaches:

1. In the case of newer buildings with Building Information Models (BIM) — import sensors from the BIM model as “soft tags” which then get confirmed as “hard tags” by technicians as they visit the facilities to complete other repair work
2. In the case of older buildings with only 2D construction drawings — generate a 3D model based on those drawings, then add hard tags as technicians visit the facilities to complete other repair work

Revisiting success metrics and ensuring impact

Now that we have validated desirability and feasibility and are refining usability, we feel very confident about our products. However, one of the biggest challenges that we face while user testing is that the success of our tests is highly dependent on how well we can simulate the scenarios for prototype testing, which, in our case, is when a problem arises and facilities managers and technicians have to solve it. Different personnel troubleshoot differently, and there are countless paths a user can take to arrive at the same correct diagnosis. In an ideal scenario, we would love to build a demo room, where we could install a network of devices and sensors that collect real time data and send out real time alerts. That way, we could create a highly controlled setting and populate our prototypes with all the possible paths a user may take to troubleshoot a problem. If we had the resources, an alternative option to elicit more realistic user feedback is to build the products with code and beta test them with real users in real environments. Unfortunately, with the little time that we have, we are not able to execute either of the options. Therefore, to the best of our ability, we confirmed desirability and will be perfecting the usability of our prototypes. We hope that once the prototypes are beta-tested after we pass our work back to our client, we can gain more data to confirm our success metrics.

Preparing for our final presentation and wrapping up the project

In addition to iterating through our prototypes, the team has also started working on our final presentation, which is coming up in two weeks! It is crazy to think that over 6 months of immersing ourselves in the world of facilities management is soon coming to an end. We learned a lot about how to build a product from the ground up, how to navigate ambiguity in a startup environment, how to understand and focus on the people in a highly technical domain, and more importantly, how to be self-directed and take risks. At the time this article is being written, there are two weeks left until our final presentation. We are very happy with where we are, and we are very excited to see how this project will develop moving forward!

The work and knowledge gained from this project are only intended to be applicable to the company and context involved and there is no suggestion or indication that it may be useful or applicable to others. This project was conducted for educational purposes and is not intended to contribute to generalizable knowledge.