Developing a UI in Mixed Reality: Spot Panel

Introduction

Over the past few years, we at Spiral Technology have explored different ways to make the most of Mixed Reality, especially with the Hololens. The specific cases we’ve tested could be a story in itself, but for now, we’re zooming in on Non-Destructive Testing (NDT). Even though every industry has its own jargon, at the end of the day, it’s all about the same thing: ensuring quality by taking a good look.

Our app, Spector, uses Mixed Reality to overlay findings during industrial quality checks. When you’re working with large, sometimes massive, composite parts, spatial marking brings some major perks:

• Automatic defect measurements
• Visual marking
• Real-time defect status updates
• Location-specific info and instruction for leveraging, inspection and repair stages
• View-through for multilayer components to accurately layout the correct repair schema
• Smooth handovers between shifts, inspection teams, and repair teams

So, defect processing moves from chalk-drawn rectangles and hand ruler measurements to fully digital assets, each with its own virtual object that’s fully spatially aware. We call this object a Spot.

Spots are the building blocks of the process, giving operators all the inputs and outputs they need related to a defect. A Spot has a polygonal area that shows where the issue is, and the panel is where most of the interaction happens. Let’s dive into how we found the right spot for the Panel.

First Approach: Each Spot Has Its Own Panel

The simplest idea was to put the Panel right next to the Spot. This direct connection made it clear what the Panel was for. But during testing, we hit a snag: often several defects located in small area and the interface got cluttered. Panels started overlapping, and while you could drag them apart, it took more effort than it should. In the wide-open space of Mixed Reality, dealing with this kind of mess is a no-go — it’s like crowding all the fishermen in one spot.

Second Approach: A Single Panel

To get rid of the stacking issue, we tried using just one Panel. This approach had its pros: no more stacking, and you could access any Spot’s data from anywhere. But a big problem popped up — it was hard to tell which Spot the data was linked to. Getting used to a Mixed Reality device is a skill that not everyone picks up quickly, especially in heavy manufacturing environments where new tech can be overwhelming. Confusing UIs and virtual interactions can make things even tougher. One of the best ways to cut down on that confusion is to make virtual objects behave like real ones.

Despite the challenges, we gave the single Panel idea another shot by adding a “tail” or connector to show the link between the Spot and the Panel. When you selected a different Spot, the tail would move to the new one while the Panel stayed put. For those who are used to Mixed Reality, this seemed like the perfect solution — flexible, always within reach, and intuitive. But it didn’t really click with everyone.

Third Approach: Each Spot Has Its Own Panel

So, we went back to the drawing board but still wanted to keep the space as clear as possible — after all, this is Mixed Reality, not Virtual Reality. Users kept telling us they wanted fewer distractions. The flip-up visor on the Hololens 2 (a big upgrade from the first model) is a game-changer for industrial use, letting operators check info only when they need it.

To tackle these issues, we came up with a new idea: different states for the Panel — Active and Inactive (or just Closed and Open). When closed, the Panel only shows the most important info, like the defect level and stage. We also replaced the tail with what we call “the stick,” a chunky but useful connector. The stick links the Spot and the Panel, and it also makes it easy to grab and move the whole setup — a big help for newcomers who might struggle with clicking.

It also naturally organizes all the most important information about the defect into a panel header that’s accessible even when the panel is closed. This way, the operator can quickly spot any red flags (literally) with just a brief glance at the whole area.

To address the issue of long-distance accessibility, we added some flexibility to the Panel, allowing it to be reached from afar and easily pulled closer.

Conclusion

And that’s how it went down. Over the past few months of testing on the shop floor, we’ve gathered plenty of feedback. There’s always room for improvement, but we haven’t heard many complaints about the Spot, Panel, and Stick lately.