A New Design Paradigm in Mixed Reality: Using HoloLens for Architectural Design

By Michael Shyu, Iffat Mai, and Fei Xie for Autodesk University

Mixed reality (MR) combines the virtual and the physical realities into one space and offers an exciting new design paradigm for architects. By projecting a BIM model directly over a physical site in mixed reality, architects can communicate design ideas to the team and clients in an immersive and interactive way.

This article will demonstrate case studies of mixed reality, using Microsoft HoloLens, applied for different phases of architectural projects. We will share our exploration of the process of precision alignment of a BIM model with the physical project site, followed by using mixed reality for virtual mock-up and review of design options. Finally, we will examine the potential of mixed reality tools for construction administration and envision on-site mixed reality clash detection using Microsoft HoloLens.

Design communication is advancing rapidly. The use of tools such as virtual reality (VR) has become a commonplace practice for design teams. Mixed reality offers additional benefits to virtual reality by overlaying virtual objects onto a real physical environment. This could be a game-changing tool for all phases of design and communication. By integrating the Microsoft HoloLens and BIM modeling with real-world sites, mixed reality offers new possibilities for architects to communicate design ideas to the team and clients.

Our project began as an incubator proposal initiated by Michael Shyu and Fei Xie. Once the proposal won as an incubator project, we formed an ad hoc team to work on the research, design, and building of a mixed reality tool. The objective was to explore new opportunities that mixed reality could provide for the design and construction industry. We hope to use the mixed reality tool in all phases of design, from initial concept and site analysis to construction administration. It will include design model options, interior designs applied to physical sites, and clash detection for construction administration.

Objectives

Explore design communication using mixed reality for all phases of architectural design.

• Concept Design: Small-scale building modeling
• Schematic Design: Interior space designs, spatial alignment, and options
• Construction Administration: Clash detection for interior spaces, spatial annotation, and recording data

VR vs. AR vs. MR

Virtual reality uses a computer to create a simulated environment that is completely isolated from the actual physical environment around you. It offers you an immersive experience, but it also blocks out any relationship between the virtual and physical worlds. On the other hand, augmented reality (AR), which is a technology that has been around for decades, presents virtual information on top of a real physical environment. AR gained notoriety for its popular Pokémon GO app where users can see virtual Pokémon as they walk around town using their mobile phone. MR using the HoloLens falls between AR and VR, where one can experience the virtual objects merged with the physical objects, not by looking at a mobile phone or tablet, but through the transparent lenses of a HoloLens headset and interacting with the virtual object using a natural interface.

Mixed Reality App

What separates our idea from other MR tools is that our goal is not simply to represent the BIM model in real time, but to have the machine learning algorithms align the model to the site and determine what in the project would really call for the attention of the architect and contractor via the HoloLens. Our methodology first starts with alignment. Utilizing SLAM (Simultaneous Localization and Mapping) technology embedded in the HoloLens, it is possible to recognize surfaces and thus we are able to align reference points from the BIM model to the real space.

We created a simple user interface, where a user with no prior experience with the HoloLens will be able to experience their design model both as a site model and as a full-scale interior space. The user interface of our MR tool uses the designer’s gaze for cursor control, a voice command system for designers to interact with the model in a hands-free fashion, and a gesture control series of commands using simple air-tapping gestures.

Tools

Before starting the project, it is important to secure all the necessary hardware and software tools needed for the project. The HoloLens headset is the primary device that will be used for the project. We also needed a laptop that meets the minimum system requirements for MR development:

• 64-bit Windows 10 Pro, Enterprise, or Education (the home edition does not support Hyper-V)
• 64-bit CPU
• 8 GB of RAM or more
• In the BIOS, the following features must be supported and enabled: hardware-assisted virtualization, Second Level Address Translation (SLAT), hardware-based Data Execution Prevention (DEP)
• GPU (the emulator might work with an unsupported GPU, but will be significantly slower): DirectX 11.0 or later and WDDM 1.2 driver or later

Software Tools

• Visual Studio 2015
• Unity 3D version 5.6
• Holographic emulator and holographic templates

Note: As of October 17, 2017, Microsoft has released the Windows 10 Fall Creators Update which works with the newer version of Visual Studio 2017 and Unity 3D 2017. If you choose to use the newer version software, be sure to use all the newer version tools to maintain the compatibility of software and drivers.

Mixed Reality Workflow

Here are the workflow steps for importing 3D models from Revit to HoloLens:

The Site

Our initial proposal called for using an active project that included an interesting renovation and addition as our on-site test case. However, after gaining a better understanding of SLAM and the construction schedule of the project, we had come to realize it was better for us to test out the MR tools in a more controlled environment before going on an active project site. The tools that needed to be specifically developed and programed were alignment, layering, and annotation. We anticipate that after the steel structure is erected, that the HoloLens will be a much more effective tool in that it can virtually tie itself to the superstructure and begin to overlay augmented virtual information on the site.

If we were to use the SLAM technology to align to an exterior foundation footing, the sensors might have a hard time locking the edges in place and the model would experience severe drift. In addition, the visual limitation of the screen lends itself to a more interior-oriented reality experience. We decided the best testing ground for our experiments was the 12th floor of 225 Franklin Street, which happens to be the floor directly above the Perkins and Will Boston Office. It also happens to be an empty office space that is easily accessible to our team.

Windows Mixed Reality Academy

Windows Developer center provides many helpful tutorials and resources to guide newbie developers with sample codes and best practices. Another useful tool is the HoloToolkit, which is a free download.

Using SLAM for Model Alignment

The model alignment between the virtual and the physical worlds is the first step needed to anchor the model onto the exact location in the real world. In order to align the two spaces, it is necessary to lock three virtual points to three physical points in order to lock the X, Y, Z axes. The initial sketch below shows the first steps of the process.

Light, Transparency, and SLAM Limitations

One critical point to understand about HoloLens is the way it creates virtual holograms through an additive process utilizing light to create the holographic projections. It essentially cannot subtract information, and the color true black would read as transparent. Shadows can be achieved through greys and dark blues, but active shadowing is very computationally intensive for the HoloLens at this time.

SLAM Limitations: Exterior

Alignment is the critical first step in being able to project a believable hologram. In order to align a virtual object, the HoloLens essentially utilizes SLAM to recognize edges and subsequently allows it to anchor the virtual object into place. When edges are not present, as in an exterior space, or there are too many shadows, the sensors cannot read the correct edge for appropriate alignment and essentially cannot anchor the model down. This technological limitation hampered our ability to pursue the original scope of the incubator. In the future, we imagine it would be possible to tie exterior models into place with GPS in conjunction with SLAM; however, it would require hardware development and external sensor tools that communicate with the HoloLens to achieve this result.

SLAM Limitations: Interior

Through hands-on testing, we determined the SLAM boundary for HoloLens to be an area of about 20 feet by 10 feet. This is critical to understand because that will place a limit on how far the HoloLens can keep its alignment.

Two-Point Alignment System

With the help from our team member Ryan Zhang, a researcher from the MIT Media Lab and GSI, we developed a two-point alignment system that enables the user to easily place a virtual model and align it to the physical world. The user would identify two coordination points in the real world and the virtual world. To align the spaces, one would place the first anchor point (represented by a white ball) at the first coordination point using the SLAM to snap to the exact point, then drag the second point (represented by a red ball) to the second coordinate point, which will then define the scale and orientation of the virtual model. Our UI designer, Chance Heath, developed a nice series of user interface menus to guide the user on how to use air-tap gestures to select and place the coordination points.

HoloLens Design App

Our final HoloLens Design App starts with a simple visual instruction on how to use air tap to select the functions. The menu will show three distinct modes that offers Model Observatory for concept design, Design Options for schematic and design development, and Construction Assistant for the CA phase.

Model Observatory

The first mode is the Model Observatory, where the designer can show a preloaded model appearing as a scaled down desktop model. The designer can then place the model onto a space using HoloLens’s SLAM capability, the model will snap to the selected surface. The designer can then interact with the model and click on the buttons to review different options. This mode is very useful for designers to view the entire model at a smaller scale.

Design Option

The second mode is Design Option, where designers would place the virtual model using the two-point alignment method so the model is at full scale anchored correctly onto the real world. Then using the design option buttons, the designer can review the various different design models at full scale. Being in full scale, the designers can walk around and experience the space in an immersive fashion. The design option includes materials such as marble, wood, and concrete.

Construction Assistant

The final mode is the Construction Assistant mode. Currently, this section is in the design phase, where we are showing potential functions that our construction administration managers could utilize during that phase of the project. We interviewed with our construction manager, Heather Miller, who had completed numerous construction administration work on projects. Here is some input and feedback from her:

1. The ability to export to JPEG

• Share JPEGs and aid field reports.
• Take pictures with model and without model.
• Possible tie-in with a document management system (Newforma, for example).

2. Voice command and notations

• Typing in HoloLens is cumbersome. Superlinks will save voice comments, and you will be able to create video reports.
• You want to be able to put icons in the video so you can question the contractor, but don’t want to directly talk to them immediately.
• Sharing views is incredibly important — possibly sync between tablet and HoloLens or HoloLens to HoloLens. A multiuser experience is very important.
• Referencing is also important. You want to be able to find the right detail from the contract documents quickly.

3. Architect liability

• While it is good to know all of the information and anticipate clashes, it is important to note that the architect has to show judgment as to whether to act on it. It is better to check and verify first before pursuing change orders.
• HoloLens vs. tablet: HoloLens gives a degree of privacy so the HoloLens can review drawings with the architect alone, whereas the tablet shows all information to all parties.

4. Precision

• Understanding the precision for the HoloLens will be important, as different construction types require different degrees of accuracy.

The technology for the HoloLens is truly amazing, and there is no other MR product on the market that combines its rapid SLAM recognition, fast tracking, untethered portability, and ability to upload high fidelity models and programs. If we combine the HoloLens with machine learning and BIM modeling, in the near future the sensors detecting the SLAM boundaries will be able to extrapolate that into actual object recognition.

The power for a computer to recognize an object from different angles and be able to actually process what that object means in terms of its inherent data is an immensely important step. Imagine in the near future when you go to a job site with the HoloLens, you scan a piece of duct work which will then cross reference automatically back to the BIM model. Using machine learning, the HoloLens then recognizes that the physical object is indeed a return duct, and will automatically provide you with all information associated with it — what kind of duct is it, how much air it moves, where it runs in the overall building. Or better yet, you aren’t even actively looking at the duct, but the HoloLens automatically detects that the duct is not in the correct location and brings up the contract documents for you to review as a reference! After it detects the anomaly, you are able to produce a field report recording your findings for review when you return to the office.

Another hypothetical experience is if you are trying to communicate this clash to the general contractor, and you want to compare what you see to what the contract documents state. You both put on a pair of HoloLens and are instantly able to see what the intent is on the physical site, clearing up any potential confusion or mistakes thus saving money for the project.

The future is bright for the technology, and the shared experience is one that will dominate the market in the coming years. It takes a lot of effort to develop the tools needed. However, once you develop the tools, your initial investments will instantly pay dividends for years to come.

Michael Shyu began his research in smartphone applications and their interface with physical architecture in 2009 with his Bachelor of Architecture thesis at Syracuse University. He then carried his research to Columbia University GSAPP where he participated in design studios focused on augmented reality and space planning and developed application interfaces and designs for various project types.

Iffat Mai is the firm-wide design application development manager of Perkins and Will. During her more than 20 years of working in the AEC technology field, Ms. Mai has shown leadership in making strategic technology decisions, developing innovative solutions, and integrating cutting-edge technologies into AEC design workflow. Her recent focus has been weaving VR, AR, and MR with BIM into professional architectural practice.

Fei Xie started augmented reality research during his internship at Adrian Smith + Gordon Gill in 2013. He successfully developed an app via AR-Media SDK allowing clients to compare different design options with AR technology. In 2014 he began a project which allows people to create their own AR portfolios. Fei earned a bachelor’s degree in physics before receiving his master’s degree in architecture from Washington University in St. Louis.

Learn more with the full class at AU online: A New Design Paradigm in Mixed Reality — Using HoloLens for Architectural Design.