Type In Space — Explore Spatial Typography In Mixed Reality with HoloLens
Your physical environment becomes a new canvas for typography
Microsoft HoloLens allows you to place and see holographic objects in your physical environment. Just like real-world physical objects, you can place it on a table or wall, you can move around and see it from different angles. Since HoloLens analyzes the environment and remembers the spatial mapping information, when you come back to the specific space, HoloLens restores the digital objects. It feels magical when you see your holographic object stays right there just like other physical objects, even after days and weeks.
As a graphic designer who loves type design and typography, I wanted to see beautiful type in space. We have been seeing many examples of type overlay in motion graphics and TV commercials but they were stuck in flat 2D screens. Since I met HoloLens in 2016, I have been exploring type in mixed reality space.
My previous projects on type in mixed reality
Typography Insight for HoloLens(2016) was my first experiment with holographic type in space. With HoloLens, I was able to make an app that allows you to lay out and explore type in the physical space. Being able to lay out and see beautiful holographic type in the real-world physical environment is one of the most exciting experiences.
The holographic type shows magical tactile quality, feels like a neon which you cannot experience in the captured videos. The video below is closer to the actual holographic type seen through the lenses. (Captured through a phone camera)
I expanded my spatial type experiments to a large scale three-dimensional type composition in Holographic Type Sculpture(2016). In this project, I created a geometric 3D composition and placed it in a museum. Just like a physical architecture, literally, you can walk into the shower of your favorite type.
News Space for HoloLens(2017) was a spatial type experiment with text-heavy content, the news article. News headlines fill up your entire room with the spherical layout. With HoloLens’ high-resolution display, small text for the headline and summary text rendered beautifully while securing the readability.
In Museum of Type(2017), I continued my journey in virtual reality space. I wanted to create a virtual environment where you can learn about historically important typefaces with more direct interactions. Using the motion controller, you can grab and observe the beautiful shape of the type. If you love type design and typography, this will be one of the most satisfying experiences interacting with type.
With these experiments, I got great responses from the design community and augmented reality enthusiasts. I also had the honor to demonstrate my app Typography Insight for HoloLens to Matthew Carter, a legendary master type designer.
Toolkit for creating mixed reality experiences
I made these HoloLens apps using HoloToolkit, the open-source project for the foundational components for Windows Mixed Reality. Since then, HoloToolkit has evolved a lot and it is re-branded as MixedRealityToolkit(MRTK). A lot of new building blocks for the common interactions and UI controls have been added to MRTK. I wrote an article to introduce these building blocks in MRTK.
With these new building blocks in MRTK, I was able to evolve the type lay out experience in my app Typography Insight. Eventually, I have extracted ‘type playground’ component and created as a new app ‘Type in Space(2018)’.
Type In Space (2018)
This new app has some additional features including two-handed gesture input, improved spatial mapping, and multiple slots for save & load scenes.
Interacting with text using two-handed gesture
By default, text objects are floating in the space. You can grab and move the text object with one hand. To rotate or scale the text object, you can use your both hands. Based on the distance between the hands and their position, text scales and rotates.
For the rotation, you can specify axis constraints using the toolbar. Using this constraint, you can make the text rotate around a specific axis only.
The toolbar contains a collection of a common actions such as Save, Load, Clear Scene and Add New Text. It also has global actions such as ‘Snap to Surface’ for the spatial mapping activation and ‘Gravity’ for the physics.
The toolbar uses the tag-along and body-lock technique to avoid interfering your viewport. It is moving and always stays within a certain range, with a specific offset from your waist. It provides adjustable positioning buttons so that you can easily move up/down to favorite offset position from your body. Pin button makes it stay in a fixed location. You can also show or hide the toolbar with a voice command. “Show Toolbar/Hide Toolbar”
Using the floating toolbar, you can change the font and color of the text.
Text object menu
Contextual text object menus include object-specific actions such as Edit, Duplicate, and Delete. This text object menu automatically scales up/down based on the distance to maintain usable size. For example, if the text object is at a distance, it scales up to make it easier to target with the gaze cursor.
Snap text to a surface with spatial mapping
One of the most exciting features of HoloLens is spatial mapping capability. HoloLens scans and analyzes the physical environment and gives us information about the surfaces. Using this surface information, we can make objects interact with the physical environment.
Air-tapping the ‘Snap to Surface’ button on the floating toolbar initializes scanning experience. When you look around your environment, HoloLens will gather information about the surfaces around you. You can understand the progress with blue visual meshes.
Once it gathers enough information about the surfaces, the blue mesh will disappear. Now you can air-tap any text object and gaze on the surfaces, the text will snap to surfaces, following your gaze cursor. You can make the text align with various surface types such as walls, floors. and ceilings. Air-tap again to place the text object.
Gravity with physics
Once you have information about the physical surfaces, you can play with physics. Using the button ‘Gravity’ on the toolbar, you can toggle gravity to the entire scene. The text objects will fall to the floors and tables.
You can use the system default keyboard to type in the text. By using the dictation feature, you can easily input the text with your voice.
Save and Load the scene
You can save and load your spatial type composition. [Update 9/1/2018] Now you can export type composition as an XML file and import XML or TXT text file. On RS1 HoloLens, you need to install the OneDrive app. From RS4, the app uses system File Explorer.
[Update: Sep. 2018] Outline Text
Type in Space now supports the outlined text.
Where can we use it?
Architectural Signage Design
‘Snap to Surface’ feature allows you lay out the type on the physical surface. In real-time, you can change the font, color and observe from different angles. It helps you quickly sketch signage without building physical prototypes.
Object Labeling / Annotation
Three-dimensional, world-locked text objects are perfect for spatial annotation. Since the text object stays where you placed, you can use it to annotate three-dimensional physical objects in the real world. The video below shows the example of the labeling on the physical object(jet engine).
Floating, world-locked text object can be used as spatial wayfinding signs.
Product Package / Book Cover Design
Since you can lay out a type on any physical surface, it could be used to design logotype on a package or book cover. You can change font, color and observe it from different angles.
Data Visualization with dynamic data binding
I have been experimenting with spatial text + dynamic data. News Space is one example of using external API with spatial text. Holographic Type Sculpture could be used to visualize data too. (*Data binding is not available in the Type In Space app)
Typography in Mixed Reality
Typography in three-dimensional space introduces interesting challenges and exciting opportunities.
Type size in 3D space
Since it involves another dimension — the depth — it becomes a little bit tricky to communicate type metrics. For example, the 24pt type could be very big or small, depending on the distance in 3D space. In HoloLens, since the recommended distance for placing a holographic object is 2 meters for the user comfort, we can use it as a standard distance to define and communicate type size. In my previous app Typography Insight for HoloLens, I added example type ramp based on 2 meters.
Another challenge is the headset’s hardware display resolution. The current generation of virtual reality headsets shows the screen-door effect which is not ideal for displaying type. Designers are using larger type size or minimize the use of text in virtual reality applications. HoloLens does not suffer from this screen-door effect because it has a much better resolution(PPD, Pixels Per Degree). It can render a beautiful type with crisp contour. It also works well with serif typefaces too.
Improving type rendering quality
Unity’s Text Mesh and UI Text are blurry in default. Therefore, it requires some optimization effort. In my previous article, I described optimization options for Unity’s Text Mesh and UI Text component.
TextMesh Pro is a great solution to secure the text rendering quality. Since it uses SDF technology, it renders crisp outlines of the text regardless of the distance. From Unity 2018.2, TextMesh Pro is included in Unity editor. If you use it with a large scale font, it shows rounded stroke edges. You can improve this quality by increasing the texture size in TextMesh Pro’s texture generation process.
Scaling and positioning text in mixed reality
There are interesting techniques for the text display in mixed reality such as ‘billboarding’ which makes the text always face you regardless of your position or view angle, or ‘constant angular size’ which maintains type size regardless of the distance to secure the legibility of type.
When we talk about type in 3D space, we tend to think of extruded, volumetric 3D text. However, besides some logotype design or limited applications, I don’t think extruded 3D text gives us that much value. Extruded text degrades the readability, especially for displaying information. You can easily imagine how difficult it is to read news headlines with extruded 3D text :)
Thanks for reading!
I hope you can experience beautiful type in space with HoloLens!
*I work at Microsoft as a User Experience Designer. Type In Space is my personal project. All experiments and related opinions are my own.