Designing Typography Insight for HoloLens

A designer’s journey into type in mixed reality

I would like to share my journey of designing and developing Typography Insight for HoloLens. Typography Insight for HoloLens is an app that allows you to experiment and play with types in three-dimensional mixed reality space. I first published the app on iOS about five years ago and it was embraced by the design and education communities. Here is my story on how I brought the app from iOS to Windows, and from Windows to HoloLens.

My Background

My doodles on robotic machines and company logo types

In college, I studied electrical engineering and worked as software engineer. Then I went back to design school to study graphic design. With this mixed background, I have been creating iOS apps and have written an iOS programming book published in Korea.

Typeface Explorer (2008)

Typeface Explorer (2008) — Made with Adobe Flash and Papervision3D

Typography Insight (2011)

Typography Insight for iOS (2011)

Typography Insight is a toolkit for learning and teaching typography. I published it in May 2011 with my graduation and it was featured on iTunes AppStore several times, ranked second in the US education category. I presented this project at AIGA/NY. It was reviewed in FastCompany, The Atlantic, and Gizmodo.

Typography Insight (2011) — iPad and iPhone

Introductory video https://youtu.be/ZSrdvc_ItBM

“Anyone who loves books, words, history, or fine art — even in the slightest bit — will find Typography Insight as intoxicating as Wikipedia and as fun as a video game.” — Fast Company

“Learning the subtleties of Helvetica and Garamond used to be a pain — but a sleek new app has made the process easier” — The Atlantic

“Typography Insight is sort of like an iPad typeface encyclopedia. Only encyclopedias are boring, and Typography Insight is beautiful and fun.” — Gizmodo

Bringing Typography Insight to Windows with Windows Bridge for iOS

With Windows Bridge for iOS, I was able to bring Typography Insight to Windows. It was published to Windows Store in June 2016. Now it is running on Windows 10 destkop and tablet devices as a UWP (Universal Windows Platform) app.

Since Windows Bridge takes care of translating iOS’ UI controls and UX patterns into Windows version, I didn’t have to worry about updating visuals of UI. It supports keyboard and mouse as well as touch input. Luckily, AutoLayoutConstraints on iOS worked well on Windows which made it easy to support varying window sizes and different resolutions. The app was introduced as an example of Windows Bridge for iOS, at Microsoft Build 2016 developer conference.

Typography Insight on Windows 10 — Universal Windows Platform app made with Windows Bridge for iOS
Typography Insight on Microsoft Surface Pro

Designing Typography Insight for HoloLens

Microsoft HoloLens

In HoloLens, type is constructed as a hologram with light based on the additive color system. Since you can place type in the actual environment and the type is world locked, you can observe it from any angle just like real-life objects. The parallax effect between the type and the environment makes it even more magical, especially when there are multiple types layered in mixed reality space.

When we talk about type in 3D space, we tend to think extruded 3D text. However, besides some logotype design or limited applications, I don’t think extruded 3D text gives us that much value. Extruded text degrades the readability, especially for displaying information.

After some experiments, I started thinking about bringing ‘Type Inspector’ feature from my original 2D Typography Insight app to a holographic experience. ‘Type Inspector’ allows you observe and experiment with type in various scales. In HoloLens, I thought this could effectively become a toolkit for experimenting with types in mixed reality space, with different sizes, colors, and distances.

Idea sketches for Type Playground

I was also interested in translating learning components such as ‘Historical Typefaces’ or ‘Typeface Anatomy’ into 3D space. Since ‘Historical Typefaces’ involves chronological order based on type classification system, I thought I could visualize the timeline in 3D space.

Idea sketches for Type Anatomy

Design Iterations

The biggest challenges in the design process were scale and position. It was hard to imagine an object’s actual size and position until deploying and actually experiencing it in the device. I spent a lot of time finding proper dimension and position for the objects.

Design iterations

UX Elements — Toolbar UI

Floating Toolbar UI

I used Unity’s cube primitives with UI Text and PNG 2D Sprites to construct the button toolbar interface. For the secondary menus, I used Unity’s Animation Controller with triggers to achieve animated menu system on air tap event. Tag-Along and Billboarding were important concepts to design this floating user interface. Tag-Along makes the object follow you. With Billboarding, you can make holograms always face towards you. Using these two techniques, I was able to create a floating toolbar that stays in the view frustum.

UX Elements — Gestures

Gesture Input for manipulating type objects

For the manipulation gestures, I used Navigation and Manipulation gesture examples from the Holograms 211 tutorial. These gesture inputs are used to move, rotate and resize the type. Basic tap gesture is used for text object or menu selection.

UX Elements — Spatial Mapping

Spatial Mapping — Projecting types on the walls

Spatial Mapping is an unique and powerful capability in HoloLens. Using spatial perception capability we can place holographic objects in the real environment. In Typography Insight, I used Spatial Mapping to project and place text objects on the wall. Spatial Mapping prefab from HoloToolkit and Tap To Place example from Holographic Academy projects were very helpful to get basic environmental mesh and placing interaction. Currently, I am looking into ‘Plane Finder’ in HoloToolkit which provides advanced spatial understanding such as walls, floors and tables.

UX Elements — Text Input

System keyboard for text Input

Currently, the system keyboard is only supported in 2D XAML mode. Because of this, there is an abrupt context change between exclusive 3D and 2D mode when using the keyboard input. The system keyboard has integrated voice input.

Publishing v1.0 to Windows Store

Typography Insight for HoloLens v1.0

Experiment at Bellevue Arts Museum

Holographic Type Sculpture at Bellevue Arts Museum
Early experiments and studies for Holographic Type Sculpture
Early experiments and studies for Holographic Type Sculpture

Historical Type Timeline and Color Picker

Together with this new feature, now you can change the color of the types using newly added color picker. I hope this color feature can help you experimenting with types in mixed reality space.

Distance indicator, Save & Load scene

Type size and distance information

The app was already showing the distance information — the distance from HoloLens to the text object. However, it was hard to see this information since the distance information was displayed on the toolbar. In the latest update, I have added this distance information as an annotation close to the text object, together with the font size information.

The Save and Load feature is finally ready. Now you can save out your layout composition and restore it later. These feature updates are available in the latest version in Windows Holographic Store.

Tips: Getting Beautiful Type for HoloLens in Unity

I did an experiment with printed type ramp on the wall. From 2-meter distance, using Typography Insight for HoloLens with type ramp and debug menu, I overlaid and resized holographic text to match the physical one. With this experiment, I was able to find proper scaling factor. (2-meter is the recommended comfort distance for placing holographic objects)

Type experiment to get proper scaling factor

The scaling factors from this experiment are approximately 0.0005 for UI Text and 0.005 for 3D Text Mesh. Even though it might not be perfectly accurate, with these scaling factors, you can get the relevant type size which will be similar to physical type size at 2 meter distance. If you use this scaling factor, you can say 48pt type size at 2 meter distance are same in Unity and Adobe Illustrator.

For UI Text, you can adjust additional parameter ‘Dynamic Pixels Per Unit’ in Canvas to improve text rendering quality. However, if you increase this too much, you will get an unexpected type size and performance will degrade. With scaling factor 0.0005, DPPU value between 2–3 gave me decent rendering quality.

*For further details on text scaling factor, please refer to this README.md file in HoloToolkit. Font Scale and Font Sizes section describes how these values can be calculated by Points and Unity units.

Unity UI Text with different Dynamic Pixels Per Unit values
Unity 3D Text Mesh with different font sizes

Below is the comparison between UI Text and 3D Text Mesh. As you can see, using these scaling factors, you can expect the same size of the text. This size will be also same as the text from Illustrator or Photoshop.

UI Text and 3D Text size comparison

As you can expect, type sizes that we’ve been using on PC or tablet devices (typically between 12–32pt) look quite small at 2-meter distance. It depends on the characteristics of types but in general, recommended minimum type size for the legibility without stroke vibration is around 20pt, based on the scaling factor introduced above. If your app is supposed to be used at a closer distance, smaller types could be used.

For the font selection, Segoe UI (the default font for Windows) works well in most cases. However, I would recommend avoid using light or semilight for the type sizes under 42 pt since thin vertical strokes will vibrate and it will degrade legibility. In general, modern fonts with enough stroke thickness work well. For example, Helvetica and Arial look gorgeous and very legible in HoloLens with regular or bold weights.

So, that’s my journey so far. I look forward to seeing what you can create in Windows Mixed Reality!

Update: Type in Space(2018)

Type in Space for HoloLens (2018)

Video: Watch my presentation at AltspaceVR

Read my new story: Museum of Type — Windows Mixed Reality App Design & Development Story

Open-Source Building Blocks for Windows Mixed Reality Experiences

Text Prefabs and sample layout in HoloToolkit

Sample layout Prefab under HoloToolkit-Examples/Text:

Resources for HoloLens app design & development

Windows Holographic — Documentations and guidelines

Recommended articles on HoloLens app design & development by designers at Microsoft

Links

*I work at Microsoft as a User Experience Designer. Typography Insight for HoloLens is my personal project. All experiments and related opinions are my own.

Yoon (Dong Yoon Park)

Written by

Designer & Creative Technologist. Creator of Typography Insight App. UX Designer @Microsoft HoloLens. Parsons MFADT alum. Opinions are my own.