Magic Memoirs (Augmented Reality)

Published in

Daniel Lin

7 min readDec 30, 2016

Designing for AR/MR, Advanced iOS Prototyping

I’ve always been fascinated with those little flipbooks that “play” a video as you thumb through its pages. There’s something special about having a memory come to life in your hands — these physical artifacts that sit there, that wait for you to rediscover it someday, offer a unique sense of nostalgia that scrolling through a digital photo feed simply can’t provide.

I started wondering how this concept might evolve with the advent of augmented reality (AR) technology…

When I was in CodePath’s Fall 2016 “iOS For Designers” bootcamp, I shared this excitement with my 2 group members as we were brainstorming ideas for our final group project. We started geeking out at the possibility of working with AR, and thought it’d be compelling to use it to overlay videos over a photo to bring it to life — similar to the moving photos in Harry Potter:

What you’d see on your phone screen while holding a regular photo

Designing for Emerging Technologies

…But when designing for emerging technologies, typically the challenge isn’t figuring out the what, but really hammering out the whys — how to go from “some cool use of some new technology” to “an actual product that people would use and love”. To this end, we kept ourselves grounded with questions like:

Why would users ever prefer watching videos like this over Snapchat, Facebook, Dropbox, Youtube, etc?
Who is this for? Would users “create” these for only themselves to see? Or their parents, their friends, or the world?
What are physical photos commonly used for now? What purposes do they currently serve?

Productization & Scope

Wrestling with these strategic questions helped us align on an overall vision for the project — which we started calling “Memoir”:

Memoir helps users relive their favorite memories — the stories & videos that matter but are too often lost in unfrequented digital folders, or buried in timelines and newsfeeds.

Doing this helped clarify some fundamental principles of our app:

Accessible. We keep our favorite photos in intimate places — our bedroom walls, wallets, our iPhone home screens, because we want to keep them close to us — ready to steal a glance at, in a moment’s notice. Similarly, in our app, we should aim to minimize the friction required to view your content, and structure the rest of the app around that.
Shared. Most of our favorite memories become special because they were experienced with the loved ones around us in that moment. Whether it’s a travel trip with buddies, a special first date, or a family reunion, we want to give users the ability to create shared “Memoirs” together with the people they were with.

Crucially, these principles made it much easier for us to prioritize between the list of desired features we initially wanted to include in our MVP:

Content Creation: A flow for setting up and “attaching” videos to a photo
Content Viewing: “Scanning” a photo to play the videos
Social Components: Co-curate or comment on videos with contributors (minimally considered for MVP, but definitely important)
Printing: Ordering, printing, and shipping a physical copy of the photo to users (ruled this out for MVP, but definitely important)
Capturing: Recording photos & videos in-app (ruled this out indefinitely)
etc…

Explain The “What” By Emphasizing The “Why”

One thing in particular that I iterated on was the onboarding experience. With augmented reality in the mix, I knew it was especially important to clarify & explain our product for any users unfamiliar with the technology — simply telling without thoughtfully showing just wouldn’t cut it here.

For us, the word “photo” had many different meanings — internally, we used wonky terms like “image targets”, “activators”, “trigger photos”, and even had to clarify with each other when we referring to the digital vs. the physical world.

So first, I focused on getting the copy/story right before moving onto visuals and animations. To this end, I made an effort to leverage familiar metaphors like “albums”, and opted to use relatable terms like “relive your stories & adventures” over something too stilted like “replay a number of videos that are visually superimposed over your photo”.

I fleshed out transitional choreography in Principle, before recreating them natively in Swift

Start At The End

With accessibility of content in mind, I pushed to make the “Scanning View” the very first thing users see whenever they open the app, and so that it just automatically starts scanning/playing without user input.

Augment, Rather Than Obstruct, The Content

When designing for AR, we believe reality should be kept in center stage, and the ways we choose to “augment” it needs to enhance — not distract from — the world in front of you.

Thus, I continually simplified our navigational controls on the scanning screen — from full-width tab bars, to a single FAB, to eventually a background-less menu icon. Text-boxes are displayed without a background, coupled with soft gradients to improve contrast and legibility.

Iterations on the menu, detail screen, scanning screen, and logo

A multi-purposed line utilizes animation to communicate both system status or video playback progress.

For line animations, I first explored generatively on Principle, then fleshed out in Framer, before finally recreating in Swift

Video playback controls were distilled down until they were simply gestural — users tap on the edges of the screen to jump forward/backward.

Prototype Or Die!

A key part of the design process is fully fleshing out a feature in high fidelity — it’s what allows ideas to get battle-tested with users, and what forces designers to consider edge cases before it’s too late. I personally took to diving in the Vuforia (AR platform) iOS SDK to prototype the full experience and start grounding our ideas in reality.

Though I’m very comfortable with coding in Framer…learning how to work with AR, developer SDKs, video/graphics manipulation, and the basics XCode/Storyboard all at once was quite the learning experience! Thankfully there was a solid SDK demo for me to use as a starting point.

It took me a while to even realize that the SDK I was working with wasn’t in Swift, or even Objective-C — it was in something called Objective C++…a hybrid language between Objective-C and C++ ! ?

Looking back, it was a tremendous personal challenge to wrap my head around foreign engineering concepts like delegate patterns or publish/subscribe message busses, while simultaneously dealing with the nuances of interfacing between languages in iOS (bridging headers, anyone?), while trying to pick up the basics of multiple languages (Swift, C++, Objective-C, and Objective C++) on the fly!

Prototyping weapons of choice: Framer and Principle

I prototyped other parts of the experience (onboarding animations, progress bars for variable # of videos, bar animations, etc microanimations) in either Framer or Principle before recreating them in Swift. Doing so allowed me to (1) iterate on details and (2) think through any edge cases orders of magnitude faster than I could have in Swift alone.

Living With It, And Getting It In Users’ Hands

Thankfully, before our demo day, I had some time to user test what we had with my friends on our mini-trip to Iceland. Their excitement for the product was contagious, but I was keen to observe how they would choose to use Memoir during this travel trip together, so I could check my assumptions.

I was half-expecting that we’d end up with only one single Memoir that captured the entire trip, almost like a “pseudo-album” for our time in Iceland. However, in this first example they made, I learned that it seems more natural for users to create Memoirs for discreet events (~1–3 hour timeframes), as opposed to multi-day episodes.

In this example, my friends used Memoir in a totally exciting and, frankly, unexpected way — they conceptualized, planned, shot, and edited a music video for one of the landmarks we visited. This well-done video had a truly unique effect of leaving first-time viewers in awe, (“wow I want to go to Iceland right now”!) and have a completely different reaction whenever they ever see the physical photo again in the future. From this, I learned there’s tons of entertainment value in using Memoirs to preserve not just the “Snapchat-esque” raw & silly moments, but also as “Instagram-esque” highlight reels. It was apparent that making Memoirs could serve as a creative outlet as well!