Prototyping and exploring custom interactions for the Apple TV
Considerations and thoughts on designing for the 10-foot UI experience
I was interested in working out some possible approaches to prototype for the new Apple TV. I started by taking a look at the apple TV simulator in the new SDK in Xcode. After using it for a while, it was evident that the current simulator had a number of deficiencies that would make using it for prototyping quite difficult.
Here are some of my main concerns with simulator:
- No full screen ability
- Requires constant reboots of the simulator after changes
- Interaction is very clunky with the remote panel. Its so hard to move around the screen with the mouse and the remote panel. It requires you to hold down the option key to interact with the remote but as soon as your mouse moves off the small target area, the interaction is lost. I found it hard to just move across a grid.
- Simulator is slow and not very responsive
I felt that designing and prototyping in this environment would make it hard to explore genuine interactions that could feel natural for users when they are sitting back on their couch and have a real remote in their hands. How could I possibly get closer to that environment?
Setting up your environment for prototyping on your TV
I viewed a great composition created by Julius Tarng on the origami community (https://www.facebook.com/groups/origami.community/) which showed how quickly Apple’s new focus parallax could be re-created. This got me thinking about using Quartz composer for building out larger interactions and interface elements for the Apple TV. I used the origami app on iOS as a remote for interactions as Julius had done, and this was already a far better remote than in the simulator in Xcode. It felt more real, was in my hand and made performing real gestures far easier.
The first thing I explored was how to test prototypes on a real TV quickly at a distance. Many of us are now accustomed to looking at designs on small high density screens or really close up on Apple cinema displays. To design for this platform, I believe you need to create an environment that is as close to the real thing as possible. You need the feeling of a remote in your hand and also to be sitting on your couch.
A quick early test on my TV from the couch immediately changed my thoughts on designing for it. I had so many different ideas on the size and use of fonts, the speed of animations, the types of animations etc.
Creating a blank composition
To start, I set up quartz composer to render a full 1080p scene as the canvas that I then sent full screen to my TV. When starting the new project, I went file > new from blank, instead of new origami file in this particular case.
Then, I set up the viewer to render a 1080p scene. To do this, I showed the status bar on the bottom of the viewer by going viewer > then show status bar.
I then clicked on custom in a drop down box on the left hand side and selected HD 1080p. I then re-hid the status bar on the viewer.
I also hid the tool bar on the viewer before going full screen. Right click on the top bar in the viewer to hide it.
Rendering to a TV at a good frame rate
The next problem I had was getting it to be full screen on the TV at a decent frame rate at 1080p. If you hide the status bar & toolbar in the viewer, which was discussed in the last steps, and then go viewer > enter full screen, you will get an output that works correctly on a TV.
I initially used AirPlay to send the data to the screen. This works, however the latency and lag is far too high. The benefits of AirPlay meant that I could sit far back on my couch from the TV, which I knew is going to be essential to testing these interfaces properly at a distance.
I tried a number of things to improve performance via airplay but none gave me a frame rate that was inline with my expectations. I ended up using an extra long HDMI cable, which had much better performance and seemed to make running the composition on the TV viable while still being able to sit back on the couch. Finally, my new TV prototyping environment was all set up and ready to go! Here is what my prototyping environment looked like once everything was running well at 60fps:
Re-creating the Airbnb TV app experience on my TV
I then set myself the goal of re-creating some of the interactions from the new Airbnb Apple TV app that was shown at the Apple Keynote event. I did this for two reasons: firstly, to see if I could re-create it at a high fidelity in QC, as I had a reference point from the video attached at the Airbnb blog (http://blog.airbnb.com/stay-tuned-airbnb-for-apple-tv/); and secondly, it contained a number of custom transitions. If I could re-build these, I thought I would feel quite comfortable creating a range of completely new interactions in the future, or even exploring new ideas for the Airbnb app once my prototypes were in a good state that allowed for some new exploration.
If you watch the video in the blog link contained above, there around 5 key animations/interactions:
- Focus scroll a carousel of properties on the home screen
2. Zoom into a property
3. Move through photos once in a property
4. Read property description
5. Heart/like a property
I decided to focus on re-creating the first two.
When prototyping, it’s essential to follow this key principle:
Break up what you are trying to build. It does not alway make sense to build every interaction in a single prototype, in most cases other than certain long workflows, your prototypes are easier to manage and build in seperate compositions.
“The goal is to just illustrate and try the idea, so you can evaluate the result in the most realistic way possible with the least amount of effort.”
By setting up quartz composer in the way described above and using the Facebook origami iOS app on your iPhone as a controller for controlling your Apple TV prototype, I was able to evaluate ideas while sitting on the couch at the right distance from a TV.
Statically laying out the different Airbnb layouts in Sketch
I started by setting up a few sketch templates with the Airbnb app layouts, to work out a full workflow for prototyping on the new Apple TV. I setup the template and art board for quick translation over to Quartz Composer at 1080p resolution.
Here are the links to the sketch working files I used as inputs for creating the prototypes.
Airbnb TV Spain Full Screen:
Airbnb TV Template — Paris Focus:
Airbnb TV Template — Spain Focus:
I moved these into quartz composer and started brining them to life!
Zooming into a property animation
The first interaction I looked at was the zooming into a property animation that requires a simple tap on the remote/iPhone screen to open up the selected property. As I didn’t have access to the new Apple TV hardware or Airbnb app, my re-creation of these were simply by eye from re-watching the video on the Airbnb blog and Apple Keynote (apologies if these are not accurate).
The zoom animation is quite a custom animation, with a number of fine details that have been used as you tap in and out of a selected property. Here are some of the details that needed to be re-created:
Based on the above details, I wired up the patches in quartz composer and the composition started to come together.
My final quartz composer composition can be accessed here, which you can use and also try in full screen on a TV via HDMI. To view this, you will need to install Quartz Composer and have the Facebook Origami plug-in. More information about origami can be found here: https://facebook.github.io/origami/
Move between properties focus interaction
The second interaction I looked at re-creating was the interaction in which you can swipe left and right from one property to the next on the Airbnb App home screen.
I then tried this out on a real TV at a distance:
Playing around with the prototype on the TV brought up some new thoughts and ideas. I was thinking about the need to go in and out of properties to see all the photos of the listing. I decided that there wasn’t enough information displayed at this level for me to be able to make a decision on whether or not I should zoom in and get more information about the listing.
My concern was that it was quicker on my iPhone to scan through listings, however it was better on the Apple TV if I wanted to scan through listings with a group of people around the couch, as they could all see it easier than trying to see it on my iPhone. Some other ideas could be explored to allow for a quicker preview without zooming in and out all of the time to know more.
On the iPhone I can see more key information (such as price etc) and also quickly look through the photos before diving into the listing.
One of the great things about prototyping is that there is no right or wrong answer or a right or wrong way about doing things. You simply want to explore and evolve ideas with a very open mind and try lots of things.
The final quartz composer composition can be accessed here:
Could the new Apple TV circle interaction on the remote be used to quickly peek into content?
With the environment set up and workflow in place, I started exploring my idea of an entirely new Peek like interaction for the Apple TV:
Some of the details I added to the interaction can be seen here:
At one stage, I wasn’t sure if I should blur out or just drop the opacity of the two side images. The benefit of prototyping tools is that they allow you to quickly evaluate these options on the go.
The final quartz composer composition can be accessed here:
Closing thoughts
I think this is a great way for iterating quickly on Apple TV designs and interactions. It’s a credit to Quartz Composer & Origami for its flexibility as a tool to be able to quickly adapt to the task at hand.
As we begin to understand this new platform, I will be very interested to see what kind of new interactions are developed for navigating content within TV apps. Quartz Composer is a great place to start to explore what these new interactions could be and try out new ideas before committing to engineer them in code.
