Assembling IKEA’s new AR app, without a manual
Plenty of think pieces have imagined the potential of augmented reality reaching the masses, but when it suddenly did — with the arrival of ARKit and iOS11 — would the tech industry actually put their money where their mouth is? Could we? Without any guidelines or best practices to rely on, we aimed to find out and started working on IKEA Place.
More meaningful to the process of democratisation than many political measures put together. That’s what one Swedish industrialist-politician would call IKEA just a few decades after Ingvar Kamprad registered the company in 1943 — the first step towards fulfilling his mission to “create a better everyday life for the many people”.
In the 74 years since, IKEA has managed to become a global brand, practically ubiquitous with everyday life. Continuing to fulfill that original promise today, though, requires more than flat-pack furniture. The world is changing faster than ever before, and in this brand new landscape we need not only to be aware of the changes but to stay ahead of them.
This is what we do at SPACE10, a future-living lab where we explore better, more meaningful and more sustainable ways of living — based on the needs of people. An outside-in lab, if you will, involved with everything from open-source architecture, to hydroponic food systems, to machine learning, making sure IKEA is always prepared for new opportunities.
In the summer of 2017, augmented reality became one of those opportunities when Apple announced ARKit for iOS11. By enabling millions of existing iPhones with AR capabilities, it would suddenly reach the masses and thus be relevant for IKEA. We were invited to take part in shaping this new adventure for IKEA.
The only twist was that if IKEA were to premier a potential new AR app with the launch of IOS11, we had only two months to meet Apple’s deadline.
Besides being challenged by a very short timeframe, we also needed to completely rethink every UI and UX convention we’ve come to know in the smartphone era.
With the iPhone being 10 years old, everyone has spent the best part of a decade getting used to communicating with, working on, and designing for small screens.
With AR, people are no longer just looking at their screens, they’re looking through them. Meaning that every guideline from the past decade has to be reconsidered.
APPROACH
Placing furniture in augmented reality has always been an obvious use case, so much so that it’s easily taken for granted. Yet the potential importance for IKEA cannot be overstated. Today, not everyone is on the doorstep of an IKEA store, and nearly 40 percent of people deal with an “imagination gap”: a lack of confidence in taking risks regarding changes in their homes. There’s never been a more essential time for IKEA to innovate and find new ways to reach people. With augmented reality they can suddenly meet people wherever they are. AR is a way to bridge the imagination gap and not only continue to democratise design, but to potentially democratise interior design, too.
We felt an obligation to fulfill this potential, and knew that the only way we’d stand a chance would be by putting the people front and centre. It’s easy to get caught up in the technology and just look at the innovation of AR, but that would be a limited point of view. Instead, we ensured we worked closely with IKEA, of course, and involved many other points of view, in order to make IKEA Place as natural to use as possible. Longtime SPACE10 collaborators Sebastian Gram and Mathias Høst Normark, from studio Norgram, led the design. As they each live in a different city, the philosophy behind their decentralised studio quickly became the ethos of the project: “It’s nice to work with people who sit right next to you,” says Mathias, “but it’s far more important that they’re the right people.”
Our own team at SPACE10 was set up similarly, with Kaave Pour (Creative Director) and Daniel Friis (Digital Lead) operating from our office in Copenhagen, Bas van de Poel (Creative Strategist) working between Copenhagen and Amsterdam, and Bryan Wolff (Writer, of both the app and this article) primarily based in New York. Also essential was our collaboration with TWNKLS, located in Rotterdam, the Netherlands, and one of the few companies completely dedicated to the development of augmented reality and computer vision. “Having all these different minds on the project made the difference,” says Daniel van der Schoor, manager at TWNKLS. “It meant we all pushed each other to our limits.”
FIRST DECISION
As soon as we had the main team in place, we knew we had to hit the ground running, and so we did. The first thing we needed to decide was the primary state of the app. Would this be an experience akin to browsing the classic IKEA catalogue, after which people could bring it to life in AR? Would it be a social experience that opened on a feed of other people’s places? Should we dive into a full 3D world where every menu was situated within the place? Or were we perhaps still thinking too much from our own perspective?
We had to stop staging scenarios defined by ourselves, and focus on creating something on the terms of the many people around the world. We decided to start with what matters most to them: their own place, or rather the camera through which they can augment it.
This would form not only the primary state of the app, but also the primary challenge of its design: “Most of the time when designing an app, you’re designing for one screen, maybe in a few different sizes, but generally everyone experiences it in the same way,” says Mathias from Norgram.
“With IKEA Place focussed on the camera, every instance of the app is unique, based on the environment the people are in. It’s a super-interesting challenge, a chance to redefine what it means to design for digital as it becomes less about screens, and more about the physical world.”
How much we would have to shake off our digital-design habits became apparent when we moved on to the experience of adding furniture in AR. We wanted to never take people out of their place entirely, and thus designed the product drawer to be an overlay, which keeps a small sliver of the actual place visible at all times. The layout of the browsing experience was next, a problem that has plagued mobile design for a while now.
Websites are great when you know exactly what you’re looking for. You search for a product, you find it, you buy it. But if you want to be inspired, a wall of products isn’t the best experience.
That’s especially true when it consists of thousands of products, which happens to be the case for IKEA. So, instead of a straightforward index, we looked to those who’ve come to set the standard for inspirational browsing: music and video streaming services. Similar to those browsing experiences, we created themes and styles under which we bundled certain furniture, in addition to their typical categories such as “chairs”, “desks”, and “lighting”.
We made a rule: Lifestyle image would bring people deeper into the browsing experience, whereas an image of a product on white would lead to placing it in AR. This was the first opportunity to truly embrace the design advantages augmented reality offers: instead of having to explain or display the usual information that would come with an IKEA product, we could now let people experience it.
Materials are visible in the textures of the 3D models, and the size is displayed with 98 percent accuracy, so no more pinch-and-zoom. It may be a little strange at first, but it’s the only way it makes sense. Forcibly enlarging the digital furniture would only lead to disappointment once its physical counterpart were ordered and delivered, and experiencing the true size is the whole benefit of AR.
It’s a benefit not only for people, but for the creators too: when people interact more with the world behind their screen than with the screen itself, we can start thinking of digital interactions more like real-life ones. Getting a sense of the size of something means simply moving around it. And to see the materials, you get up close, naturally.
3D MODELS
As one of the world’s leading design companies, IKEA makes more than two thousand new products a year — products that can each be assembled in a variety of combinations. Now, imagine having to shoot or reshoot photos of these products with every little alteration or modification that comes from a company of over 200,000 co-workers.
At some point, photography simply becomes inefficient. Which is why, in the summer of 2004, IKEA started making serious steps towards CG-rendered imagery of its products. In the catalogue published in autumn 2006, the “BERTIL” chair premiered in all its polygon glory. In 2009, the first complete 3D room-set followed. Today the majority of the catalogue is put together without so much as a single flatpack or Allen wrench. “About 55% is completely 3D, with another 20% being mixed,” says Dan Frith, manager of IKEA 3D communications.
What’s amazing when working with Dan and his team is that such a technology-driven part of the IKEA business doesn’t lose any of the brand’s inherent human sensibilities. It carries the same artistic touch that defined the department in its pre-3D era, which is the direct result of IKEA’s choice to school its photographers in 3D and its 3D artists in photography when it first started making the switch 10 years ago.
“Lighting is probably the single most important thing in 3D, and that knowledge comes from photography,” says Dan, a photographer and 3D artist in his own right.
It’s this passion and attention to detail that was crucial to the conversion of the existing 3D models — consisting of a few million polygons that could take up to half an hour to render — into those that would be less than 1.5 MB and able to be almost-instantly generated in AR. As Dan tells it: “We’d get a converted model back, one of our 3D artists would look at it and right away know what would need to be refined. ‘Oh, it needs more emphasis in the form’, or ‘The curve isn’t bold enough,’ whatever it may be. That’s knowledge that’s ingrained in IKEA’s work culture.”
Together with TWNKLS, we made sure that IKEA Place could present these laboured-over models in the best light possible. Literally. “ARKit is so tightly integrated with the iPhone’s hardware, with only a limited number of supported devices, that we can completely rely on features like the lighting sensors,” says Daniel of TWNKLS. While we were unable to know the exact type of environment in which people would use IKEA Place, we could at least get a sense of the most important element in that environment: light. With this information, the shadows and brightness of the 3D models could be adjusted and matched (as much as possible) to that of the room in which they’d be placed.
BRINGING THE FURNITURE TO LIFE
We now knew how we wanted the furniture to look once it was placed. But how would people actually place it? This sounds straightforward, until you start thinking through the details. What happens to the price and name of the product — do they stay in view or not? Is there a different state the product is in when being moved compared to once it’s put down? Can you even put it down, or is it just always on the floor?
As with IKEA’s 3D department, we refused to let ourselves be distracted by the tech involved and focussed again on human sensibilities. Based on how we would interact with any real piece of furniture in our home, we created two states for placing products in AR. The first state is similar to picking something up and moving it around: the product floats slightly above the floor and can be moved by simply swiping it around. In this state, the price tag and product name stay in view, but as soon as you’ve chosen a spot and placed the product there with a tap, the product lands on the floor and everything just disappears from the screen. The price, the name, even the “favourite button”. Suddenly, it’s just the product in your place, and the imagination gap begins to close.
When the furniture lands on the floor, it does so with a little thud or bounce. We further enhanced this with haptic feedback and custom sound design, each attuned to the product’s weight and size. For the audio we brought in a fourth collaborative party: Plan8, a sound and technology studio based in Sweden. According to Oscar Tillman, one of its creative directors:
“Sound can always play a role. But it should be purposeful. Why do we need a sound here? In AR, the answer is quite obvious. It helps what you’re interacting with to feel like part of your environment, part of your place.”
Sound design in augmented reality might be useful, but it’s also a lot harder to accomplish. Unlike the light sensors that could help us adjust the brightness of the 3D models, there was nothing to inform the size of the room or the materials the sounds would be bouncing off. “We don’t know if you’re on a soft carpet or a concrete floor, whether you’re in a dorm or a palace,” says Oscar. “The one thing we know is the size of the furniture you’re placing.”
From this starting point, Oscar and his team were able to produce a beautiful set of sounds inspired by IKEA’s heritage of wooden furniture. They began with true foley sounds — using recordings of actual wooden materials — and eventually processed them to keep them slightly more snappy, modern and appropriate in any environment. “Basically it’s about making the sounds more felt than heard,” says Oscar.
LEARNING CURVE AND FUTURE PROOF
With the amount of detail bringing the main AR experience to life, the rest of the app couldn’t stay behind. We wanted to ensure a smooth learning curve while making the app future-proof for other features we were already envisioning. Luckily, these two goals were not mutually exclusive. When thinking about the easiest way to communicate with people, including some basic set-ups such as allowing access to the camera and scanning the floor, we found the answer to be, yet again, the most natural one: conversation. Now, “conversational UI” might sound like another foreign technical aspect to add to an already novel experience, but it’s all in the name.
We don’t tell people, “Hey, this is a conversational interface.” We just have a conversation.
In fact, conversational UI and AR are just two elements of a much larger paradigm shift that’s actually much less daunting than the technical terms would have you believe. Through what we call “natural interfaces”, we’re finally seeing technology learning our language, instead of us having to learn its — making it better at anticipating our needs, understanding what we’re saying and allowing us to focus on the real world, instead of being glued to our screens.
A conversational UI also allowed people to stay more rooted within their own place, as we could easily overlay on top of the camera view. No need for pop-up notifications or onboarding screens, just one simple voice: the voice of IKEA.
Developing this voice was key. While we would start with a text-based conversational interface, we knew we’d be laying the groundwork for what’s bound to become a voice-based one sooner or later. To nail this tone-of-voice we turned to the core of the IKEA brand: its democratic character. One of the many examples of this is how it refer to its employees, all 200,000 of them, as co-workers. We felt that IKEA’s voice shouldn’t speak from one individual point of view, but rather as the co-workers; as “we”. Additionally, we decided that the tone of voice could never be demanding, but instead should be suggestive, both in giving advice and making requests.
While the conversational UI carried the bulk of communication within the app, we still required some traditional UI elements to ensure a low barrier to entry. Through Norgram’s detailed sense of brand language, these elements ended up feeling as true to IKEA as the furniture people were placing.
For their inspiration, Norgram looked to where most people had interacted with IKEA’s visual identity: the stores. Sebastian elaborates: “Obviously the app was anchored in IKEA’s logo, colours and typography. But beyond this we developed an iconography that functioned digitally while feeling reminiscent of the universal signage within the IKEA stores.”
CONCLUSION
In the end, despite the innovations and tech involved, human sensibilities were what guided the creation of IKEA Place. Accessibility to the many people and their needs is what formed the app’s main purpose. Without much time to prototype or reiterate, our design approach was almost philosophical, as we thought through natural behaviour to shape the experience. At SPACE10 we see this approach as a necessity, especially when the interfaces we’re designing become more and more contextual, fluid and integrated into our lives and the physical world.
But there’s one human aspect of this project that was most essential of all: trust. This many collaborators, decentralised, stretching across countries, languages and disciplines, all the while working on an impossible deadline without guidelines, would mean the downfall of most projects. For IKEA Place, it was this secret weapon that made it all come together.
Of course, this trust wouldn’t have been possible without IKEA leading the charge. No other company would have been able to pull off, or allow us to do what we did, in the time frame we did it in. “IKEA let us go at an incredible pace and allowed us to make quick decisions that kept the project moving,” says Mathias from Norgram.
Trust enabled us to take advantage of our differences. It allowed us to push each other into new territories and beyond our conventional thinking, from Plan8’s insights into sound, to TWNKLS’ experience in augmented reality, to Norgram’s knack for design that touched every bit of the experience. “Because of everyone’s background, a lot of new ideas came to the table that would’ve otherwise never been there,” says Daniel van der Schoor of TWNKLS. “Take the conversational interface, something we, as a pure dedicated AR company, wouldn’t have immediately thought of.”
For us, the most important takeaway is how much can be achieved when having a little more faith, letting go of some control and simply focussing on the overall vision. Of course, it goes a long way when that overall vision is one proven to steer a brand in the right direction ever since its founder first wrote it down 74 years ago: “Creating a better everyday life for the many people.”
