3D will be NORMCORE
When we stop thinking about designing in 3D or 2D, then we will have entered a new era of computing, spatial computing.
This is summary of a talk I gave at AWE 2019 in California.
NORMCORE was a fashion term invented around 2013 in rejection of the over-hyped senses of I have to look this way, I have to be so fashionable, look at me on Instagram, look at me on Facebook, look at me everywhere. People rebelled and said collectively: I am just going to be normal, boring, as much as you can be. So it became a fashion statement, and a bit of a political statement as well.
This fashion mindset is how I think about 3D. It’s hyped at the moment, but it will become dull, just normal. Technology follows this trend and becomes NORMCORE after awhile. And so will spatial computing and all the other fantastic technologies we have today, each will become just normal.
I like to look a little bit ahead for inspiration, and I find that movies are one of the best ways for us to … well, for me to be inspired about ways that you can look into the future and see maybe how spatial computing is going to evolve. There’s a film called Her by Spike Jonze.
As you can see from the film, the textures and the fabric blend in with the technology itself, it becomes this seamless experience. The film looks into the future portraying technology that it doesn’t have to prove itself through its sophistication. It doesn’t have to be like a smart sleek iPhone or stand out against other devices that are around you. It blends in with your world, a little bit like a 1920s cigarette case or cardholder or something like that. Here it’s even saying it’s looking to the world as a more straightforward place. Maybe the technology is more complicated, but it actually simplifies the experience we have with technology.
What do we want technology to look like?
There have been so many terms for spatial computing. You could also think about it as part of it is augmented reality. Part of it is in reality, part of it is a voice user interface, part of it is a gesture, part of it ambient computing, sometimes it’s called natural computing. I particularly like the term spatial computing because it encompasses all the technologies that can live in the 3D world that we live in.
When I’m referring to 3D, I’m actually alluding to the future. When I think about 3D, it’s not just in its visual form as we see it today, but also when sound is spatialized, when we use gestures in all three dimensions, and when we begin using our proprioception, the sense of understanding space around us.
I want to show you other ways to view spatial computing through two people’s eyes. The first one is Adam Savage after he’s tried on a magic leap for the first time:
“What really, really blew my mind, was there’s a whale. Now I know that in the preliminary press hype for magic leap, there was a whale diving in a gymnasium and everyone was looking at it. The whale [you see through] the magic leap doesn’t look quite the same. But the experience I had was really transformative because I turned it on and I could hear the whale, but I couldn’t see it. And I’m looking around my office for it, and then it swims by my windows on the outside of my building. So, the [magic leap] has scanned my room.
It knew my windows were portals, and it rendered the whale as if it was swimming down my street. I actually got choked up because I haven’t had such an intense experience of that bifurcation of knowing that something is fake. Yet, it feels so real, and in that personally feels to me like the future and the possibilities in this, of telephony, remote communication, collaboration, get me really, really, really excited.”
The other perspective I’d like to share is by Kenji Matsuda, who is known for his films. Kenji described the future a bit like when Frodo puts on the one ring, and the one ring doesn’t take Frodo out of the world, but it shows him the world from a different perspective. There’s a cautionary aspect to this: If the one ring can control the way you perceive the world, who is going to own that?
We’re at spatial computing’s early stages, and we need to talk about it.
It’s like AOL of the desktop internet, that made way for Google. Then we had Facebook and WeChat, and the mobile internet. And now we’re heading into a new computing platform, and asking who will own the one ring for it, or should anyone? These questions are being continuously debated in our communities, and I would encourage you all to join the debates because we need more people. There’s not enough focus on this right now.
I’m the head of Emerging Design at Adobe, and I’ve been working in augmented reality for about 10 years now. I co-founded a startup called Dekko, and the goal there was to try to create the operating system for the world. No small vision. And now I’m supported, having gone through working at several other companies, all in the same sort of vein. My personal love is how do we communicate better in the future. And I’ll talk more about that later.
The three trends of 3D.
First, 3D is entering our real life, and it amplifies creativity, connection, and productivity. Second, 2D and 3D are converging, and convergence requires new tools. Third, emerging roles are going to shape our future.
3D entering our daily life has three primary purposes: One is amplifying creativity. Also, it improves communication in that it makes online communication closer to real-life communication. And it can improve productivity through new interfaces.
Let’s look at how 3D can amplify creativity: a Bjork concert. She works with a fantastic artist called Tobias Grumbler. If you’re a motion designer, and you want to be inspired, Tobias Grumbler is the guy. Bjork really wanted to create the 3D effect all the way through the stage and into the crowd, and I think you’ll see how she does that in the next two videos.
Another way we see 3D enter our world is through Avatars. For example, Lil Miquela. She’s got 1.5 million followers on Instagram. She’s an invention of an advertising company called Brud. I see her as a personification of the people who run the company, and I really like her strangely. I talk about her even though I know the people behind this brand, and I’ve spoken to them about this. I see them in her.
3D is being used to creatively and more effectively communicate complex topics. For example, those working to fight Malaria have used spatial technologies to visualize hard to comprehend data sets. The team is using what we might call visual effects, but for a great purpose. Maybe you can think about how visual effects in your next project can be put into the real world.
A company called Spatial.ai is building 3D tools for more effective communication. They’re trying to convert thought into visual imagery that can be shared collaboratively in a way as close to reality as possible. Remote collaborators can actually see the avatars of each other, put notes up on the wall as they would in a real conference room. Each collaborator can all walk around an object, point to things, and use their sense of proprioception, all while sharing a space with each other even though physically they are far away.
What does this all mean? There’s a lot to design.
At Adobe, I created a team called Sensory Design to begin understanding our role as designers moving forward into a spatial computing world. This is a diagram that is taken from somebody who works at Facebook, Margaret Stewart. She’s saying as designers you’re moving from pixels to perception.
We’re moving from a static image to a living product, and we’re responsible for how that product affects people. Our responsibilities as engineers, designers, and everyone involved in building a product, have exponentially grown in number. Products can reach exponentially more people today and affect their daily lives in all sorts of ways. The Sensory Design Team is tasked with thinking about these things.
How do we design for spatial computing? We have material design guidelines for designing mobile. We’ve got other guidelines for designing everything else from buildings to web pages, but little to nothing for spatial computing. We decided to start simple and pick little projects to begin understanding how we should be designing for the future. The first project we called the closeness study.
The closeness study. How do you design a tool to deepen relationships?
We realize that while the mobile platform has brought us a lot closer together, it has also created new forms of isolation that, in hindsight, seem preventable. So our goal is to try to see how we can make communication not suck in the future. How are we going to prevent isolation and build relationships using spatial computing?
We gathered a group of couples, people who know each other really, really well. We wanted to know how to deepen existing relationships. We realized straight away that the situation a couple is in affects closeness. For example, if they knew someone else was listening in, the conversation became immediately shallow.
We came out with a list of aspects of communication that support closeness. Things like physical confidence: do you feel comfortable where you are? Belief in privacy: if you think somebody is listening in, then the conversation immediately goes shallow.
Designers building communication tools need to ask themselves if they’re making a person believe that they’re not being listened to and if they can trust the tool. If our tools are untrustworthy, we’re going to have a really bad future ahead of us. Relationships won’t get deep, and the world will be a little bit sadder for it.
We also found that the content people share together was as relevant as what they said to each other. Social media is good at sharing lots of links and sharing content, but it’s between lots of people and the content trends towards distractability. Building tools without rewarding distraction and instead encouraging users to share the content they wish to share could help.
People require a sense of Agency, a sense that they have a choice.
If you don’t feel that you can choose what to eat, where to work, where to go to school, then you have a hard time enjoying your life. The sense of agency is so important to us as human beings, we’ll move countries to have more of a sense of agency. But what we found mainly in this computing platform is the sense of agency is more important than it’s been in any other platform before. When you’re thinking about designing the next future tool in spatial computing, you need to give people the ability to change things. Move around, ask questions, have control over the privacy controls, and who sees what.
Spatialized sound makes all the difference.
The 3D sound was the most crucial thing our study participants experienced. If you want to improve any communication tool you have, spatial sound is the most significant improvement you can make to the quality of conversation. Similar to when you’re in a real-world meeting room with others, where people sit affects how well they are listened to. We don’t have that in any other video conferencing.
With spatial sound, you only need a small indicator to tell that a person is sitting next to you or across from you, which gives you a sense of ‘we are here together’ and helps the conversation. Sometimes in a large room, the shy people go to the back of the room and get ignored. With spatial audio, you can engineer it, so everyone has a better or more equal say in the group conversation.
Learn how humans behave with humans, not how humans behave with computers.
In the 80s, this was a new concept, and we all thought we were doing it, when we weren’t really doing it. We can do it today because computers can better understand us with sensors. With a growing number and quality of sensors, computers can understand what our gestures mean, what our facial expressions mean, what we’re saying in any language, or even just where we are in the room. All these things can be used to design tools that encourage good human behavior and better communication, productivity, and creativity.
I encourage you to think about the product(s) you’re working on and how can you encourage or enhance more natural human behaviors.
The convergence of 2D and 3D requires new tools.
For spatial computing, we need to connect the skills of 2D designers and 3D designers. New ‘motion design’ tools need to bridge 2D & 3D and skill & knowledge gap. Illustration, for example, has always been need across both fields and so together we can inspire new ways to add motion to illustration.
One tool along this vein is called Calvary. A disjointed workflow is a real problem in 3D tools today, and Calvary is attempting to remedy that. They’re working to make it easy and straightforward for 2D designers to understand as well as 3D designers. The program is based on procedurally generated animations, which means that you can go back and change anything without losing your animation.
Design challenges.
We know we need new tools, but we also need to understand how to design for the future using spatial computing. There are huge challenges, but there are also opportunities, I find them all really exciting.
Context. Remember the pixel and perception diagram. Well, we’re no longer designing the pixel to give to you, it’s the most effective, the best designs in spatial computing and augmented reality in voice in all the combinations of technologies that enter into the world will be if the content relates to the environment that is in, or it even is about the environment that it is in and reflects that.
Devices. The second thing I find, this is an ongoing debate, what will happen to our devices. We designed for the phone, we designed for a screen, with spatial computing, what do we design for? We already see, like with voice, it is shifting from our car to our desktop to our mobile or seamlessly. That’s an example of how spatial computing will start to become a bit more ubiquitous. But some people are saying, “It’s all going to be on the cloud.” And some people are saying, “It’s all going to be locked up in an ecosystem. Apple’s going to have the ecosystem and Google and whoever else.”
But I think there’s an in-between, I think we’ll be designing for devices, hearables and phones, and glasses, that are also connected to the cloud and use different features there from different companies and it’ll be this mix of both. So now, instead of designing for something, we know what the form factor is, we have to design for something that we don’t know what the form factor is. It’s flexible, and it relies on computation from the cloud, machine learning particularly. Does that make sense? There’s a really different paradigm shift for when you think about designing.
Complexity. And one last thing. It’s going to be very, very complex. Just the data alone that we’re going to get from all the cameras and devices, it’s going to be more than we can imagine. We need to break it up, modularize it. My challenge is if anyone can create models of how to think about the in components, I think if you try to work it all out in one go is not working. I can talk more about that later if you’re interested in knowing about that.
Silicon Valley has spent a lot of money, a hell of a lot of money and time and effort to move us into the new era of computing with hardware and infrastructure software, but very, very, very, very little on how it’s going to function. How are we going to use it? What does it mean for society? How are we going to educate the upcoming designers? What about the current designers? There’s just hardly anything there. So, that needs to change. I believe we need to spend just as much effort now is a time to spend on spatial design because we’re far enough along with technology today. That’s what I just talked about.
The stakes are high for designers in spatial computing.
I’ve taken this concept of circles of privacy from Timoni West from Unity Labs, Director of Unity Labs. Privacy is the main concern when we think about the ethical issues of design, as well as designing, so something is enjoyable to use.
The two extremes of privacy are public spaces and a private room. Public spaces are generally where you can speak and show things publicly, and private spaces are where you have complete control of who sees what. Between those extremes is a “controlled” space such as your shopfront, or your office that you work in. And an intimate space in your home, which you can decide who to let in or not.
We can describe these circles of privacy well in a physical context, but with spatial computing and its mixture of digital and physical, there’s a lot of layers and complexity. We are all trying to understand how to deal with some of the murky grayness there. We’re at the beginning stages of understanding how to deal with privacy issues. When you walk into a shop, and you can be tracked, and there are no laws against that. Are we okay with that?
Today, all the success metrics in advertising are based on the short term. Things like how many clicks, how many views, how many purchases, and other instant responses. But what if we looked at the long term instead and found positive ROI from useful metrics? What if we spent more time thinking about how to build long term relationships, perhaps by measuring the length of a friendship? Or we could measure emotional responses and determine if a positive relationship was formed. The challenges of doing these things well are numerous, but not insurmountable.
Be the change you wish to see.
I recently had a chat with Jon Lax, who’s head of VR/AR for Facebook. He believes the stakes are higher for the spatial design than the previous web or app design. Spatial design affects more of the human experience than flat screens ever could. Conversations like that are invaluable. Having conversations about how we should design spatial computing is essential in this moment of history. If we can talk about the challenges, then we can design solutions for them.
When I speak to people who do a lot of interface design, I say you need to be able to speak the language of business leaders, for them to listen and change. If you can learn about it, then you can get promoted to important roles. If you want to stop designing because you feel passionate about creating change, then stop designing and move into a real leadership role. And if you want to see things change in the policy and privacy space, well then you have to learn how to speak like a politician, you’ve got to learn how it all works, and you’ve got to work with it. You’ve just got to make sure you’re in the area where those decisions are being made if you want to make the change.
From many years of mentoring people, I’ve found it’s a hard lesson, but it’s a very valuable lesson. The challenge is to speak up and move up. I’ve talked about three trends, and I’ve talked about three challenges. The trends are 3D in real life, learning about human behavior, and applying it to a product or design. 2D and 3D will converge and will require new tools, like a new design language. And new emerging roles will shape our future.
