Art Direction for VR: Lessons Learned from Puzzling Places
In this post I will cover some of the stuff I learned doing the art direction for Puzzling Places. This includes covering similarities and differences between art direction for VR and traditional games as well as a bonus discussion about designing VR visuals that are friendly for 2D marketing and a discussion about art direction and AR.
Puzzling Places
First a bit of context to the game for those who don’t know it.
Puzzling Places is a 3D Jiggsaw puzzle game where you put together hyper realistic minature models of real life places from around the globe. The game design was centered around a sense of meditative flow and satisfaction.
We started looking around for an Art Director to help us realize the potential we were seeing in the game idea. Despite some good candidates, nothing we tried felt like a good fit. So, reluctantly, I decided to take on the role myself. In what felt like quite an intense year, I devoured whatever reading material I could get my hands on, and tried to map out a system where we can work on our visual problems and get satisfactory results.
Was the result satisfactory? Nothing is ever perfect, but I am quite happy with the work we did on Puzzling Places. For a year, Puzzling Places was the highest rated game on Oculus Quest store. It was a launch title for both Pico and Sony’s PSVR2 and for the most part our players seem to be content with all there is in the game. Here are some of the stuff I learned, I hope they would be of help to you too.
VR Is not All That Different
VR as a medium might be new, but it is mostly dealing with old problems. These problems are always a by product of our way being, how we see, how we think and how we interact with the world around us. In some aspects you could say designing visuals for VR sits somewhere between traditional games and architecture.
This was an important realization. Once you get there, you realize you still need to focus on the same points as everyone else when designing for VR. This is good news, because other fields are well documented.
You still communicate information with your visuals. You design for Readibility. Good communication is about building meaningful hierachy of information. In this hierachy what is important to you should be at the top, and whatever is not important (aka noise), is pushed to the background. To do this you work with affinity and contrast. Affinity groups objects and pushes them down the hierachy, contrast highlights elements and pushes them up.
You still design for emotional ressonance. All we exprience can remind us of our pre existing memories and brings us to exprience the emotions we associate with those memories. Using your shared expriences with your users, you try to emotionally align what you are saying, with how you are saying it. A useful tool to build a map of these associations is to gather references, make mood boards and get out there from time to time to touch grass!
And lastly you still design for visual interest. Visual interest is all about having a lot of variation in your visuals to keep your viewer engaged.
On top of this comes the concept of harmony and chaos. A lot of unrealated variation and bad hierachy of information gives you chaos, which you rarely ever want. Chaos is when your brain looks at something and has no idea what is going on, harmony is about when the relationships between the visual elements are simple enough that looking at them results in understanding.
Last but not least, you are still designing in a team. People of various skill sets come together to create a product. That means you have to deal with various stakeholders with various constrains while also designing something that ensures a certain outcome within the constrains of a business. In short you have to deal with workflows, systems and politics.
The above was a condese summary of my Introduction to Art Direction for Games blog post, which is a required reading for this one. I won’t be further repeating the stuff in that blog post here.
So in short, making games for VR and AR, you still need to make sure you nail all the above. Your player is confused about what she is supposed to do? Map out at which section of game loop she is in and what needs to be communicate with her. Then come up with visual solutions on how to achieve that. Your game doesn’t feel how it plays? Find visuals that share the same game play feel, and extract learnings from it and apply it to your VR visuals. You find yourself constantly in design by comittee meetings and esclation and control from third party stakeholders? You have design politic problems, allocate resources for a person to rework workflows.
VR is Somewhat Different
Not everything is the same. If it was, there wouldn’t be this follow up blog post for VR. So what is different?
Player Agency is High in VR
Compared to films, traditional games already offer a much higher agency in how the content is viewed. This is a source of a lot of headaches.
Compared to traditional games, VR games offer an even higher degree of freedome. The player has full control over the position and rotation of the camera, as well has her hand and whatever the hand can hold. Games that support hand tracking or match mouth movement or allow climbing expand these possiblity space even further.
This means players can clip through every geometry in the game if they wish to, and there is not much you can do about it. Because controlling the camera position in VR is a result of direct embodiment, messing with the players’ ability to move their own head could lead to horrible motion sickness. Or the player can choose to play the game with a tilted neck or crawl through every level. These obviously have drastic effects on the visuals of your game.
The more perspectives and mods of interaction the player can have with your game world, the more scenarios you need to design for. This means more work and less optimized solutions.
I talked in the previous blog post about how higher player agency forces you to consider three things: First, does your player actually need all the agency? Second, prioritize the points of view that are most important and third, focus your effort on aspects of design that have more of a universal impact on all the different scenarios you need to design for. That advice is still valid in VR.
Limiting player agency in VR is considerbly harder. Still doable, here are some methods we use to achieve that.
We have a locomotion system that some players use to get around. I say some, because we activly don’t put this system at the forfront. We made the decision early on to design the system so that most people play seated, facing forward. Our desire was that people neither move around nor rotate around much. We made this possible by designing everythign else UX wise in a way to enable people to comfortably play like this. Expressed in other words, we limited the player’s interaction with the game world to a frontal facing 180 degree view.
Limiting the player to this type of interaction with the world doesn’t come to a VR developer naturally. After all you are in 3D! Why not place things all around. However beside the simplicity this design decision offered our UX, it also massivly simplified our art direction efforts. Now we just had to optimize for one point of view!
We still limit the locomotion within a circle, which has a radius of few meters. The players can of course still stand up and physically walk out of the area, but qualitive and quantitative evidence shows that most people play VR in a limited room space. Remember point number 2, and prioritize! If only 1 percent of the players even have that physical possibility, put your focus on the 99 percent.
This is of course the most straight forward way of limiting movement. But there is more!
Early in the prototyping of surrounding environment in Puzzling Places, we tried a bunch of different stuff. From realistic and contextualized enviroments to stylized and contextulized, to abstract. With themes ranging from real world, to fantastical locations. Very soon we realized that putting interesting stuff in the environment is not a great idea!
By putting points of interest outside of where the player should be, you create an affordance for the player to go and explore! That is not a great outcome, because you don’t want your player to go there. Now you have a mismatch between your player’s desire and what you want to happen.
Hence we decided to massivly simplify the environment to get rid of any affordance to explore. To satisfy visual interest, you can still add visual variation. However these variations shouldn’t have a strong semantic associated with them that would call the player to action.
Another example that does this in a more subtle way is Walkabout Mini Golf. There are stuff out there in their maps, but the further away you go from player center area, the less interesting stuff there are to see. The mountains in the background are clearly decoration, and all the visual action is happening in the areas around you.
Regarding prioritizing what mods of interaction or perspectives are most important, you should closely observe how your players play. Just like in city planning, interior design and architecture (which offer an even higher degree of freedom), most people use a VR space and VR objects in the same ways again and again. This is related to the affordances, which the world and the objects create and our mental model of how we are suppose to interact with them. You walk through a corridor, look through a window and sit on a chair.
Last but not least, keep in mind that the fidelity contract you close with your players doesnt have to include everything VR can offer as a medium. Does this object really need to be movable? Do you really need to be able to break that? Accepting sensible constrain is often what differentiats the bad from the good outcomes.
Designing for Spatial Perspective
This is probably the biggest visual change. Depth on 2D screens is an optical illusion. There are various techniques painters use to create depth, the three common ones are foreshortening, areal perspective and layering elements. Foreshortening leads to parallex in movement, which for moving images greatly contributes to a feeling of depth.
In games we simulate foreshortenin gthrough the act of projecting 3D geometries on a 2D canvas using a 4x4 projection matrix with perspective divide.
Areal perspective refers to the effect Rayleigh scattering has on objects further away from us. It causes blue and greenish objects to loose texture detail the further away they are from us, and everything to blend towards a tint of blue as a function of the distance to the viewer. Various games simulate this through various ways, the most baisic being through a fog function in the fragment shader.
VR and Stereo-AR has another type of depth preception. That is real stereoscopic depth that comes through the fact that each eye is recieving a different image. This is still not hunderd percent “real” depth, since the VR headset lenses are still not light field displays. You are still looking at two screens.
However it is still a different type of depth you need to design for. It is an added dimension to the visual attributes your brain uses to classify and understand image elements (refer to the previous Art Direction post for more details).
What does that mean? That means everything we design for in the other visual attributes, you need to do here. Contrast in depth (the actual position of the object in relation to your eyes) highlights an object and makes you notice it. It increases readibility. Objects that are closer together, have comparitivly worse readibility and get grouped together.
If you have a lot of layers in your depth, things can get chaotic. If the distance between things are more regular and have simple mathematical relationships, you induce a feeling of harmony. Beside readiblity, emotional resonance and visual variation still play the same role they do in the other attributes such as color.
One thing to note regarding the depth attribute is the effect it has on things like perceived size and size differences. Of course a spatial element is added to these. That means contrast in size and how size relations feel are enhanced in VR. In Puzzling Places, the statisfactory feeling which the mini models create is not reproducable on 2D screen, except through visual tricks like depth of field and projection techniques. Size is not the only one. This effects all attributes to certain degree, for example color gradients feel different over 3D topologies than they do over 2D projections.
One thing we learned over the years is that how strong the feeling of depth is, is interestingly enough also depended on the headset’s resolution (beside the obvious things like IPD of the lenses). Regardless, at a certain distance the stereoscopic effect of VR headsets goes away, and you might as well be looking at a 2D screen. This point is around 15 meters, everything beyond this point is perceieved as 3D because of the old reasons, mainly forshortening and areal perspective.
Realizing that depth is basically another attribute to design for helped me greatly with Puzzling Places. Here was the constrain we had for the visuals in Puzzling Places. The game featured highly realistic puzzles of various locations. These locations had vastly different themes. Anything from a kimono mini model, to an industrial site. So the visuals had to play nicely with realism, as well as accomodate various subject matters. That ment the environment itself shouldn’t have a strong statement which would conflict with certain subject matter.
We wanted to conceptually bring the players as close as possible to the subject matter they are puzzling, and not distract them with other stuff.
At the time I was having one of my deep dives where I consume whatever I can get my hands on regarding a topic. This time it was Japanese Woodblock prints and Japanese and Chinese traditional paintings. Different compared to western paintings, chinese paintings attempted to capture the esscence of an object. Having its root in caligraphy, chinese (followed by japanese and korean paintings) attempt to portray what defines a core of a subject matter through an expressive arrangement of visual elements.
A tradition which woodblock prints followed was the concept of eliminating space. The idea was that space itself is a concept that the brain needs to parse and understand. By removing space in the painting, you bring the subject matter closer to the mind of the viewer, and thinnen layers between the viewer and the subject matter. This sounded perfect! Just what we needed in Puzzling Places!
Woodblock prints achieve this spacelessness through various ways. Two which were specially interesting for VR were: limiting the number of layers (depth queues), and eliminating the horizon line. In film language woodblocks that don’t use the single point perspective typically go for a strong flat space or limited space as opposed to deep space. As for point two, they just place gradients on the backgrounds.
We went ahead and replicated that in Puzzling Places. This change was also not obvious from the begining, since in VR you usually like to put the grid to enhance the feeling of deep space. It is 3D after all! However we immedietly felt the advantages of the approach. The background started disappearing and playing in the environment felt lighter.
You Design for Asymeterical Viewing Condition
When you are designing 2D games, your marketing materials and images shared from your game aligns with the player’s exprience playing the game. In VR you have an asymmetry. You market the game on 2D, but the game happens in 3D.
It has happened quite often that I look at the trailer of a VR game and think it doesn’t look great! Then I checkout the game and change my mind. This difference comes from the difference between the real depth discussed before and its projection on 2D screen.
You could look at a scene in VR and think it is quite harmonious. Everything relates with eachother and you can easily read all elements and their relation. Take a picture from it, and the depth differences which were distinguishing the visual elements from eachother disappears and the image becomes chaotic and unreadable.
This means when you are designing VR visuals you have to design for both scenarios at the same time. You can’t just rely on depth difference for readibility of the objects. You need to add areal perspective and vary the colors. You should also try to avoid placing too many objects along the view direction. This way whoever takes a screenshot or a video from your game would get a good looking footage. For your official marketing material you would of course frame your game world the same way a photographer frames a chaotic city, finding the perspective that offers order!
In Puzzling Places we did all the above by designing a very simple environment. Limiting depth queues along the main view direction which majority of the game play happens meant that the game’s visuals are still readable on 2D screen. We did this partially by emptying the space infront of the player, and not placing anything on the ground.
Techincal Difference: Half the Budget for Twice the Work
Not much to say on this. You have to render two images and you need to render at a faster refresh rate than most 2D games. To top that off the most popular VR platform is a couple of years old mobile chip. So all techart tricks are welcome to realize your art direction vision, but you would need to compromise on may fronts for now.
What about AR?
AR and VR are usually put together in discussions. There are some similarities between stereo AR and stereo VR. Both having stereodepth ensures some similaries as one might expect. On the other hand the player agency is also high on AR. The player similary controls the camera position through her head movement in the case of headset based AR.
Some problems however are even more severe in AR. For one, what does the background of the player look like in a AR project? You don’t know the topology of the scene you are designing for during your design process. This complicates the matters for all aspects of visual design. The readibility takes a hit but so does emotional charge and the sense of harmony.
We did a bunch of stuff to mitigate this in our passthrough experiments.
Give power to the users: When there are too many special edge cases to design for, one option is to give the users the tool to adjust your visuals to their own need! We have a color picker in the game, which the users can use to adjust the environment colors to their liking.
We extended this so that the users can do the same for the passthrough environment. If your living room is visually messy, you might want to reduce contrast and darken everything down while playing Puzzling Places in AR. If your player space looks like a designer office, you can leave it as is.
Use classic techniques to Boost Signal and Reduce Noise: Poster design deals with a lot of the probelms AR has. It can’t control the environment the posters are used on. You can take some learnings from that discpilne. The core idea is to boost the signal. You strenghen the visual message you want to convey, until it stands above your expected background noise. Posters typically do this by including high internal contrast, simplified visuals, and graphically strong shape design. We do this in Puzzling Places by including internal contrast on our hand and ray when in AR mode. You can use anything that comes to mind to boost signal.
Instead of boosting the signal you can also reduce the noise. You can blur the background or eliminate complexities in any of the visual attributes, such as desaturating the background behind visually important areas.
Just like in VR, in AR you don’t have to use all which the medium offers you. Does the player need to see everything that is happening around her?
One sensible way to design is to always think about the value you provide to your players through any added feature. What desire of the player are you fulfilling? Why even make the exprience in AR to begin with? Running some tests with our players we realized that there are plenty of people who would play Puzzling Places in AR, even if they get a worse overall game exprience. Why?
They might have a baby they need to want to keep an eye on, or don’t want to kick their cat, or would like to talk to their partners while puzzling. Paying attention to what they are saying, it seems like not everything needs to be fully pass through. We experimented with movable portals to the real world, where the players could select how much of their environment they would like to be exposed to.
This means the player can again selectevly filter out the background noise she doesn’t need (good for the visuals), while still keep an eye on the kid playing with lego.
Design to the strengh of the medium: My last point for this post. Art direction can inform game design as much as game design informs art direction. Certain game ideas work visually better in AR than others. If your game play demands from you to interact with visually important elements that are hovering infront of you, chances are you will have art direction problems. After all, people typically have visually noisy backgrounds infront of them. This could include shelfs, monitors and decorations. But if you bring the point of interest down on the ground (or a table), the background’s visual noise is massivly reduced. For one the game elements are placed on a flat surface (reduces contrast in depth). On top of that floors are usually kept free so that they remain walkable, so not a lot of objects that also demand player’s attention through strong affordances and semantics. A good example of this is Angry Birds by Resolution Games.
That was all for this post. Thanks for reading, and if you have any questions you can reach me via various socials. You can find out more on my website https://ircss.github.io/
