Invisible mechanics in virtual reality. How experience is created.

My name is Pavel Sovushkin. I create the experience.

My instrument — virtual reality.

In 2018, together with the ArtVR team, I made a project for the Tretyakov Gallery.

The project tells about the artist Natalia Goncharova. The project is on permanent display right in the building of the Tretyakov Gallery and after the experience in virtual reality, the visitor can walk through the halls of the museum and see the real paintings of the artist.

Natalia Goncharova — an artist who made a great contribution to the Russian avant-garde; Most of her paintings are still lifes.

The task of the ArtVR team is to tell about the artist, securing the story through the actions that the Player will perform in reality.

We decided that the player can create still lifes in the style of Goncharova. We took objects from her paintings and recreated them in virtual reality.

We reconstructed the studio in which Goncharova worked, repeated the details of the design down to the window views.

Then we put all the objects with still lifes in the studio, and put the canvas in front of the player.

The player can take an object into virtual hands and expose his own still life.

From now on, I will analyze one step of our thoughts in a little more detail.

So, we are faced with the question: “How will the image appear on the canvas?”

Let’s go back to the methodology. We stand in different positions, look at the task.
Story: In the artist’s work there is a gesture, which is called the method of sight. Surely you saw him. The artist puts a pencil in front of him and, having closed one eye, adapts to the size of the object in such a way as to transfer it to the canvas. The gesture is recognizable, corresponds to the image of the artist, we try to apply it in mechanics.

Mechanics: We make a prototype. We choose to call the mechanics of the Touch Pad under the thumb. The player presses the button — and a frame appears next to his virtual hand. Anything in the frame will become part of the picture when the player releases the button.

Questions arise from the aesthetics. The result gives the impression that we have a camera frame in our hands, and we need a more artistic perception of this action.

We solve the problem of Aesthetics through Technology: we write a shader, which stylizes a picture at the moment of choosing a perspective in a pencil drawing. Putting it all together, testing.

The result is interesting. Mechanics and Aesthetics work. We check how everything looks from the standpoint of History: Experts of the museum tell us that the placement of still lifes on the table is typical of Goncharova’s works.

OK. We return to the solution of the problem.

Technology: The image fell on the canvas, because an invisible camera was tied to a virtual hand, which received an image of the environment and superimposed it on the canvas. We disconnect the invisible camera from the hand and place it in the room so that the camera looks at the table. Customize the angle. We put in the studio a canvas on the easel, on which everything that happens on the table is reflected in real time.

Mechanics: Now the player can take objects that are in the studio with virtual hands, put them on the table and so create his still life. We are testing.

Aesthetics: Now, in terms of content, the image looks more like an artist’s work. But after all, Goncharova did not paint realism. Objects in her paintings are written in the avant-garde genre. How can we solve this problem?

Technology: We can make a duplicate of each object in the style of Goncharova. One object will be real, and it will be in the photorealistic environment of the player, and the distorted object will appear on the canvas. We make several objects. We are checking.
Some technical processes for creating a model take a lot of time, and some take even more. For example, you can distort the proportions quickly, but drawing textures is harder and longer. We consider how much time it will take for us to do the same work for all objects. We get a figure — we see that this technological solution does not suit us.
How else can you make all the objects in the room in the style of Goncharova? You can write a neural network. We try. We teach the neural network at the works of the artist, embed it in the Unreal Engine workflows (the program within which the project is going to be in virtual reality). See the result. It turns out interesting, but the miscalculation takes too much time. From the moment a person put an object on the table in front of an invisible camera, to a modified image on the canvas a delay of 35 seconds. We transfer part of computational processes to the cloud — now the miscalculation takes 2 seconds. It is still a lot. We need the player to immediately see the result of their actions. How can we solve this?

Mechanics: Add a pencil shader to the canvas, which we wrote earlier. The pencil shader is calculated in real time — without delay, and when the player realizes that the objects on the table make up the still life that he wants to see, we will remove the pencil shader.

Good. And how do we understand that a player is satisfied with his picture and decided to publish it? The easiest way is to attach the “Publish” button somewhere near the canvas, and by pressing it we would show the picture. But such a decision from the point of view of Aesthetics is not entirely harmonious. All these buttons, virtual menus, and so on do not fit in with the work of the artist. What elegant solution will satisfy the Aesthetic side and solve our questions in Mechanics?
Add to the studio Goncharova another physical object — “Artist’s Brush”.

When the player is satisfied with the exposed still life, he will be able to pick up the brush that lies next to the canvas and, having touched the picture, begin to show the picture in Goncharova’s style.

For mechanics, the solution with the addition of a brush carries another positive point. When the necessary action in virtual reality is not limited to “pressing buttons,” it offers the player movement, which ultimately works better for diving. Such mechanics work especially well when movements are recognizable. Stretching a bowstring bow, swinging a sword, waving hands while running — all this is an example of recognizable movements that formed the basis of many games in virtual reality. I called a similar design approach Animation Mechanics.

Taking the brush in hand, the player begins to paint over the canvas. Leads a hand, making smooth movements in the air. As we saw later when the project was launched on the sites, this mechanic worked really well. Visitors to the museum, passing by the exposition with virtual reality, saw people who were experiencing. Even seeing what exactly a person does in virtual reality, visitors recognized their gestures and also lit up with the desire to try. I am not an expert in how the human brain works, but I think that this behavior is related to what is called “mirror neurons”. A person sees how the other person does something, represents how he would independently make these movements, and this attracts to the project, and then works on the quality of immersion, inside the experience.

Let’s go back a bit to the project development stage. We added a brush with which the player will often interact. How will the brush look? Indeed, in the physical world in the hand of the player a joystick. It is necessary to do so that, holding the brush in the virtual world, the player does not experience contradictions in sensations in reality.
We make several variants of the brush at once, each variant in one degree or another repeats the thickness or bend of the Vive joysticks.

We test all the options and choose the model that feels the most appropriate. Perfectly.

Now the question arises in Mechanics: “How will the brush behave when interacting with the canvas?”. Let me remind you that all the objects in the studio Goncharova — virtual. When we bring one object to another, we do not feel resistance, as if it happened to physical objects in the real world. The brush passes through the canvas, and it causes strange sensations, because it contradicts the expectations of the player. How can we solve this problem?
We look at the decision from the perspective of technology. We make a mathematical model in such a way that when a player brings his hand through a virtual canvas, the brush remains on the surface of the canvas. The hand follows the movement of the hand, even if the hand is physically located behind the surface of the canvas. The player has the impression that objects in the virtual world have physical properties. How else can we enhance the sensations of the interaction between the brush and the canvas?

Add a joystick vibration at the moment of touching the canvas. Having understood the technology, we associate the force of vibration with the parameter of speed with which the player brushes the canvas. So we enhance the feeling of writing a picture.
Can we somehow strengthen this moment? Let’s look at the situation from the standpoint of History. We need to highlight the moment of painting the picture emotionally. I would like to create a feeling close to the inspiration that the artist is experiencing, creating a picture.

We write music. In our projects we use an approach called “Adaptive Music”. It changes depending on what the player is doing. At the moment when the player touches the picture with a brush, the emotionality of the music increases, the faster the player leads with a brush, the more influence he has on the music.

Well, it turns out well. Can we enhance the feeling of painting a picture in some other way? We look at the question from the standpoint of aesthetics. We have already agreed that when a player paints on a canvas, the pencil shader dissolves, and after him color painting appears in the style of Natalia Goncharova. But how exactly does the pencil sheet dissolve? Investigate the issue and make the dissolution so that it resembled the spread of watercolor paint. Trying — it feels great!

We have come a long way. There are only a few mental steps left to make a great job as a result. Through the site at the Tretyakov Gallery every day are 250 people who create their paintings. At the end of the experience, we give the opportunity to save the picture and publish it on Instagram. We need the pictures of each player to be different from each other. How can we do this?
Let’s look at the task through Aesthetics. On the table in front of the player there is a tablecloth. We make it so that every time you start a project, the drawing on this tablecloth is different. Anything else? We see that a person, passing through experience, often takes objects for a still life that are closer to him. We make additions to the logic of the project — at each launch we “put” objects in different places of Goncharova’s studio. As a result, we get different pictures from each other.

So, we are almost done. We look at the whole project, at all the steps that we have gone through, and think about how we can use this knowledge to once again improve the aesthetic side of our pictures with you.
Remember the step in which we tested, how best to stylize realistic objects under the work of Goncharova? Fully create models with textures left too long. But to make a copy of the object and distort its contours took quite a bit of time. We make copies of each model that are invisible to the player, and when the neural network begins to redraw a realistic picture under Goncharova’s style, we give it distorted copies of objects for processing. So we are one step closer in aesthetics to the desired result. In order for our pencil shader to bring us even greater benefit, we mix it in with the images that the neural network works with — and we end up with outlines around objects that are typical of Goncharova’s painting.

Brush strokes — and we have created an experience with you. Thank you for thinking with you. Come to the Tretyakov Gallery on Krymsky Val in Moscow and create your own version of the painting by the great artist.

You can learn more about the project on the ArtVR studio website.

WOWlab: wooowlab@gmail.com