The Problems of using Data from Virtual Reality Experiments

Is Research Data from Virtual Reality Experiments Automatically Invalidated?

KITATUS in Kitatus
Feb 12 · 14 min read

After a painfully long birthing period, the current generation of Virtual Reality devices have finally entered the mainstream. What was once reserved for curious developers and early adopters is now available for researchers, consumers and curious individuals.

The early hype of the wonders of Virtual Reality has already begun to die down and from this lull, inquisitive minds have begun to push and prod at the extended pools of activities you can perform using virtual reality.

During the inception of the current incarnation of virtual reality, a limited focus was put on uses of virtual reality outside of the video-game and film related industries. Medicine, training and education were all topics toyed with at a theoretical level yet in terms of attention, all eyes were mainly plastered on the gameplay extension possibilities more so than the practical, non-game non-film applications.

Virtual Reality promised us new experiences and immersion simply not possible via other means. Has it succeeded? The jury is still out on that one. Image source: Oculus / Facebook

There are a few reasons for the limited focus on virtual reality and they are not essentially restricted to non-game non-film projects within virtual reality. In fact, these issues stem from the limitations and issues at the core of this current manifestation of virtual reality.

Limitations of Virtual Reality

There are two key areas in which Virtual Reality hasn’t quite hit the mark with the current incarnation of virtual reality — immersion and interaction. These aren’t the sole reason why all research utilizing virtual reality should be invalided in my eyes but they are two key areas that should not be overlooked.


One of the marketing buzzwords that has followed Virtual Reality around since the dawn of the idea of virtual reality devices has been “Increased immersion”. The big selling point of VR has been the idea that you can only get so immersed in 2D images on a television screen whereas with these goggles, you can be transported into another world.

The film Ready Player One features an immersive virtual world known as the OASIS, accessible with technology that is designed extremely similarly to the virtual reality technology available today. Image source: Entertainment Weekly / Warner Bros.

The key ways that these recent virtual reality devices are more immersive are:

  • The goggles block out external visual stimulus, forcing you to focus on the images right in front of you
  • Sensors such as accelerators and gyroscopes built inside of the headset track the position and rotation of the user’s head, sending this data to the rendering device to offer the illusion that the user is looking around the virtual world
  • Most virtual reality devices offer built-in headphones or audio emitters that help overwhelm another sense.

This means that virtual reality goggles serve three main functions to facilitate immersion: They alter your visual and auditory senses and they also emulate your natural head movement in the virtual world. Whether or not these immersion techniques are successful (and to what degree) is open to debate.

There have been numerous infamous issues with current virtual reality devices that hinder the progress of immersion made by these techniques, such as the “Screen door effect”, where the lenses in front of the screen inside the goggles amplify the dead space between the pixels to give off the impression that you’re seeing the images rendered through a screen door.

A mock-up image of the “Screen Door” effect using the video-game Team Fortress 2 as an example. Image source: Reddit

Another key reported issue is latency, which is the delay between performing an action (such as moving your head) and the visual or auditory feedback from said action through the goggles.

There is a more generic immersion issue that affects virtual reality but extends far beyond VR solely. For now, we’ll focus on the issues specific to virtual reality before pointing out the flaws in the video-game medium as a whole.


There are a number of virtual reality options available on the market at the time of writing. The biggest ones available at the moment are the Oculus Rift, HTC Vive, Google Daydream and Sony’s Playstation VR. Each one of these options offers different control methods:

  • The Oculus Rift allows you to either use a gamepad or their premium “Touch” controllers that allow you to virtually place your hands within the virtual scene and move them around in 3D space.
  • HTC Vive uses motion controllers that also allow you to move your hands in 3D space.
  • Google Daydream is more limited than the previous two options and only features two interaction methods; a controller that allows 2D movement (translation but not rotation) and head-tracked context-sensitive actions (only applicable to specific applications).
  • Playstation VR allows for two different methods of control; either a gamepad or one of their Playstation Move wands that use a camera to track 3D movement (limited to a 180° angle from the tracking camera).
A brief overview of the many control methods available for the current popular virtual reality options.

Each of the most popular virtual reality solutions that are currently on the market offers a number of interaction methods. Sadly, each of these interaction methods has both positives and negatives:

  • Head-tracked context-sensitive actions — This is one of the most limited interaction methods in virtual reality. Sometimes it can be a challenge to naturally navigate and orientate yourself around your surroundings in virtual reality when depending on what you’re looking at, an unexpected action might be performed.
  • Gamepad — Using a gamepad in virtual reality means that you can control your avatar or character in virtual reality using a similar control scheme to conventional video-games. The upside of this is that if you play video-games (either casually or more intensely), than this method will feel more familiar to you. In terms of immersion, this appears to be the weakest of all control methods for virtual reality currently.
  • Touch Controllers / 3D Wands — The quality and depth of movement from the Touch Controllers / 3D wands differ greatly from service to service. By design, working with 2D controllers, such as the controllers featured with the Google Daydream are extremely limited and offer functions that are not too dissimilar to using a computer mouse to interact with the world around you. Wands that offer 6 degrees of movement (6 DOF) offer greater immersion, able to replicate exact wrist positions as well as rotational data in 3D space.
Degrees of freedom has a massive impact on virtual reality experiences. In emulating hand movements, rotation is a key component. Imagine only being able to move your wrist without turning or any sort of rotation whatsoever. Image source: RoadToVR / Wikipedia

There is currently no industry standard for control methods inside virtual reality. The most popular and common input interface at this moment in time appears to be the 3D touch controllers that accurately emulate your hand transforms in the virtual world.

The controllers in question are still quite limited. One of the limitations of touch controllers is that most of the currently available solutions do not track individual fingers, only hand rotation and position are supported.

Due to the fact that 3D touch controllers can only track hands, shoulders and elbows have to be dynamically assumed by developers using data transforms from the headset and the controllers — leading to certain situations where in-game avatar’s elbows, arms, shoulders or even whole upper bodies will convulse, snap into a new position or desynchronize from the user’s actual limb position.

An example of a person dancing in the video-game “VR Chat”. Notice the parts where desyncronization of his real life actions to his in-game avatar occur. You can spot these when the arms and legs of the avatar unnaturally deform. Source: Yske (YouTube)

The interaction methods mentioned are not the sole methods of interaction in virtual reality but they are the most popular of the commercially available options. Motion capture (either using a Microsoft Kinect or via a motion capture suit) is a popular interaction method among researchers and developers but interaction solutions such as motion capture require many different tweaks and changes on a person-by-person basis.

When you have to make tweaks and changes to the input method on a person-by-person basis (especially in regards to experiments), you’ll have a difficult time performing control tests — how can you have reliable experiments if you are forced to make adjustments to external parameters in an attempt to get as close to the baseline of the experiment for every person you perform the experiment on?

Limitations of Video-Games

Stepping away from Virtual Reality exclusive issues, video-games as a whole bring a number of important to note limitations — many of which can cause issues with data collected from virtual reality experiments.

Even though not all virtual reality experiments are video-games or even presented as such, these experiments usually share many of the same tools, programs and design philosophies that video-games also utilize in their creation, such as a 3D game engine (such as Source, Unity or Unreal Engine 4), 3D models and 2D texturing programs.

Due to the fact that experiments in virtual reality share a lot of the same DNA with video-games puts them at risk of the same issues and possible flaws that video-games also suffer from. This means that many of the hurdles that video-game studies suffer from are also a risk for virtual reality studies.

On the top is a screenshot from a moral study performed in 2014 and on the bottom is a screenshot from the video-game Half-Life. Both the experiment and the video-game shared many of the same tools in their creation. Image source: HCILabUdine / Valve

The two key issues with video-games that I feel can severely affect virtual reality studies and experiments are the abstraction from reality and the psychological behaviors of video-game players.

Abstraction from Reality

One of the hottest debated topics in terms of video-games that has persisted over the years has been whether video-games influence the behaviour of the player, such as if violent video-games make players more violent or aggressive in nature.

The very recent studies (at time of writing) on video-game violence have claimed that a direct link between Video-games and behavioural changes exist but these studies have yet to be validated.

These studies and the conflicting studies before them unite together in the theory that the player is segregated to the actions taken by their avatar. The recent studies point to a lasting effect of being exposed to violent imagery but do not directly conflict with the idea that players separate their avatar actions with their own.

Newspapers and studies into video-game violence directly conflict on idea of player detachment to their in-game characters and actions. Image source: Eurogamer / The Sun

This is a point of contention between these violence studies and newspapers. Newspapers tend to ignore the studies on player detachment to their in-game actions and avatars and instead make an erroneous link between the two. A good example of this is the Grand Theft Auto series of video-games.

The Grand Theft Auto series acts as a sandbox for players to take control of an avatar living a life of crime. The world has been crafted to allow players to steal vehicles, murder and perform dangerous feats such as drug trafficking. These are not the only actions players can perform in the series but they are the unique selling points that have helped propel the Grand Theft Auto series into critical acclaim.

Specific sales numbers for the Grand Theft Auto series (of even video-games in general) is notoriously difficult to source from outside of the publishing or development houses. According to collected sources over on the GTA Wiki, the series today has sold roughly around 174.7 million units.

The Grand Theft Auto series is a sandbox for players to live out their ideal life of crime. From stealing cars to selling drugs, almost anything is possible when it comes to ethically questionable choices in the world of Grant Theft Auto. Image source: DarkStation / TakeTwo

We can not take much from such volatile data due to variables such as people purchasing multiple copies, the data itself being unreliable as well as issues such as no way to track retention rates of the players of these purchased copies of games in the Grand Theft Auto series. What we can take from this data though is roughly 150+ million players have been exposed to Grand Theft Auto and the gameplay contained within.

From this estimation, that would mean that there are 150+ million people have been in a situation where they can murder, steal and commit various other criminal acts with no consequence to their psychical being. Most of these 150+ million people have not become cold-blooded killers, professional thieves or even commit common crimes.

It is a struggle to argue these facts just as much as it is difficult to disprove them. This is why studies and discussions in regards to video-games are debated quite significantly (especially as of recent) — gathering reliable data is a constant struggle in a medium that is constantly growing and changing to industry trends.

Jack Thompson famously debated the psychological effects of playing Grand Theft Auto. The BBC dramatized the debate in the docudrama “The Gamechangers” starring Daniel Radcliffe as Sam Houser, co-founder and president of Rockstar Games (the creators of the Grand Theft Auto series). Image source: BBC

One place you can look for data on subjects such as “Does Grand Theft Auto turn people evil?” is to talk to people who have played games from the Grand Theft Auto series. If they do not attempt to murder you, steal your car or commit various crimes against you — you’d be able to gauge quite easily if Grand Theft Auto has turned them into the killing machines that newspapers would have you believe.

We have established (basically and with unreliable data) that at a glance, it would appear that players have a clear understanding that the actions taken by their in-game avatars are not necessarily actions they would take in real life. There is a distinct disconnect between players and avatars.

From this connection, you could argue that people are more likely to act differently knowing that they are controlling an avatar and not themselves which could even extend to a subconscious level. Somewhere along the line, the player understands that actions taken in video-games are abstracted from real life repercussions.

Between the player’s desires and the video-game is the interface method. Is this layer of abstraction enough to alter what a person would do in specific situations? Image source: Business Insider

Until we can establish where this disconnect begins and ends, can data from virtual reality experiments be validated? External variables (such as this layer of abstraction between player and avatar) could affect and even misconstrue data collected from experiments, especially when player action is involved.

Psychological Behaviors of Video-Game Players (Diminished consequences)

Branching out from the core idea of the divide between player’s in-game actions and reality is the idea of diminished consequences. People will happily shoot enemies (and sometimes even “innocent” people) in video-games without feeling a drop of guilt. This is due to diminished consequences.

Even if consciously or subconsciously, players understand that committing violent acts and other various acts that are shunned or outlawed in the real world in a virtual world will not get them in trouble with the law, going as far to understand that there will be no real life consequence to their actions.

The video-game industry would be a lot different than it is today if players felt good for every gun they held or every bullet they used out of that gun. Image source: Electronic Arts / Respawn Entertainment

In fact, an overwhelming amount of games subject the players to learn that the only consequence they will face for their actions is that they’ll either win or lose the game they are playing. No FBI raids, no social exclusion for their in-game actions — just a simple “You’ve won” or “You’ve lost”.

Winning or losing is important in mostly every game, the states are ingrained into the very definition of a game (video or otherwise). The aim of a game is to achieve either outcome. Naturally, there are outlier to these rules with some games more intent on storytelling or otherwise but that is an argument unrelated to the point at hand.

The main point is that players (especially those who have grown up around video-games) understand that they will not get in trouble in real life for their in-game performance and actions and this embedded mentality has a high chance of affecting experiments performed in virtual reality.

Even people’s perception of games has the basic understanding of you will either win or lose. Image source: Bandai Namco

The correlation between video-game behavior and virtual reality behavior can be seen with the Illusory truth effect (as written in Frequency and the conference of referential validity study). The basic idea of Illusory truth is that if you’re told something over and over, you’ll eventually come to believe the statement — even if it is falsified information.

From a young age, we are taught that with games — you will either win or lose. Virtual reality experiments use a lot of elements that are known to even non-game players as game elements such as 3D models, textures, 3D rendering and gaming devices such as the Oculus Rift.

Currently not measured before virtual reality experiments is the exposure that a test subject has to the preconceived motion of what is a video-game and more importantly what they would define as a video-game. Because of this, there is no way to understand if the test subjects subconscious is playing a part in their decisions in virtual reality — thus affecting the outcome of these studies.

These days it is impossible to gauge who has or hasn’t played video-games or even understanding their subconscious understanding of video-games as a whole. Image source: The Daily Mail / Newsteam

If they have no problem shooting an enemy in the face in a video-game, why all of a sudden would they feel guilt doing the same thing but having the screen in front of their eyes instead of on a 50 inch television? When you factor in the Illusory truth effect and the possibility of obscuring the data, it makes it difficult to extract meaningful data on psychological patterns in virtual reality when dealing with “real world” situations.


You could argue that the points raised are invalidated with more recent studies that tend to use 3D videos instead of video-games or gaming engines within the studies. This, I feel, only further cements the points raised.

A 3D video has the least amount of immersion out of all the available options in virtual reality, having even more drawbacks than games or gaming engines. There is currently limited data on how immersed a subject is within a virtual reality experience because it varies from test subject to test subject.

This data is essential due to the fact that without it, you can’t tell if the actions taken by the test subject are what they would really do or if it is an impulse reaction to virtual reality or a hard-coded response to a video-game situation.

There are simply too many questions and too many issues present with the current implementation of virtual reality that doesn’t make it a worthwhile investigatory tool into psychology and the psychological responses of test participants.

Perhaps in a future incarnation of virtual reality, we can get the data needed from the test subject to understand how immersed they are prior to launching the experiment.

Even then, your posed with further issues such as their understanding of games and their subconscious reaction to a gaming situation — facts that will almost definitely affect the data extrapolated from virtual reality experiments.


A collection of game development related write-ups, break downs and a whole truckload of games industry analysis. There’s even tutorials and guides to help take your project to the next level!


55 claps

Written by


Hello! Kitatus Here.



A collection of game development related write-ups, break downs and a whole truckload of games industry analysis. There’s even tutorials and guides to help take your project to the next level!