Upside Down
Adrien Lafage, Anna Székely, Lukas Egger, Maxence Blanc
Introduction
Have you ever thought about how to turn right your car’s or bicycle’s wheel on your ordinary way to home? How much time does it take to you to type your name on the keyboard? Would these answers change if suddenly “left” became “right” and “right” became “left”? What if keys on your keyboard were flipped upside down? The human brain saves us lots of effort by “automating” our actions in order not to think about them every time. Brain scientists would say that the brain is adapted to these tasks. However, if something that the brain was adapted to is changed, the task becomes more complex. The brain needs to adapt to the new situation which sometimes can cause hard times to oneself. To understand how the brain adapts to new situations, how it learns new visuomotor coordination skills (when the visual perception is changed by some reason, some adaptation is needed to perform everyday tasks) neuro-, and cognitive scientists make numerous researches in this field with tricky experiments. Many of these experiments concentrate on examinations where people have to perform different tasks among manipulated conditions. This can mean that the examined subject’s perception is distracted, or simply things appear differently than they normally do.
To better understand how the brain receives information and perform accordingly, it is worth to see that information is gathered by the peripheral nervous system (nerves which encompass our body) and processed by the central nervous system (in the brain). The whole system, which collects information about the surrounding world and even about the body by receptors and the peripheral nervous system, is called the sensory system. Receptors get stimulated when some stimulus appears in the environment and the peripheral nervous system transfers the information to the brain. The brain then processes the information and if it is needed, transfers the adequate reaction to the peripheral nervous system to create a so called motor output, like movement of the body or activation of glands. The process of information perceiving, processing and reaction is shown in figure 1.
Humans were born to live in a world in which they have to face a lot of sensory input and solve different motor tasks (e.g. walking, speaking, writing or playing an instrument). However the human nervous system is able to adapt even to functional or structural changes. (Della-Maggiore, Landi, & Villalta, 2015) “In mammals, however, learning is a major inducer of adaptive plasticity: the nervous system translates new knowledge into long-lasting plastic changes that lead to the formation of memories. The mechanism by which these memories consolidate and resist degradation by newly acquired knowledge or simply decay in the absence of practice are major interest to the field of neuroscience.”(Della-Maggiore et al., 2015, p. 109). If one learns playing an instrument for a certain time, then stop practicing for a long period of time and starts playing again, it only takes a short time to get into playing well again. However, to learn something new, the so-called sensorimotor adaptation is a key procedure in human cognition. It is a “type of procedural (motor) learning, that allows maintaining accurate movements in the presence of environmental or internal perturbation by adjusting motor output”(Della-Maggiore et al., 2015, p. 110). This adaptation occurs daily in our everyday life (using new tools, wear new shoes). Today it is still a challenging research topic in the field of neuroscience and psychology how the brain adapts to changes when it needs to re-learn a well-known task to perform in a new way.
From the 19th century researches are conducted on the adaptation of the brain to manipulated visual fields. One of the most famous experiment is the so called “Innsbruck Goggle Experiment” conducted by Theodor Erismann and later by Ivo Kohler. Kohler used goggles which were inverting the visual field vertically by 180° for several days (including day and night phases). For the end of the experiment the participants were able to perform daily tasks, even riding a bicycle or drawing a picture in the same quality as without wearing reversing spectacles (Sachse et al., 2017). These studies give evidence how well our brain can adopt to changes and learn new ways of visuomotor coordination, which includes the proper processing of sensory input (e.g. a view) and the proper performing of the motor tasks (muscle movement).
Our goal by our later described experiment is to show how long it takes to adapt to a new visuomotor challenge and perform tasks in a similar level than under normal conditions. To gather these data we developed a platform game (where the player needs to move a character through levels without falling down or bumping into different obstacles) which gives the possibility to invert the screen by 90° and 180°, and also to invert the arrows (up, down, left, right). In this way we can also challenge the visuomotor abilities by the screen inversion and the motoric performance by the inverted arrows. This tool makes us able to test participants during visual and motoric distraction and also combining them. We are also interested to see the adaptation pattern of people who started with non-inverted game and how they adapt to the inverted game, and also those who starts with inverted and need to adapt to the normal conditions. In the following we are going to introduce a few experiments where visuomotor and motoric adaptation were tested to show which researches inspired our research design and realization.
Researches about visuomotor and motoric adaptation
A research group in 2018 examined the visuomotor adaptation by asking their subjects to play a videogame which was manipulated in different ways. The main finding of the study was that the subjects’ brain was able to develop kinematic and dynamic skill components. The first component was measured by moving object in the videogame with the help of arrows, while the second by shooting on different targets. They also tested the combination of these tasks (moving and shooting targets). The different groups of participants who learned the components separately or combined were both able to adapt to the new environment in a similar fashion. That means if they have learned the components separately the adaptation time to the combined task is similar to the adaptation of the participants who learned it the other way round. They also found out that “combined practice had transitory performance advantages in both skill components. This transitory advantage reveals that the transference limitation to combine sensorimotor memories is momentary and dependent on the global context or coordination of the task.”(Burgos, Mariman, Makeig, Rivera-Lillo, & Maldonado, 2018, p. 3847) That means that it is possible to transfer combined learned skills (e.g. playing the game with moving the object and also shooting on targets) to single skills (e.g. only shooting on targets or only moving an object). But the tasks have to be in a certain context and the transfer itself is only momentary.
Another study from 2016 examined how people can perform different locating tasks when the screen is rotated with different angles (30°, 75°, 150°). Participants had to move a pen on a digitizer without seeing their hands or seeing digital feedback of their movements on the screen before the completion of the task. The researchers found that adaptation to the 30° rotation is easier than adapting to the 75° or 150° rotation as subject in the first group tended to reduce their direction errors, while subjects in the 75° group usually slightly overcompensated the extent of rotation, and the adaptation was scarce, meanwhile the 150° group strongly overcompensated the rotation, and they couldn’t really adapt to the task, similarly to the 75° group. Furthermore, the researchers also found that looking directly at the expected target point of the place was a common coping strategy (Rand & Rentsch, 2016).
Another researcher group in 2013 designed an experiment which involved many different experimental versions. The main task was always the same: locate a point in a screen under the effect of some disturbance regarding the visual perception. They used a 20° rotation of the visual field and right-left reversing, both by a prism. The subject had to watch a screen and touch a point signed by an appearing and disappearing coloured light. After that, subjects had to touch the given point by one movement (without any correction of the movement after they started to move). After the trial the light appeared again to give feedback. The location of the target also changed trial by trial. For another group the target “jumped” into another place, so they were obligated to change their movement after they started. There were groups which could see their hands when they pointed the target and others couldn’t. Lillicrap and his co-authors also tested a long-term experiment when one wore goggles for 8 days which inverted her vision in an upside down way (Lillicrap et al., 2013).
Scientists found that those subjects who were tested on the 20° rotation with non-jumping target could adapt easily, and after testing also the non-rotated case, they found that it also takes time to return to the normal operation. With the right-left reversion there were subjects who weren’t able to adapt to this conditions, and after, under normal conditions their results improved a lot, while another group of participants could achieve an improvement, even though it wasn’t consistent, and haven’t show “after effects”. In the case of jumping target, the adaptation wasn’t really palpable. However, in the case of long-term experiment, when one used an upside-down goggle for 8 days the amount of adaptation was significant. After putting on the reversing goggles the subject was unable to perform even very easy visuomotor tasks, while after 8 days she was able to perform different tasks even riding a bicycle. Nevertheless, during the 8 day of adaptation the performance hasn’t achieved the level without the prism. 45 minutes after taking the goggles off the performance returned to the original level (Lillicrap et al., 2013).
Marotta (2005) and his co-authors designed a similar experiment as discussed above. In this case subjects had to touch some sign in the screen, looking through a prism which reversed the right and left sides and became opaque when they started their movement in order not to see their hands. In their studies Marotta and his co-authors also tried to find out whether the adoption is global or modular. Their results showed that the adoption is mainly task dependent and this knowledge is hardly enough for generalization.
During numerous experiment the sensorimotor adaptation also examined by tracking the eye or body movements (cf. Marotta et al., 2005; Rand & Rentsch, 2016) in our experiment we are going to analyse only the time needed for adaptation.
Methodology
A. Participants
For our experiments we randomly asked friends and people of our surroundings. All in all, we got 14 participants — 3 female and 11 males. The participants are aged between 17 and 28 years (mean 22,5 years) and had to do different experimental tasks. Therefor we created 4 different experimental groups, which will be described later. Only one of our participants is left-handed, all the others are right-handed. With a short questionnaire we asked all of the participants different questions before and also after the experiment. We asked them about name, age, right- or left-handed and their experience with platform games, reflex games and their playing regularity was asked. Only one person (= 7,1%) has more than 100 hours experience with platform games, 6 participants (= 42,9%) have played platform games between 20 and 100 hours. Four of our participants have played platform games less than 5 hours (= 28,5%) and 3 participants (=21,4 %) between 5 and 20 hours. The answers to the experience with reflex games are nearly the same as with the experience with platform games. To our question about the playing regularity of reflex games we got 8 “No, I only tried it a few times” (= 57,1%), 2 participants play it in every 3 months (= 14,3%) and 4 participants play reflex games every week (= 28,6%). After the done experiment we asked all of them about their feelings in the different experimental parts. For the questionnaire we used “Google forms”.
B. Platform game and experimental groups
For our experiment a simple platform game was programmed. Before we decided to do that, we also tried out the so called “flappy bird” game on our own, just to see what data we can collect and to find ideas how an own game could look like. There we have seen, that it is more useful to make a platform game, because for the “flappy bird” game you only have to use “UP” and “DOWN” keys and an inverting of the screen has not that much influence on our playing and learning habits as we want to see with our experiment. With our own programmed platform game, the player has to use the arrow keys “LEFT”, “RIGHT” and “UP” and by changing the screen 180° degree (upside down) you can also play the game against our understanding of gravity.
In the following figure 1 you can see our playing environment of the platform game. The task is to move the orange block through the different obstacles from left to right. Therefor the player can use the arrow keys “LEFT”, “RIGHT” and “UP”. The game itself is divided into chunks, which are without interruptions at the beginning of the game. But at a certain chunk there are interruptions and the orange block can fall down and the game is over. Then the game starts again from the beginning. Also, when the last chunk has passed, the player has to start from the beginning again. You can see 4 different chunks on the figure 1. Top left shows the beginning of the game and you can see that there are no interruptions of the white ground. The other chunks show more difficult arrangement of the white obstacles. With the help of some programmed features we collect the number of the passed chunks and also the time for each try. These data we can finally use for our analysis.
As we want to see how our brain adapts to a change in the game, we decided to make 4 experimental groups. You can see our 4 experimental groups in Table 1 . For example, participants of group 4 starts with “NORMAL” — keys, that means that pressing the “LEFT” arrow key means a left movement of the orange block and so on. In this group the participants have to play the game with an 180° inverted screen in the 1 st phase of the experiment. In the 2nd phase the screen is “NORMAL”. Each phase lasts 7 minutes and the player has to play the game as often as she/he is able to do in that time. Either the player finishes the game or fall down (game over) he has to start the game from the beginning. With the help of these 4 groups and a certain number of participants we try to find patterns how our participants behave during the different experimental phases.
For doing our experiments we used our own notebooks and the screen is just inverted with the in-build commands of the operating system of our notebook. The time was recorded with a smartphone. If you want to try the platform game on your own you can find it here.
C. Analysis
After collecting the data and asking the participants the questions of the questionnaire we analyze the data. The data itself is saved into CSV files, so we can easily load the data in Python with the help of Pandas library. Our analysis consist of an explanation of our game’s relevance and a study of the performance of the participant over each game modes: normal mode, reversed screen mode and inverted key mode.
Results
A. Why this game is relevant?
The goal of this part is to show that the game we chose, can actually be used to do our experiments. So what we need is a game where the outcome is not based on some kind of randomness. The more you play the more you succeed should be true for anyone who plays this game. Add to that, the game should have three modes: one has to be easier for the brain to understand and the two others have to be unusual.
As expected the more experience you have on platform games the more you are likely to get a high score. So in this type of game there is no such thing as luck and by learning you can improve your performance. We need this kind of results to show that the used game is relevant for our purpose.
We have defined three game modes for our platform game: normal mode, reversed screen mode and inverted key mode.
Here we can see the performance for each game mode. The players seem to be better on the normal game and this is logic since the normal game has been created based on what our brain is used to do: press the left key to go left, press the upper key to jump and so on. The other game modes have been designed to be harder since they are unusual. So we have our three relevant game modes to do our experiment.
B. Individual performances
Here we display the performances of all the players over both experiments they did. For each plot we add a straight line corresponding to the linear regression and the mean score of these points. Add to that we calculate the average slope coefficient for the first part and the second part.
Learning rate (first part): 0.82
Learning rate (second part): 0.12
What we observe is that there is some kind of a progression for most of the trials. If we divide the first and the second part of the experiment we can see that on average the learning rate during the first part is higher than in the second. Maybe learning from scratch is easier than changing our initial learning. Note that we are talking about just one game.
C. Progression and experience
Earlier we have said that people who have played more platform game are on average better than the others. But how do they manage with the unusual game modes (for instance inverted key mode) compared to the ones without experience.
What we see in these charts is that the experience on similar games doesn’t decrease our capacity to learn how to play with the inverted keys compared to people that have just a few experiences in those games. With a lot more of data, we may see a true difference between those kind of people that enable some conclusion. For our case we can only say that it doesn’t seem to be harder to change our . So the thing that may disturb people with the inverted key mode is that it doesn’t make sense and it’s not correlated to if we have played games which are “normal”.
D. Performance on experiment
For now we have seen individual performances but what about the performance on each experiment? The following plots will help you to see how the scores for each part of each experiment.
As we said above, the players are mostly learning for each experiment, except for one: experiment 4, first part, but it’s due to some noise and since just two people did it, three unexpected scores are able to change drastically the slope coefficient.
E. Participant feedback
As expected the unusual modes were harder for the participant to complete. Some participants felt some frustration with the inverted key mode more than with the reversed screen. What we can also note that the most of the participants found the second part not so harder even if the keys or the screen were inverted. So it seems that they have learned something in the first part that they use in the second one. Probably, they got used to the physic of the game.
Conclusion
In our experiment we were searching for patterns in human adaption to manipulated visual and motoric challenges. Doing our research our goal was to understand better and add something new to the field of neuroscience in the broadly examined topic of adaptation. Our research was inspired by the research history of this field and we chose to design a game by which we can measure learning and adaptation. We were particularly interested in the following questions:
i) does it make a difference to play a game with inverted screen (180°)?
ii) does it make a difference to play with inverted keys (left and right inverted)?
iii) if it does, how much does it takes to adapt?
iv) what is the difference between starting normally and continue with manipulated version or vice versa?
v) how all of these results are affected by gaming experience or other demographic parameters?
In our research we have found that even a few minutes of playing are enough to learn and improve performance in a basic game. Gaming experience proved to affect learning performance, and it had a positive effect. Similarly, as someone gained some experience during the first phase of the experiment, it could be used during the second phase as well. This result shows us that it is more important to be familiar with the general principles of the game to perform well than to get used to it with the modified conditions. Once one can play with the game at a good level it doesn’t seem to be a big difference to adapt to unusual conditions. Our results show that playing a platform game under different conditions doesn’t require different skills, it only requires to make slight changes in the general way of thinking.
Our results correspond to most of the findings in the international literature, however we didn’t explore much of the “after effects” (effect that follows its cause after a period of time, like muscle memory) due to the fact that 7 minutes of gaming shifts are probably not enough to induce concrete adaptation. Moreover the challenge was perhaps too easy and it was maybe not challenging enough. Making further experiments and collecting a bigger amount of data could help getting closer to a broader understanding of human adaption.
References
Burgos, P. I., Mariman, J. J., Makeig, S., Rivera-Lillo, G., & Maldonado, P. E. (2018). Visuomotor coordination and cortical connectivity of modular motor learning. Human Brain Mapping, 39(10), 3836–3853. https://doi.org/10.1002/hbm.24215
Della-Maggiore, V., Landi, S. M., & Villalta, J. I. (2015). Sensorimotor adaptation: Multiple forms of plasticity in motor circuits. Neuroscientist, 21(2), 109–125. https://doi.org/10.1177/1073858414545228
Droual, R. (n.d.). Nervous System. Retrieved December 15, 2019, from http://droualb.faculty.mjc.edu/Course Materials/Elementary Anatomy and Physiology 50/Lecture outlines/nervous_system.htm
Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of the visual field: A new twist on an old problem. Experimental Brain Research, 228(3), 327–339. https://doi.org/10.1007/s00221-013-3565-6
Marotta, J. J., Keith, G. P., & Crawford, J. D. (2005). Task-specific sensorimotor adaptation to reversing prisms. Journal of Neurophysiology, 93(2), 1104–1110. https://doi.org/10.1152/jn.00859.2004
Rand, M. K., & Rentsch, S. (2016). Eye-hand coordination during visuomotor adaptation with different rotation angles: Effects of terminal visual feedback. PLoS ONE, 11(11), 4–7. https://doi.org/10.1371/journal.pone.0164602
Sachse, P., Beermann, U., Martini, M., Maran, T., Domeier, M., & Furtner, M. R. (2017). “The world is upside down” — The Innsbruck Goggle Experiments of Theodor Erismann (1883–1961) and Ivo Kohler (1915–1985). Cortex, 92, 222–232. https://doi.org/10.1016/J.CORTEX.2017.04.014
