Rats in Doom VR

Viktor Tóth
Oct 11 · 14 min read

TL;DR I built a VR setup for rodents from scratch and trained three rats in an automated fashion, without manual intervention, to traverse a corridor rendered in the DOOM II engine. Although I did implement the mechanisms to further train rats to shoot monsters in-game, I lacked the time to actually reinforce the behavior. The promise of the project is a relatively cheap (<$2000) VR setup that automatically trains rodents to traverse 3D environments without restraining them too much, while refraining from surgical procedures to provide the least stressful circumstances for them. Rodent VR rigs have had their presence in neuroscience experiments in the past; I hope to see more such studies to come! The following project aims to ease those experiments by automating some tasks of the experimenter.

This article describes the implementation of the automated training procedure proposed in my previous post: A Neuroengineer’s Guide on Training Rats to Play Doom

VR setup

Rodent VR setups usually consists of a (polystyrene) ball, on which the rat runs suspended in a harness, along with motion sensors tracking the movement of the ball. Either a screen or a projector displays the virtual environment. The ball could be levitated on airflow or just be rolled on ball bearings. I tried both options and the latter prevailed with its robustness despite the higher friction it imposed. I housed the whole system on an aluminum frame precut and ordered from Misumi.

Hardware

Custom parts (attachments, items holding electrical components, mechanical contraptions) were 3D designed in Blender then printed. Dozens of versions were tested to arrive at the final rig.

A rubber-coated 16" (40 cm) foam ball was held in place by three ball bearings on the bottom, two at the front and one on the left side. The guiding principles for designing the base were as follows: 1) the ball bearings should contact the spherical treadmill at a 90° angle to minimize friction, 2) the arms should be long enough to support the treadmill from the sides preventing it from rolling off the base, 3) the base should be able to hold up foam balls of different sizes without manual redesign — at this point the diameter of the ball had not been fixed.

To meet all these requirements I wrote a Blender Python script that spawned the arms in the 3D editor just in the right position above the base, hugging the ball, given the dimensions of the foam ball, of the 3D printer and of the ball bearings. After changing the radius of the spherical treadmill in the script, a new arm arrangement can be spawned at the right size, angle and position. The arms were printed separately from the base to maximize their length.

The motion laser sensors were fashioned from gutted computer mice, and were installed at the front and left side to cover all three axes of ball movement. The motion detection range of the laser mouse maxed out around 1 cm. To account for irregularities on the surface of the foam ball, frontal bearings were positioned on magic arms distancing the ball from the front sensor. Magic arms further held the motion sensors and the reward stations, which made all this equipment easy to (re)position.

Rats were trained to move on the spherical treadmill by means of operant conditioning, which required a feedback mechanism that reinforces the desired behavior. To deliver positive feedback, I chose feeding them sugary water, as it’s relatively simple to dose in small amounts (tens of μL): took a precision solenoid valve, gravity-fed it (check the above intro video from 0:42) and drove it with a microcontroller. The above shown 3D printed tiny bowl functioned as a buffer for any excess water, while Quidditch goal posts held the tubing tight.

The two larger tubes conducted air puffs from the 12V pumps. Air puffs in the face or the butt are often used as negative feedback in rodent VR experiments [4]; specifically to signal when the animal hits the wall, from either side, while running. The foam padding were added to absorb the vibration of the pumps, which reduced noise. Reducing noise across the setup is in general of importance, as rodents can easily get startled by sounds of sudden onset. Although the air pumps worked fine, in the lack of time and out of caution, I refrained from applying air puffs in my experiments not to discourage rats from staying on the treadmill.

Novel Hardware

Train to Walk

The video above first shows how the motor attempts (but fails) to rotate the ball: a linear actuator was extended until the wheel reached the ball, the motor rotated the wheel for two seconds before the actuator was detracted. Failing to roll the ball in this case was both a bug and a feature. If the rat put all its weight lying on the treadmill, the traction of the silicone wheel on the rubber-coated polystyrene ball turned out to be insufficient. The rat was not entirely forced to walk, only encouraged, which, I can anecdotally say, eased the acclimation process.

The second portion of the video demonstrates the 2D movement training. As you may notice, the rat was slightly confused about the second motor, which kept turning him to the right to face the exit sign — this was the first time Romero (that’s his name) ever encountered the enforced left-right rotation of the ball.

The last part of the video (from 0:32, turn the audio on) records a later part of the training, where Romero already learnt to take advantage of the motor and roll with it. He also just began to walk on his own there.

Placing a spring between the actuator and the wheel helped to maintain the traction of the wheel on the imperfect spherical surface — the distance from the center and the surface of the ball varied. A spring allowed for such variance without pushing the ball too much to the side.

Train to Shoot

Let’s say the behavioral action associated with shooting is the rat raising its body by extending the front legs (a form of rearing). It should be trained to perform this action every time a monster appears ahead. Every such time, the experimenter could manually lift the rat by the harness, initiate an in-game shot and release some sugary water for positive feedback; the timing of all these operations are essential to shorten the training period as much as possible, and to avoid confusing the animal by giving inconsistent feedback. So, how could this be automated?

A cute little harness (selected from a set of 5) was attached to a 3D printed arm allowing for vertical movement. A solenoid push-pull actuator, secured to the base of the arm, served as the training mechanism to teach the shooting behavior. Just above the actuator a button sat. When pushed, either by activating, pulling up the solenoid itself, or by the animal raising its posture and thus lifting the inactive solenoid shaft, an in-game shot was discharged. Note: the actuator was activated using pulse-width modulation to drive a more gradual pull not to scare the animal.

Simply put, the training procedure would go as follows: the rat walks into a monster → the software detects that the monster is in the proximity of the player (and for now, let’s assume that the player is facing it) → initially the rat has no idea what to do in this situation, so the training software activates the push-pull solenoid lifting the animal slightly upwards → the head of the actuator then touches the button → monster gets shot down → reward in the form of sugary water is released to reinforce the behavior.

Ideally, the rat learns on its own to rear up the body in front of every monster, which then lifts the shaft of the solenoid, pushes the button, the monster gets shot and the reward is dispensed.

Control

I wanted to push as much logic to the PC (Python) as possible without compromising on the response time of the setup. The PC sent small packages to the microcontrollers as commands: first byte being the message type (e.g. extend the linear actuator, dispense sugary water, etc.), then the packet load (how long to extend the actuator in seconds, amount of water to dispense in μL). The experiment described below was implemented in a single Python script (<300 lines), concurrently receiving movement and shot info, sending out commands to the devices and interfacing the game.

Interfacing Doom

As the player pushed the button, the animal was teleported to the beginning of the map to start all over again. A Doom (ACS) script registered the player’s position along the corridor and the number of monsters it killed, and exposed all this info to the Python experiment script.

I ran Doom, accessed game states and sent game actions using the ViZDoom engine. ViZDoom was primarily built for training reinforcement learning algorithms, but its seamless Doom API made it perfect for the job. From motion data that evdev supplied, I calculated the speed of movement over time, scaled and relayed it to ViZDoom at every iteration of the game to translate rat steps into in-game steps. ViZDoom in return exposed the game state, sprinkled with some extra variables that were defined in the ACS script.

Experiment

I spent around 6 weeks with them during their (and my) night cycle, ~1 hour with each daily. It took two weeks to habituate them to me — this process could be accelerated, I was just a little slow with them. One week to teach them to stay put while I was dressing them; I also tested multiple harnesses and their combinations during this time.

I spend another week of habituation to the spherical treadmill and feeding from the bowl. First, I kept filling the bowl with baby food to make them stay on top of the ball. Baby food has a more compelling scent to recognize than sugary water; also, I could avoid instantiating water restriction for this period.

For the remaining 11 days I ran the actual VR Doom training. Romero outperformed the others in general, though Tom was the first to actually walk on its own without the help of the motorized ball roller. I had two sessions to train shooting with Romero, who was pretty confident running in VR at that point. Two sessions were far from enough; getting stuck into the monster and pulled up by the solenoid confused him. In the last part of the video below, you can hear the push-pull solenoid activating, and see it pulling Romero slightly upwards before the shots are discharged.

Amid the VR training I kept improving the software simultaneously with the hardware, which inconsistency most probably interfered with the learning progress, but was necessary to finalize the setup.

Conclusions

Computer games, virtual worlds are abundant and easy to make. Once we can reliably train animals to play games, designing an experiment becomes a software problem instead of an often costly hardware problem. However, consistent training of animals to perform complex, context-dependent actions in VR happens to be pretty difficult. This project was an attempt to automate this process. However, even after 11 days of VR training, I had to be present and help the rats to get on or off the ball, as they desired. Although their training was software-based, it still required my presence.

Unfortunately, I did not have the time to carry out the experiment in its entirety — I gave myself a hard deadline out of personal reasons. Rats lacking the ability to shoot is an obvious major limitation of this study. In hindsight, I would much rather implement an already tested response method to train shooting, like a nose-poking system, than the button-pushing contraption. Although the mechanisms to train left-right turning was present, I did not have time to train movements with turns included, so in most sessions only one axis of motion was registered.

XXI had fun building a rodent VR rig and training rats to kinda play Doom. Apart from paying for the animals, their housing and access to lab space, I financed the project myself; I wanted to see it through and worked most nights building, coding and conducting experiments. I left out a great deal of details from this article: rubber-coating 4 huge foam balls, one with a 24" diameter, testing various coating materials; levitating the treadmill with air pumps implementing a design [5] that I then had to throw out entirely; several attempts at recalibrating the motion sensing code, so rat movement got translated into immediate in-game steps of the right speed and consistency; and many more. Honestly, I thought it would be way easier, especially the training part, but one can’t underestimate the often unyielding independence of these free spirited animals.

The above presented VR system is ridiculously cheap to build at a price below $2000, compared to the other commercially available option costing north of $70,000. There’s an undeniable difference in product quality, and the scope of testing, but not as high to justify this contrast in price. I hope to see more preclinical neuroscience research capitalize on VR experiments to capture behaviorally and cognitively rich neural recordings.

Hit me up at viktor.toth@protonmail.com if you have any questions! To see more on the why, check out my previous article: A Neuroengineer’s Guide on Training Rats to Play Doom

Acknowledgements

Thank you Fylaktis, Jose, Rich, Santosh and the two Jasons for giving me support on electrical and mechanical issues! Yuniel, thank you for delivering all those packages! Thanks Christine for patching up the rat harness and all the encouragement on the way! Umair, Ibrahim and Franka thank you for helping me to receive animals and for providing guidance in handling them! Naveen, thanks a lot for supporting the project throughout and giving the idea to attach springs on the linear actuators! Thanks Todd for sharing my love for Doom and encouraging me! Thank you Pedro for carrying and making space for this huge setup! Special thanks to Theo, Pato and their lab members for providing me with lab space and putting up with my crazy idea!

Romero, Carmack and Tom, both in rat and human forms, thank you for making this possible!

Combined video of all the above to share
Bonus behind the scenes Paint designs

References

[1] K. Thurley and A. Ayaz, “Virtual reality systems for rodents,” Curr Zool, vol. 63, no. 1, pp. 109–119, Feb. 2017, doi: 10.1093/cz/zow070.[2] B. K. Young, J. N. Brennan, P. Wang, and N. Tian, “Virtual reality method to analyze visual recognition in mice,” PLoS One, vol. 13, no. 5, p. e0196563, 2018, doi: 10.1371/journal.pone.0196563.[3] G. Chen, J. A. King, N. Burgess, and J. O’Keefe, “How vision and movement combine in the hippocampal place code,” PNAS, vol. 110, no. 1, pp. 378–383, Jan. 2013, doi: 10.1073/pnas.1215834110.[4] T. Muzzu, S. Mitolo, G. P. Gava, and S. R. Schultz, “Encoding of locomotion kinematics in the mouse cerebellum,” PLOS ONE, vol. 13, no. 9, p. e0203900, Sep. 2018, doi: 10.1371/journal.pone.0203900.[5] D. B. Aharoni, “Rats in Virtual Space: The development and implementation of a multimodal virtual reality system for small animals,” UCLA, 2013. Accessed: May 03, 2021. [Online]. Available: https://escholarship.org/uc/item/0wd4p4mr

The experiment was approved by the Feinstein Institutes’ Institutional Animal Care and Use Committee.

Mindsoft

neuroengineering fun