Real Guns as Controllers: How to Bring Anything into VR

I’m Christopher O’Hagan, a software engineer at @KainosSoftware and I am working in the Applied Innovation department. Focusing on Immersive technologies. This blog is about how we recently turned an almost completely functioning rifle into a Virtual Reality controller and how we overcame the issues that came with it.

We have recently been approached by a Canadian customer looking for help building a realistic shooter as an attraction for their arcade. This is a part of a growing trend we have observed where immersive technology such as VR is being used for in-location entertainment businesses such as escape rooms and roller coasters. While the equipment and setup can be expensive it is viable due to the high footfall and novel environments offered by these businesses. A great example of in-location entertainment is virtual paintball which companies, like ‘The Void’ have succeeded in doing.

Our primary challenge for the immersive shooter we were helping with was to build and utilise an actual firearm… or at least something that looked and felt like one.

Getting the Controller

Originally we believed sourcing a controller would be difficult. We reached out to a local air-soft gun supplier, Gear of War. Our requirements were:

• A replica air-soft rifle that looked, felt and was weighted like a real rifle
• A gun that, when fired would produce realistic recoil
• A trigger and safety that were wired so that we could use them for inputs of our simulation

We thought this would be asking a lot but were surprised by the fact that this seemed normal to them. They have done a lot of props for TV and Films so had received weirder requests. We then used a mounted Vive tracker to track the gun in VR.

Setting up the Controller

This is where we experienced our main issue, vibrations: The issue was caused by the realistic recoil of our gun. Since the Vive Tracker uses IMUs (Inertial movement units) for tracking at a high frequency, and the Vive lighthouse system for tracking at a comparatively lower frequency and for drift correction. IMUs work by measuring acceleration in each axis then calculating displacement from that measurement. When IMUs are aggressively vibrated, such as when being mounted to a realistically firing weapon, they are unable to track accurately. This was a problem.

As we are not experts in vibration, we made some attempts to solve the problem but didn’t make much progress. We then decided to make a reddit post to r/Vive. The community response was helpful, correcting some simple stuff we missed such as the tracker being too far up the rail. To our surprise HTC contacted us about the project showing interest in what we were attempting to do. They had a beta version of the Vive Tracker firmware which could help, a low pass filter for the IMU, which they gave us.

Graph of normal input versus input after the use of a low pass filter

The low pass filter could be configured through the USB HID interface. We were able to send ‘0xB3’ reports, which sends basic information about the device being used as well as the low pass configuration, from a MacBook to the Tracker with the right settings to sufficiently dampen the vibrations. The next step was to send the feature report, which allows us to communicate changes in settings as well as control HID devices, to the Tracker from a raspberry pi as we needed something that could fit inside the gun’s cartridge so that the user couldn’t see the wiring.

This simply creates the report we want to send, based of the HTC documentation and constantly sends these reports to the Tracker

Low Pass Filter

These are the steps to set up the Raspberry Pi for this sort of project

• Install raspbian on the raspberry pi
• Update pi’s version of ‘Node.js’
• Install the ‘node-hid’ library into your project, for specific instruction to do that on the pi look here
• Add a file in udev called `60-HTC-Vive-perms.rules` with this line in it

KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2022", TAG+="uaccess"

An issue which you can have is that the wrong wIndex value is being used by node-hid, in this case wIndex is used to specify the interface or where the report is sent. Here’s something to help get your head around USB requests. ‘Node-hid’ doesn’t allow you to specify the interface. Fortunately when I looked back at the ‘show-devices.js’, which comes with the ‘node-hid’ library and outputs all connected HID devices, it had different paths for each interface.

Using the show-devices.js, which comes with the node-hid library. You can find which path to use in the code in case it appears different

If you have any other issues with the script, ‘usbmon’ comes with Raspbian and allows you to view all the usb requests being made. Here’s a useful link to help you decode those requests.(Starts at page 15)

Conclusion

Using a gun as a VR controller was by no means easy. The vibrations of the gun firing are strong enough to disrupt the IMUs ability to accurately track itself. Using the Tracker’s low pass filter feature, and an on-board raspberry pi we were able to overcome this issue without compromising tracking or the force at which the gun vibrates.

Thanks for reading
@cohagan154