Controlling A Video Game With My Mind
Simply controlling a game by blinking.
After being incredibly inspired by the projects and research completed by neuroscientists Andrea Stocco and Rajesh Rao (specifically their recent project on technical telepathy), I wanted to experiment with brain-computer interfaces myself. So, I decided to create the first part of their project using the Muse 2 headband and python.
In case you don’t know what their experiment was, here’s a quick summary. Neuroscientists Andrea Stocco and Rajesh Rao tested the world’s first human brain-brain interface. Both participants sat in different rooms across campus so there was no possibility of hearing or seeing each other during the test. Dr. Rao was hooked up to an EEG device as he was the sender. He was watching a game on a TV screen that has missiles flying by however, there were no controls so he couldn’t shoot the missiles. Andrea was hooked up to a machine with a magnetic coil positioned directly on top of the part of his brain that signals him to move his hand. He was the receiver. His hand was placed on top of the space button on a keyboard. Unlike Dr. Rao, he was not watching the game however, he had the ability to control the game. So, in order for the missile to be shot, Dr. Rao had to focus on a thought or an idea, which sent a signal through the EEG device, travelled through the internet, reached the same machine Andrea was hooked up to, beamed a magnetic signal through his brain and pressed his finger on the space bar.
As a part of my research on technical telepathy (check out my article on technical telepathy here), I wanted to recreate their project (or at least attempt to). Additionally, for a higher chance of success, I wanted to keep the project as simple as possible. Hence, here is how I made a square appear on the screen of my laptop solely by blinking.
Step 1: Setting Up The Muse 2
The first thing I needed to do in order to create a mind-controlled video game was to get a device that gave off a live reading of my brain waves (hence the Muse 2 headband). This EEG device consists of 5 sensors that are located on the scalp, which are used to track brain signals.
Step 2: Outputting an LSL
In order to connect the Muse 2 to another device, I had to download an app that was able to synchronize live streaming data. In this case, I used one of the most popular streaming apps for the Muse 2 which was BlueMuse.
First, I downloaded the latest version from their GitHub page and unzipped the file.
Next, I went into the folder, located the InstallBlueMuse file and ran it using PowerShell.
After following the installation instructions from the GitHub page, I opened the application and connected my Muse 2.
I started a stream in order to get live EEG data from all 5 sensors.
Step 3 (Optional): Visual Reading of EEG Data
At this point in the project, I had created an LSL, however, I didn’t know what the numbers meant. So, I decided that I needed a visual reading in order to understand what my Muse 2 was outputting. Therefore, I downloaded a program called Muse LSL which outputted a visual chart and live reading of my brain waves.
After following the installation instructions and running $ muselsl view in my command prompt, an application popped up with the reading of all 5 sensors.
After analyzing this data, I concluded that my data pulled from the TP9 and TP10 sensors (right and left ears) were the most stable and would be the easiest to work with.
Step 4: Receiving Numerical EEG Data In Python
I needed to turn data into numerical data in order to predict patterns and detect changes in the brain. From the MuseLSL GitHub page, I downloaded the neurofeedback.py file from the examples folder, which took raw EEG data from the Muse 2 and classified it into delta, theta, alpha and beta frequencies. In order for the program to run, I also needed to download the utils.py file from the same folder for lower-level epoch and buffering functions.
After running the program, I wanted to make the code delta-frequency specific. So, I copied the file and made a couple of changes in order to print only the delta waves coming from the left ear sensor on the headband (T9 sensor). After computing the band powers in the original program, instead of computing the data or all 4 frequencies (alpha, beta, delta and theta), I changed it by editing this line so that it outputted only delta waves.
Step 5: Building the Game
Finally, after acquiring a numerical value for my delta waves, I needed to build the game I wanted to control with my mind. I built a simple game using pygame which added a random coloured and sized square to the screen and moved at a random speed when the space bar was pressed.
Step 6: Replacing Space Key With Blinking
The final step to having the superpower of mind control was replacing the space key button with the change in delta waves. In order to do this, I located the code that added a square if the event of a key press occurred and changed it so that the same square was added if my delta waves changed from 0 to 1 or greater (which occurred if I blinked my eyes).
Clearly, the final product isn’t as eye-pleasing as the original since its processing time is slower, however, it gives you an idea of how Andrea Stocco and Rajesh Rao were able to create the very basics of the first part of their experiment (simply creating a game controlled by brain waves).
Overall, this project was extremely fun to create. It gives me a small taste of what BCIs could be used for in the future, and what I want to create for my next projects.
