P300 Speller: How to spell with your brain waves
At this moment, your brain is performing an amazing feat — reading. Four to five times per second, your eyes scan the page in short spasmodic movements that you barely notice¹. Only the meanings of the words reach your conscious awareness. This extraordinarily ability evolved in humans because of the strong advantage the ability to share and record our thoughts provides our species.
One hundred and fifty thousand years ago, homo sapiens developed the ability to transfer knowledge from one person to another — and across generations — in the form of linguistic speech². Speech requires controlled and complex muscle movements with a combination of breathing patterns, vocal cord and tongue movements. Over time our predecessors learned to encode information in the form of drawings or hieroglyphics, which also required fine motor control to scribe detailed symbols onto a surface.
Ultimately, we developed typewriters, computers and cell phones to minimize the level of fine motor skills required to communicate in written form. Despite these developments, however, there is still no widespread way for our species to communicate without some level of neuromuscular involvement. In the last few years there has been extensive research investigating novel modes of communication through direct access to brain signal. With advancements in brain imaging methods, specifically with EEG, there is much promise of an effective tool being developed in the near future. One of the best tools to date is called the P300 Speller.
What is a P300 Speller?
Brain computer interfaces (BCIs) can provide new communication channels for humans, and the P300 speller is one specific BCI with promising results. The P300 speller uses an electroencephalogram (EEG) to measure an individual’s electrical brain activity as they try to spell out words, and returns auditory or visual feedback after each attempt. The P300 speller uses this information to sequentially estimate the letters an individual is thinking of in order to spell out words. Different brain spellers report a spelling speed of 5–60 characters per minute, with accuracies between 70–95%³.
Why is the P300 speller so useful?
Motor neuron diseases, such as Amyotrophic Lateral Sclerosis (ALS/Lou Gehrig’s disease), are a class of disorders that selectively affect motor control. Other conditions that result in a similar loss of motor function include muscular dystrophies, brainstem stroke, cerebral palsy and multiple sclerosis, among many more. Such disorders often cause patients to lose control over tongue or skeletal muscle movements, resulting in significant impairments in their ability to communicate. A brain speller could allow individuals suffering with varying degrees of motor dysfunction with, at the very least, the ability to communicate. Stephen Hawking was a prime example of the contributions one can make with the help of technologies that allow them to communicate when faced with partial loss of motor control. Not only was this technology important for him and his loved ones, it was invaluable to the rest of the world that gained from his rich contributions to physics.
When individuals experience full loss of motor control, however, technologies such as the speech synthesizer Stephen Hawking used are insufficient. This is where brain spellers can fill the gap, for example, by allowing communication for individuals in a comatose or near death state. This is incredibly meaningful for individuals that may want to share their heartfelt last words or make crucial medical decisions on their own. From a scientific and medical stand point, brain spellers could provide scientists with insight into states that they would otherwise have no access to.
How does the P300 Speller Work?
When multiple neurons generate the same repeated sequence of activity, a synchronized electric field potential is induced that propagates through the brain and skull. Electroencephalography (EEG) is a non-invasive tool that measures the difference in voltage between 2 locations through time. It is relatively inexpensive, portable and also has a high temporal resolution. Language (and spelling) consists of fast, temporally sequenced, cognitive events. EEGs are well suited to capture these dynamics in the time frame at which they occur. For example, the temporal precision of an alternate brain imaging tool, magnetic resonance imaging (MRI), is 2–3 orders of magnitude slower than that of EEGs (see figure 5)⁴. MRIs also only indirectly measure neural activity, whereas oscillations observed in EEG are direct reflections (plus noise) of neural oscillations in the cortex. Moreover, EEG signals provide multidimensional information (by capturing power, time, frequency, space and phase) that can be fed into models to produce more accurate estimates. For these reasons, along with many more, most research done on brain spellers use EEGs.
Event related potentials (ERPs) represent electrical signals detected by EEGs in response to a specific sensory, motor, or psychological event. ERPs are calculated by averaging electrical responses to an event over several different trials. They typically consist of unique and characteristic components (or peaks) that are thought to reflect information processing associated with the event. The components are described by specific time blocks following the event in which a peak amplitude may occur, and whether or not it is a positive or negative deflection. For example, a P300 component represents a positive (“P”) peak in electrical amplitude, roughly 300 ms after the event (see figure 6). This component has been studied extensively and is quite well understood in neuroscience to be associated with a rare or salient event⁵.
A P300 speller is one of the most researched types of brain spellers and makes use of this characteristic P300 ERP component to estimate what letter a subject is thinking about sequentially in order to spell out a word. The first P300 speller was introduced by Farewell and Donchin in 1988. It consisted of a 6 by 6 matrix called the “Matrix Speller” that contained all 26 english letters and the numbers 1–9⁶. The subject in the experiment would be asked to direct their attention to a target letter, and then a row or column would be randomly and sequentially flashed from the matrix that may or may not contain the target letter (see figure 7 a). The subjects were also asked to count the number of times their target letter flashed in order to keep their focus. A strong P300 brain signal was typically associated with each time the target letter flashed. Because the experimenters knew which row or column had been flashed during each of those events, could deduce what the target letter was by combining the rows and columns that elicited a P300 response. Another variation of the “Matrix Speller” was used with random combinations of flashing letters instead of an entire row or column (figure 7 b). The accuracy of these methods reached up to 95%, however it took roughly 26 seconds to correctly identify the letters, which is too slow to be considered helpful for healthy individuals³.
Subsequent attempts at designing a P300 speller aimed to change the method used to intensify the target character. Instead of flashing the characters, they attempted graphical effects like pans, or zooms, pattern rotations and sharpening effects as well as colour changes. Combining many of these studies showed that some models performed better with one stimulation technique over another for specific individuals. This led to the idea that the P300 speller can be personalized to achieve the highest accuracy for each subject. In addition, analyzing these studies showed that between 7–9 flashes or intensifications were required for each character before the program could say with over 90% certainty what particular letter the individual meant to spell³.
Another variation of the study that was suggested made use of the fact that the P300 response is amplified when individuals see a familiar face. There are unique areas in the brain entirely devoted to processing faces. For this reason, the response to faces is typically more acute than the response to other stimuli. One version of the P300 speller uses semi-transparent images of familiar faces overlaid on the characters instead of intensifying them. This showed a higher P300 response but resulted in similar overall accuracies³.
Many other versions of the original P300 spellers have been researched, using different character layouts, different languages and even auditory stimuli instead of visual letter matrices. Although research is still preliminary it holds a lot of promise as an accessibility tool in the near future.
What do we need to be concerned about?
As neuroimaging becomes more accurate and more accessible, it’s important to have conversations concerning the implications of this technology on our privacy, especially if they build presence in consumer goods. It took more than a decade of smartphones dominating consumer technology before it became apparent to the public that user data could be harnessed to target users with unwanted ads, political campaigns, and biased news. Our online behaviour started shaping our digital landscape and in turn our digital landscape started to forge our behaviour. Suppose a technology like the P300 speller was developed to a point where our internal signals could be seamlessly tied to a set of preferred actions from ordering a meal to writing an email. Building a day to day dependence on such a device, the same way we have with smartphones, could cause users unwittingly relinquish the privacy of their thoughts.
That said, given the way the P300 speller is currently being used, it seems this possibility is highly unlikely. While using a P300 speller an individual must visualize or think about each letter sequentially in a specific way to spell out a given word. It seems improbable that the P300 speller will advance in such a way that it can navigate the complexity of human thought any faster than another BCI technology.
Another potential concern that seems more pressing, however, is when decisions are being made based on the interpretation of results from the P300 speller. If a P300 speller is ultimately used to make important medical decisions for a patient that is unable to otherwise communicate, it is important that the person administering the test is aware of how accurate the test is likely to be the given situation and make that clear when interpreting results.
Where is this research headed and what can we do to improve?
Most designs of the P300 speller rely on visual recognition of letters. It is possible that some of the cases in which this tool may be required, may involve subjects that are also visually impaired. With further experimentation we could extend to an auditory or even a tactile (braille) speller. Although there have been several publications on the subject in the last few years, research is sparse and with a few subjects each. There are many companies that sell EEG equipment for affordable prices and software required can be accessed open source. This means that experiments can be conducted in non-laboratory environments, or in cost effective ways at companies that are working on advancements in BCI. Having more people experimenting with this research can quickly allow for higher speed and accuracy of predictions. Working with data scientists on available datasets can also help in those domains as well. Humans are inherently a social species and many argue that the advanced development of the human brain was intrinsically linked to our ability to communicate. And for healthy individuals who have no need as of yet for the brain speller, maybe this is as exciting as the magical Ouji board — but a scientifically found one without any spirits. Perhaps that in and of itself is worth the effort to explore this exciting tool.
Works Cited:
1. Dehaene, S. Reading in the Brain: The New Science of How We Read. (Penguin Publishing Group, 2009).
2. Pagel, M. Q&A: What is human language, when did it evolve and why should we care? BMC Biol. 15, 64 (2017).
3. Rezeika, A. et al. Brain–computer interface spellers: A review. Brain Sci. 8, (2018).
4. Cohen, M. X. Analyzing Neural Time Series Data: Theory and Practice. (The MIT Press, 2014).
5. Guan, C., Thulasidas, M. & Wu, J. High performance P300 speller for brain-computer interface. 2004 IEEE Int. Work. Biomed. Circuits Syst. 1–3 (2004). doi:10.1109/biocas.2004.1454155
6. Farwell, L. A. & Donchin, E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523 (1988).
