Researchers in Shanghai have reported in a study, recently published in Science Advances, that they’ve successfully decoded Mandarin Chinese language in real time with the help of a brain-computer interface (BCI) framework, a first for BCIs working with tonal languages. The participant involved in the study was also capable of controlling a robotic arm and digital avatar and interacting with a large language model using this new system.
What are mind-reading BCIs used for?
While most people may not want a computer reading their mind, those who are unable to speak due to neurological conditions, like strokes or amyotrophic lateral sclerosis (ALS), need to find alternative ways to communicate. Speech decoding BCIs, capable of decoding neural signals, offer a promising way to restore communication in such individuals. In addition to communication, BCIs also offer ways to control devices directly through thought. This is particularly helpful for neurological conditions in which disabilities extend beyond speech loss.
These types of devices are not exactly a novel technology, but most BCI speech decoding research has focused on English, a non-tonal language.
“A leading approach focuses on the ventral sensorimotor cortex, which encodes articulatory kinematic trajectories. Neural signals from this region can be transformed into discrete linguistic units or articulatory gesture parameters and subsequently synthesized into words, sentences, or sounds. This strategy is especially suitable for individuals with intact speech motor areas, aiming to re-enable functional communication.
“Recent advances in English language decoding have enabled real-time translation of brain activity into text or speech for patients with severe dysarthria caused by conditions such as amyotrophic lateral sclerosis (ALS) or brainstem stroke,” the study authors write.
Overcoming the difficulties of decoding Mandarin
Advances in BCIs capable of decoding tonal languages, like Mandarin, have been more limited. Because Mandarin is a tonal, monosyllabic language with high homophone density, speech decoding is more challenging. Some previous studies have decoded small sets of Mandarin syllables or tones, but not the full range needed for practical use and not in real time.
However, a clinical study on an epilepsy patient has enabled the researchers involved in the new study to take a different approach. The study, conducted on a 43-year-old woman, used an implanted 256-channel high-density electrocorticography (ECoG) array to monitor and record neural signals through a series of single-character and sentence reading tasks over 11 days. A 3-gram Mandarin language model was integrated to improve sentence decoding.
The team says analysis of the ECoG signals revealed distinct neural correlates for syllable and tone processing. The system achieved a median syllable identification accuracy of 71.2% in single-character tasks. Real-time sentence decoding reached 73.1% character accuracy with a language model, and a communication rate of 49.7 characters per minute.
“Our study demonstrates that combining high-density, ultraconformal ECoG grids with a syllablecentric decoding framework can yield substantial improvements. The ECoG arrays provided broad and stable cortical coverage, particularly over speech-related regions, and enabled us to decode a large set of 394 Mandarin tonal syllables with high accuracy—based primarily on neural features before any linguistic postprocessing,” the study authors write.
Refining future BCIs for speech loss
While the study demonstrates marked improvement for BCIs decoding Mandarin, the authors do note some limitations and areas that could use improvement. The study only included one participant, limiting generalizability. And because the ECoG array was meant for clinical epilepsy monitoring, the electrode coverage did not include all tone-relevant brain regions. However, future studies can build upon this one, further increasing accuracy and generalizability.
The study authors hope to extend BCI applicability to a range of patients. They say, “Beyond improvements in decoding accuracy and hardware performance, expanding the neural targets of speech BCIs represents an exciting frontier.
“Although current approaches primarily leverage signals from motor and premotor cortices responsible for articulation, future systems may benefit from incorporating activity in higher-order language, such as the middle temporal gyrus, inferior frontal gyrus, and supramarginal gyrus. Integrating the semantic and syntactic information processed within these regions may help build more stable and accurate speech decoders.”