Man Who Is Paralyzed Communicates By Imagining Handwriting

May 12, 2021
Originally published on May 13, 2021 5:31 am

An experimental device that turns thoughts into text has allowed a man who was left paralyzed by an accident to construct sentences swiftly on a computer screen.

The man was able to type with 95% accuracy just by imagining he was handwriting letters on a sheet of paper, a team reported Wednesday in the journal Nature.

"What we found, surprisingly, is that [he] can type at about 90 characters per minute," says Krishna Shenoy of Stanford University and the Howard Hughes Medical Institute.

The device would be most useful to someone who could neither move nor speak, says Dr. Jaimie Henderson, a neurosurgeon at Stanford and co-director, with Shenoy, of the Stanford Neural Prosthetics Translational Laboratory.

"We can also envision it being used by someone who might have had a spinal cord injury who wants to use email," Henderson says, "or, say, a computer programmer who wants to go back to work."

Both Henderson and Shenoy have a proprietary interest in commercializing the experimental approach used to decode brain signals.

The idea of decoding the brain activity involved in handwriting is "just brilliant," says John Ngai, who directs the National Institutes of Health's BRAIN Initiative, which helped fund the research.

"But it was only on one subject in a laboratory setting," Ngai says. "So at the moment it's a great demonstration of proof of principle."

The man who agreed to test the device is unable to move his arms and legs as the result of a freak accident.

"He was taking out the garbage, slipped, fell and instantly became quadriplegic," Henderson says. "So he's essentially completely paralyzed."

A few years ago, the man agreed to take part in a study of an experimental system called BrainGate2. It allows people who are paralyzed to control computers and other devices using only their thoughts.

The system relies on electrodes surgically implanted near the part of the brain that controls movement. In previous studies, participants had learned to control a computer cursor or robotic arm by imagining they were moving their hands.

This time, Henderson, Shenoy and a team of scientists had the man imagine he was writing individual letters by hand while a computer monitored the electrical activity in his brain.

Eventually, the computer learned to decode the distinct pattern of activity associated with every letter of the alphabet as well as several symbols.

Once that process is complete, Shenoy says, "We can determine if the letter you wrote is an A or a B or a C and then plop that up on the screen and you're able to spell out words and sentences and so forth one letter at a time."

In previous experiments, participants had been able to use their thoughts to "point and click" at letters on a screen. But that approach was much slower than imagined handwriting.

Also, because the new system relies on familiar thoughts, the participant was able to use it almost immediately.

"He was very happy when he was able to write out messages in response to the questions we asked him." Henderson says. "He was pretty excited about this."

The team's success decoding imagined handwriting is just the latest advance in efforts to link computers to the human brain, Ngai says.

"I was introduced to this concept over 10 years ago, and I thought it was quite a bit of science fiction," he says. "Then roughly about five years later it was shown to be not to be such science fiction after all. So I think we're seeing a progression. It's really quite exciting."

An editorial accompanying the study shares that view.

The handwriting approach "has brought neural interfaces that allow rapid communication much closer to a practical reality," wrote Pavithra Rajeswaran and Amy L. Orsborn of the University of Washington.

Copyright 2021 NPR. To see more, visit https://www.npr.org.

MARY LOUISE KELLY, HOST:

An experimental device that turns thoughts into text has allowed a man who is paralyzed to type messages on a computer. NPR's Jon Hamilton reports the device is able to decode signals in the brain that are usually used for handwriting.

JON HAMILTON, BYLINE: The man was left unable to move his arms and legs after an accident, says Dr. Jaimie Henderson, a neurosurgeon at Stanford University.

JAIMIE HENDERSON: He was taking out the garbage, slipped, fell and instantly became quadriplegic. So he's essentially completely paralyzed.

HAMILTON: Henderson says a few years ago, the man agreed to take part in a study of an experimental system that allows people to control computers and other devices using only their thoughts. Henderson says the system relies on electrodes implanted inside the skull.

HENDERSON: The electrodes are placed just over the motor cortex of the brain, which is the outer part of the brain that controls movement.

HAMILTON: Henderson and a team of scientists had the participant imagine that he was holding a pen and handwriting individual letters on a sheet of paper. A computer monitored the man's brain activity. And Krishna Shenoy of Stanford and the Howard Hughes Medical Institute says eventually the machine learned to decode signals associated with every letter of the alphabet.

KRISHNA SHENOY: And then we can determine if the letter he wrote is an A or B or C, and then plop that up on the screen. And you're able to spell out words and sentences and so forth, one letter at a time.

HAMILTON: The system produced the correct letter 95% of the time, so Shenoy and the team decided to see how fast it was.

SHENOY: And what we found, surprisingly, is that we can decode this information quite rapidly so that you're able to type at about 90 characters per minute.

HAMILTON: That's a bit slower than most people can text on a smartphone, but it's more than twice the speed of an earlier approach that had participants use their thoughts to point and click at letters on a screen. Jaimie Henderson says that because handwriting is a skill most brains already know, it was easy for the man to learn to control the computer.

HENDERSON: He was very happy when he was able to write out messages in response to some of the questions that we asked him. He was pretty excited about this.

HAMILTON: Henderson says the ability to communicate this way would be most valuable to someone unable to move or speak.

HENDERSON: We can also envision someone who may have had a spinal cord injury, for example, like our current participant, who wants to use email or say, a computer programmer who wants to go back to work or somebody who wants to produce text of any kind.

HAMILTON: Both Henderson and Shenoy have a commercial interest in the technology.

Decoding imagined handwriting is just the latest success in a long-standing effort to link computers to the human brain. John Ngai directs the NIH BRAIN Initiative, which has played a key role in funding the research. He calls the idea for the study brilliant, but he says the approach is probably still a long way from widespread use.

JOHN NGAI: At the moment, it's a great demonstration of proof of principle, but it was only on one subject in a laboratory setting.

HAMILTON: Ngai says even so, he's optimistic that so-called brain-computer interfaces will eventually help people who've had strokes or spinal cord injuries.

NGAI: I was introduced to this concept over 10 years ago, and I thought it was quite a bit of science fiction. Well, roughly about five years later, it was shown not to be such science fiction after all. So I think we're seeing a progression along this curve. It's really quite exciting.

HAMILTON: The research appears in the journal Nature.

Jon Hamilton, NPR News. Transcript provided by NPR, Copyright NPR.