Bring on the wetware
Ever since I read William Gibson’s cyberpunk novel Neuromancer I have been fascinated by the idea of directly connecting brains and machines. Characters in Gibson’s novels were able to connect to cyberspace using electrodes, share other peoples’ experiences through stimsim, gain new skills by inserting software into their head ports and upload their entire personalities and memories into hardware constructs.
Brain-computer interfaces
This blog looks at neuro-prosthetics or brain computer interfaces that access the brain directly, such as implants and EEG. It excludes optical and auditory devices that do not directly access the brain, which are a huge topic in themselves. (For example, retina implanted devices and cochlea implants which had been implanted in more than 70,000 people by 2005 according to the Scientific American.)This article also passes on another weighty topic – brain stimulators - which had been implanted into more than 30,000 people suffering from Parkinson’s disease and other movement disorders by 2005 (Scientific American).
Another note is the definition, which I borrow from a research paper: a brain-computer interface (BCI), sometimes called a direct neural interface or a brain-machine interface, accepts voluntary commands directly from the human brain without requiring physical movement and can be used to operate a computer or other technologies (Levine et al, 2000).
And before I begin, thanks to Wired Magazine, on whose excellent journalism this blog is based.
Send in the animals
Like much medical research, legions of animal test subjects have been used to demonstrate the feasibility of brain-computer interfaces. One result of this research was to demonstrate the amazing plasticity of mammal brains to adapt to implants. Experimental animals have treated the external devices, including robot arms, controlled by their implants as their own limbs and their brains have modified themselves to respond to signals from implants.
Early wet brain-implant experiments in animals hark back to the 1950s when Jose Delgado (pdf) became infamous for implanting electrodes into the brains of live animals (and later humans) and stimulating them using a stimoceiver (a radio receiver) planted underneath the skull. Using signals sent through the electrodes in a technique called ESB – electronic stimulation of the brain – Delgado was able to produce basic behavioural effects in human and animal subject, such as calming or producing aggressive behaviour.
In the early 1980s, Apostolos Georgopoulos at Johns Hopkins University recorded the electrical activity of single motor-cortex neurons – responsible for controlling movement – in macaque monkeys. He found a mathematical relationship between the reactions of nerve cells and the direction that a monkey moved its arm. He also found that dispersed groups of neurons in different areas of the brain collectively controlled motor commands but due to technical limitations was only able to record the firings of neurons in one area at a time.These technological hurdles were overcome in 1993 by researchers including Miguel Nicolelis. Using rats, the team recorded 48 neurons spread across the five brain structures that form a rat’s sensorimotor system, which uses sensory information to direct movements. The technological breakthrough was new electrode arrays containing Teflon-coated, stainless-steel microwires that could be implanted in an animal’s brain for months at a time.
Before this, neurophysiologists had used standard electrodes that resemble rigid needles to record single neurons. These classic electrodes only worked for a few hours, as they swiftly became coated with cellular chemicals, which insulated them from the current. The stiff pins also damaged neurons with the brain's slight movements. The new electrodes flexed to limit damage and were not coated by cell chemicals. Having electrodes that produced data for months at a time allowed the development of hardware and software to transform the neural signals from multiple electrodes into commands to control a machine. The new setup allowed the researchers to discriminate electrical activity from four single neurons per microwire.
The eyes of the cat
In 1999, researchers led by Garrett Stanley at Harvard University decoded neuronal firings to reproduce images seen by a cat. The team used an array of electrodes embedded in the thalamus (which integrates all of the brain’s sensory input) of a sharp-eyed cat. They targeted 177 brain cells in an area of the thalamus called the lateral geniculate nucleus. The researchers showed cats eight short movies and recorded the neuron firings. Using mathematical filters, the team decoded the signals to generate movies of what the cats saw and were able to reconstruct recognisable scenes and moving objects (see pictures).
In 1999, John Chapin and Miguel Nicolelis of the MCP Hahnemann University School of Medicine in Philadelphia, were first to show that brain cell activity in mammals could be used to control a robotic device. Until this point, attempts to get animals to control artificial limbs had been based on signals recorded from muscles in the stump of an amputated limb, or electrical brain signals detected at the surface of the scalp.
In the experiment, rats were trained to obtain water using a robotic arm by pressing a small lever. Signals produced by the tasks in regions of the brain that control movement were monitored by electrodes implanted in the rats’ brains. The scientists identified specific brain cell activity associated with the rat’s paw movements and developed a mathematical model to analyse the signals.
The researchers then connected the robot arm directly to the rat’s brain. The rats had little difficulty in controlling the robot arm and many eventually learned that they could obtain water through brain activity alone and stopped pressing the lever. The device read signals from 48 neurons and allowed single dimensional control of the robot arm.
Wired monkeys
Results from primates soon followed.
In 2000, Miguel Nicolelis and John Chapin, at Duke University in North Carolina used brain implants to train an owl monkey to control a robotic arm. The monkey was trained to use a joystick to lift a barrier so they could reach across and grab pieces of fruit. These grabbing arm movements were monitored using fibre optic sensors and the firings from 100 neurons were mapped. Once the implant was switched on, the monkey's movements to grab the fruit were echoed by robot arms in another room and in another lab.
Mathematical analysis showed that that the accuracy of the robot movements was roughly proportional to the number of neurons recorded, but began to taper off as the number increased. A sample of 100 neurons could create robot arm movements that were about 70% similar to the monkey’s. The team estimated that as few as 500 to 700 neurons would be necessary for 95 percent accuracy in one-dimensional hand movements.
But the experiments only solved one piece of the puzzle. The monkeys were unaware they were moving the arm and did not use sensory feed back to control it (a so-called open-loop system). Nicolelis continuing experiments in 2003 addressed this issue and also adapted the interface to work with a more complex primate, one with a brain with deep furrows and convolutions that more closely resembled a human’s – the macaque monkey.
The experiment used joystick-wielding macacques and an implant that mapped 90 neurons connected to a robotic arm. After the team had mapped the signals, the implant was switched on. At first, monkeys kept moving the joystick, but later realised they could stop moving their arm but still control the game, and later, the arm directly. The researchers said the monkeys appeared to be treating the robot arm as their limb, not an extension. The experiment also marked the first time that brain implants were used to move a mechanical object in three dimensions. Continuing experiments include modifying the implant to provide the monkey’s with tactile feedback using patches on the monkey’s skin with vibrating force sensors. The BCI will control a nearby robot arm fitted with a gripper that simulates a grasping hand.
Computational ape
In 2002, implanted monkeys were trained to move a cursor on a computer
screen by
researchers at Brown University, led by John Donoghue. Around 100 micro-electrodes were use to tap up to 30 neurons, but because the electrodes targeted neurons that controlled movement, only three minutes of data were needed to create a model that could interpret the brain signals as specific movements. The monkeys were trained to play a pinball game where they were rewarded by quickly and accurately moving the cursor to meet a red target dot. (Link to movies of neural activity recorded at Donoghue's lab).Donaghue’s research used Richard Normann’s Utah electrode array, which has been licensed to other researchers and later to become the basis of the BrainGate chip offered by his company Cyberkinetics. The Utah array was developed by Richard Normann at the University of Utah’s Department of Bioengineering in the 1990s as part of a visual prosthetic to send signals into the brain (see below). The array is a silicon substrate that rests on the surface of the cortex, embedding the platinum-tipped electrodes into the brain’s grey matter. Donoghue realised the array could also be used as an uplink. The array also reduces the risk of damage because it floats with the movement of respiration and blood flow. It had also been refined to be more biocompatible by stripping much of the metal away, making it less likely to cause scarring.
In 2004, researchers led by Richard Andersen at the California Institute of Technology also successfully gave monkeys neural prosthetics to control computer cursors. However, instead implanting their electrodes in the motor cortex, they chose the posterior parietal cortex and the high-level premotor cortex, which produce cognitive signals and are involved in higher brain functions related to movement planning. Jiping He at Arizona State University trained implanted monkeys to control a cursors in 3D virtual space by thought.
Researchers using mammals have found that experimental subjects’ brains have adapted to control the interfaces. As the animals perfected their tasks, the properties and contributions of neurons changed over several days or even during a two-hour recording session. As well as demonstrating the brain’s plasticity, this showed the need to record the firings of multiple neurons and software that could respond to neural changes so a BMI could handle changes in neurons over months and years.
First brain implants for restoring sight
Curing blindness has driven some of the earliest experimentation involving brain implants in humans.
One of the first scientists to come up with a working brain interface to restore sight was private researcher, William Dobelle. Dobelle had been working on an artificial-vision that could restore sight since 1968. After having his own eye cut open to test the feasibility of a retinal implant, he concluded the only way to create a working visual neuro-prosthesis was to attach an implant directly to the brain.
His first prototype was implanted into Jerry, a man blinded in adulthood, in 1978. Neurosurgeons embedded a 1-inch-square grid containing 68 electrodes onto Jerry’s visual cortex. Stimulating the electrodes produced phosphenes. Scientists have known since 1929 that direct electrical stimulation of the brain’s visual cortex can cause blind people to perceive small points of light known as phosphenes.
Initially the implant allowed Jerry to see shades of grey in a limited field of vision and at a low frame-rate also requiring him to be hooked up to a two-ton mainframe. Shrinking electronics and faster computers made his artificial eye more portable and allowed him to go as far as performing simple tasks unassisted. The system included TV cameras mounted on glasses which send signals to a portable computer which stimulates the implanted electrodes.
Amazingly, the percutaneous pedestal, or carbon plug, embedded in Jerry’s head proved resilient to infection and did not cause major biocompatibility problems. Dobelle had moved forward slowly because of these concerns and for years the prototype in Jerry’s occipital lobe was largely unused. Jerry remained infection-free because of the scalp’s heavy blood flow which constantly brought immune cells near the site.
In 2002, Jens Naumann, also blinded in adulthood, became the first in a series of 16 paying patients to receive Dobelle’s second generation implant, marking one of the earliest commercial uses of brain implants. The second generation device used many of the same basic components as the first (implant, video cameras, signal processor) but the implant itself was more sophisticated and the package enabled better mapping of phosphenes into coherent vision. Phosphenes are spread out across the visual field in what researchers call the starry-night effect. Immediately after his implant, Naumann was able to use his imperfectly restored vision to drive slowly around the parking area of the Lisbon research institute.Cheri Robertson was the 16th and last Dobelle implant recipient in 2005. William Dobelle died in October 2004 from complications of diabetes. Stony Brook University and Avery Biomedical Devices Inc are continuing development of the Dobelle implant.
Several other teams are experimenting with brain implants for vision and there have been significant advances in technology since Jens. Dick Normann's Utah Array is a much smaller implant that penetrates the visual cortex rather than resting on its surface as Dobelle’s did. This requires less power (1-10 microamps) to reach individual neurons. The 1-10 milliamp-range of the Dobelle implant zapped more neurons leading to a much greater risk of seizures. Devices using lower current also better resolution because more electrodes can be packed on an implant. Normann’s implant contains a 32-by-32 array or 1,024 pixels, around 10 times the count on Dobelle’s implant, and produces vision good enough to distinguish objects. Other teams also working on visual cortex implants include European group CORTIVIS, University of New South Wales (PDF), Illinois Institute of Technology, Ligon Research Center of Vision, PolySTIM in Canada and NIDEK in Japan.Cortical prostheses are expected to eventually provide sharp central vision; close to 20/20, according to an expert in the field (Gislin Dagnelie at John Hopkins University School of Medicine). Other approaches are seek to avoid the problems of phosphene mapping by targetting parts of the brain responsible for earlier phases of vision generation. Harvard Medical School research focuses on implants in the lateral geniculate nucleus, mentioned above in research on cats. This part of the thalamus relays signals from the retina to the primary visual cortex and it would be easier to map the signals it receives from the eyes that mapping phosphenes. On the downside, a thalmus implant would require highly invasive neurosurgery.
Implants that get you going
Researchers at Emory University in Atlanta led by Philip Kennedy and Roy Bakay were first to install a brain implant in a human that produced signals of high enough quality to simulate movement. After a series of successful experiments with primates in the 1990s that used implanted electrodes, the team was granted two human tests by the US authorities. The first subject was in the terminal stages of a wasting disease and died two months after the procedure.
The second patient, 63-year-old Johnny Ray had suffered a brain-stem stroke in 1997, which produced ‘locked-in syndrome’, leaving him mentally intact but completely paralysed. The team was aiming to test a cortical implant that could acquire a signal from inside the brain that was strong enough to be processed to manipulate objects in the physical world.
The researchers sited the implants on Ray’s motor cortex. The implanted electrodes were small pieces of glass moulded into two narrow cones containing a gold electrical contact and secured to the skull bone to prevent movement. The cones' hollow space was filled with a special tissue culture to attracts brain cells to grow toward the contact. When brain cells touched the gold electrode, the electrical activity of individual cells could be detected and were carried out of the skull by gold wires and amplified.
Magnetic resonance imaging was used to compare changes in the motor cortex with the voltages produced by the electrodes while Ray thought of simple ideas corresponding to distinct movements, like up or down. The researchers extracted and encoded electrical patterns signifying the thoughts. By reproducing the signal using the same thought patterns, the signal can be used to control a cursor on a computer screen.
Ray’s implant was installed in 1998, and he lived long enough to start working with the signals, which were amplified and linked to a standard PC. Ray learned to move the cursor at will.
Again, the experiments demonstrated the brain's ability to adapt. Instead of using the atrophied natural areas of Ray’s brain that produce muscle motion, the implants were sited on the motor cortex. After months of testing Ray also reported that the thoughts he used to trigger the electrode were changing. Instead of imagining arm motions he was activating the electrode by thinking about facial movements. Eventually Ray no longer even required the imagined sensation of moving his body parts to control the cursor.
The researchers could even see his cheeks and eyes twitch as he moved the cursor. Ray’s thoughts about motion were triggering clusters of motor neurons. The implant had reactivated some of the motion centres in Ray’s brain enabling him to tap into capabilities rendered dormant by the stroke. Now the neurons, formerly disconnected from the body they controlled, had a crude way to interact. Click here to hear Phillip Kennedy speak.
Reaching out
As with the experiments on primates, once researchers had linked brain and computer cursor, it was just a matter of time before they linked brain to robotic device.
Quadriplegic Matt Nagle became the first person to control an artificial hand using a BCI in 2005. He was also the first human to receive Cyberkinetics Neurotechnology’s BrainGate chip-implant (the company founded to commercialise John Donaghue research). Implanted in Nagle’s motor cortex, the cyberkinetic interface allows Nagle to control not only a robotic arm by thinking about moving his hand but control a computer, lights and TV. He can also play Pong. Click here for videos of Matt Nagle operating the cursor from Nature magazine.Before implantation, magnetic resonance imaging was used to plot the region on Nagle’s motor cortex most likely to provide readable arm-movement signals to the BrainGate. The array contained 100 electrodes and the implant is wired to a refrigerator-sized trolley of electronics.
Nagle learned to use the interface in two or three days, starting with training to decode the firing patterns he produced by thinking about basic movements. Once training was complete, Nagle mentally pictured the motions to guide a cursor. As his brain adapted, instead of operating the cursor by imagining he was moving a mouse with his hand, he was able to imagine just moving the cursor.
As part of a pilot trial, Cyberkinetics is implanting another five paralysed patients with the BrainGate and is planning a future wireless version. Donoghue aims at eventually connecting the BrainGate to stimulators that can activate muscles, undoing paralysis by bypassing the damaged nervous system. But in theory, the BrainGate could be used to control anything controllable by silicon chips.
Electrode caps
Non-invasive brain interfaces are also progressing in leaps and bounds. Recording using electrodes worn on the scalp - electroencephalography (EEG) – once thought to be insufficiently sensitive to intercept brain impulses to drive brain machines is proving able to extract the signals needed to encode limited arm and hand movements.
In research beginning in the mid-1990s, Niels Birbaumer of the University of Tübingen in Germany used EEG recordings and a computer interface to help paralysed patients learn to train their EEG activity to select letters on a computer screen. The Thought Translation Device was the result.
The devices had its genesis in the 1990s when Birbaumer was trying to stave off impending seizures in epileptics. His team discovered that controlling the brain’s 'slow cortical potential' could derail an impending epileptic seizure. The researchers were able to teach epileptics to influence their slow cortical potential and Birbaumer realised the technique could be used to assist paralysed patients. A group of ten test patients were trained to control their brainwaves . They were hooked up to an EEG machine which was linked to a signal processor that translated their brain activity to move a computer cursor. However, the process was slow requiring more than an hour to generate 100 characters and training process was time consuming, often taking many months to learn.
In 2000, Birbaumer teamed up with Jonathan Wolpaw at at New York State University to develop technologies that would allow users to choose the brain signals worked best for them to operate their BCI. Walpow had succeeded in training four subjects to use their brains' mu and beta waves to power an EEG to move computer cursors in two dimensions. The device can also detect P300 signals, produced involuntarily when people recognise objects, which may allow different categories of thought to be decoded by a BCI without having to train patients to regulate their brain activity. (P300 waves are the result of stimulus-feedback rather than biofeedback.)
P300 waves have produced experimental results. In 2000, volunteers wearing virtual reality helmets controlled elements in a virtual world using their EEG readings, including turning lights on and off and bringing a mock-up car to a stop. The research was run by Jessica Bayliss at the University of Rochester.
Movement by thought
In 1999, researchers at Case Western Reserve University in Cleveland used 64-electrode EEG skullcap to return some arm and hand movements to quadriplegic Jim Jatich. As Jatich concentrated on simple but opposite concepts like up and down, his beta-rhythm EEG output was analysed using software to identify patterns in the noise. A basic pattern was identified and used to control a switch: Above average activity was set to on, below average off. This switch enabled Jatich to control a cursor, the nerve controllers embedded in his hands and, in theory, any other devices controllable by a computer chip.
Jatich had already become the first human to receive surgically implanted electrodes in his hands to stimulate muscles and create movement, in a series of operations and therapies starting in 1986. The electrodes stimulated the muscles with bursts of electricity and were controlled with a shoulder-mounted joystick, enabling Jatich to open and close his hands. The technology has also been used by other patients to move limbs.
The technology for nerve stimulation is now highly advanced. The electrodes in Jatich’s hands are barely visible and require almost no maintenance. Jatich has used a shoulder joystick controller to move his right hand since the mid-1980s. His hands are controlled by implantable joint angle transducer, which employs a magnet and sensor attached to his wrist bones. A team of Austrian researchers has also taught a quadriplegic patient to open and close a prosthetic hand using an EEG skullcap. The patient again required substantial training lasting five months.
In 2006, Advanced Telecommunications Research Institute and Honda Research Institute Japan claimed success in developing a near real-time BMI capable of manipulating robots using non-invasive MRI recordings of brain signals. The technique uses MRI to decode neural signals to allow a robot hand to mimic the subject’s finger movements by tracking the hemodynamic responses in the brain. The researchers reports a seven second time lag between the subject’s movement and the robot’s mimicked movement, but decoding accuracy of 85%.
Getting some feedback from virtual reality
Michael Persinger at Laurentian University Ontario is using electromagnetic fields to stimulate the temporal lobe and produce sensations in the brain. This promises another way of delivering feedback to users of brain implants, or even the experience of immersive virtual reality. Persinger has stimulated the temporal lobes of around 1,000 people, using caps studded with electrodes. Like Jose Delgado’s stimoceiver, Persinger’s cerebral stimulation produces sensations such as feelings of the paranormal, euphoria, anxiety, fear and even sexual stirring. Certain bursts of these electromagnetic patterns have been decoded and tend to produce their intended effects with great regularity. The possibility of marrying Persinger’s helmet with virtual reality is s already being investigated.
Ditching the grey matter
In 2003, California researchers began tests on the first brain prosthesis, a silicon chip implant designed to function as an artificial hippocampus. The implant is designed by a team led by Theodore Berger at the University of Southern California. The prosthesis is being tested on rat-brain tissue and later on live animals. It is intended to help people who have suffered brain damage due to stroke, epilepsy or Alzheimer’s disease and researchers expect human trials before the end of the decade.
The artificial hippocampus is being designed as a prototype for the eventual development of higher-brain prosthesis. The hippocampus is an ideal testbed: it's the most ordered and structured part of the brain and the most studied area. Its function is to encode experiences for storage as long-term memories elsewhere in the brain, although how it does this isn’t known.
To build the device, the researchers designed a mathematical model of how the hippocampus performs under all possible conditions, built the model into a silicon chip and will eventually interface the chip with brains. As science doesn’t yet understand how the hippocampus encodes information, the team simply copied its behaviour. Slices of rat hippocampus were stimulated with millions of patterns of electrical signals until researchers could associate a definite electrical input with a corresponding output. Putting the information from various slices of the hippocampus together gave the team a mathematical model of the entire organ.
In a human patient the chip would sit on the skull rather than inside the brain and communicate with the brain through two arrays of electrodes. One to record electrical activity input from the rest of the brain and another to output correctly coded electrical instructions back to the brain.
How soon will the future arrive?
Companies are already staking out their claim to the future of brain implants. Already developing electrode array based invasive brain interfaces are Cyberkinetics Neurotechnology Systems, Inc. (BrainGate) and Neural Signals, Inc. (Brain Communicator).
Researchers working on animals and humans are capable of monitoring their more and more neurons. That includes mounting more microwires in electrode arrays and implanting more arrays in the brain. For example, the latest arrays used by Nicoleis at Duke university measure five by eight millimetres and contain 160 microwires.
In one experiment 704 microwires were implanted in a macquae recording 318 neurons simultaneously. The positioning of the electrodes is set to become more accurate. Jit Muthuswamy at Arizona State University has developed a micro-motor that can reposition electrodes implanted in the brain. The device is being tested on rats.
Other advances include wireless BCIs to unwire subjects from computers. Researchers are miniaturizing electronics and batteries and hardwiring software into implantable microelectronics alongside wireless chips. The first wireless neurochips have already been tested in macques. Mohammad Mojarradi of the Jet Propulsion Laboratory in California, is working not only to miniturize BMI hardware but to replace electrodes entirely. Current electrode arrays can be jarred out of place and lose the signal and wear away the tissue on which they sit. Mojarradi wants to eliminate the electrodes and their wires, replacing them with implanted wireless chips.
Million-dollar grants are now on offer to investigate the possibilities of direct interaction with the brain. While most of the money is chasing therapeutic work to restore sight or unlock paralysis, recent advances have raised the possibility of making new links to the brain beyond what we’re born with. US security research agencies, such as the Defence Advanced Research Projects Agency have shown interest. Although the civilian authorities might not like it, there are few remaining scientific barriers preventing a skilful pioneer from trying to wire in something new upstairs.
Further reading
- BCI-info, Portal for Brain Computer Interfaces
- Theodore Berger, University of Southern California
- John Chapin's page at Downstate University
- Cyberkinetics Neurotechnology Systems (company site), (Wikipedia entry)
- Jose Delgado, excerpts from the Physical Control of the Mind and Scientific American article (pdf)
- John Donoghue’s lab at Brown University
- S. P. Levine, J. E. Huggins, S. L. BeMent, R. K. Kushwaha, L. A. Schuh, M. M. Rohde, E. A. Passaro, D. A. Ross, K. V. Elisevich, and B. J. Smith, "A direct brain interface based on event-related potentials," IEEE Trans Rehabil Eng, vol. 8, pp. 180-5, 2000.
- Miguel Nicolelis’ lab at Duke University
- Scientific American article (pdf) on Miguel Nicolelis
- Garrett Stanley at Harvard University
- Wikipedia – Brain implant and Brain-Computer Interface entries
- Wired Magazine article on artificial vision
- Wired Magazine article on Jim Jatich’s implant
- Wired Magazine article on Matt Nagel’s implant
- Wired News article non-invasive implant for typing
