Web Posted on: February 13, 1998
OLD DOG, NEW TRICKS. BIOFEEDBACK AS ASSISTIVE TECHNOLOGY
Chris Van Raalte
The BodySynth(TM)
2007 Franklin Street
San Francisco, CA 94109
(415) 928-3790
Biofeedback has the goal of achieving voluntary control of a physiological function by means of self monitoring. Instrumentation routinely detects subtle parameters such as muscle activity (sEMG), brain waves (EEG) and galvanic skin response. Practitioners monitor these signals by means of audio or visual feedback.
Technological advances have allowed developers to consider using biofeedback for an entirely new application which is to use these biosignals to control an external device such as a computer. As such, these signals have the potential to be an excellent source of control for assistive devices. For example, the use of EEG to control things by merely thinking about them is now considered a realistic possibility. Eyes can move a computer cursor, and muscles can make music. The old dog, Biofeedback, is indeed learning new tricks. This paper describes The BodySynth(TM) and discusses its use as musical assistive device in recreation therapy.
The physiological parameter used by The BodySynth(TM) is called the surface electromyographic signal (sEMG). This is measurable as a very small AC voltage (on the order of 100 microvolts) produced when a muscle contracts. Using what amounts to a sophisticated volt meter connected to a pair of electrodes in good contact with the skin, a reliable output can be obtained. The output is directly proportional to the degree of tension in the muscle fibers beneath the electrodes. That is, the more muscle tension, the stronger the signal.
The present interest in music therapy has evolved from the original use of The BodySynth(TM) as a musical performance tool for dancers. The concept was (and still is) for the device to convert a performer's movements into music. Since a performer must be unencumbered, the device had to be small enough to be worn comfortably. It also required a significant amount of computing power to transform the sEMG data stream into something recognizable as music.
The system described below was designed and built in collaboration with my associate, Ed Severinghaus. It has three main components: Body Unit, Processor and Sound Gear. Electrodes are strapped over the sampled muscle and connected by cables to the Body Unit. The Body Unit (approximately the size of a cigarette pack) contains four channels of sEMG amplification and conditioning. A multiplexer combines the signals for ease of transmission. A separate commercially available transmitter (also worn) sends the sEMG data to the Processor and Sound Gear. In order to maintain portability and keep costs down, we chose not to depend on a PC. Instead we used a dedicated microprocessor and wrote our own firmware to run it. The sEMG data are converted by the Processor into various musical and trigger control parameters and sent via MIDI (Musical Instrument Digital Interface) to commercially available Sound Gear such as a digital synthesizer, sample player, sequencer or computer. The timbre of the "instrument" being played is chosen on the Sound Gear. The Processor is capable of storing a series of operational parameters in "Patches" which play or modulate preselected sounds in predetermined modes. The four muscle inputs can generate up to eight MIDI outputs which permits the creation of complex sound designs.
There are two basic play modes of The BodySynth(TM) for use in biofeedback. In one, the "Trigger Mode", muscle effort is used to trigger a specific note event. The pitch and timing of the note is determined by the amount and timing of the muscle contraction. The other mode of play is called "Autotrigger Mode". In this mode, the Processor provides a simple rhythmic pattern which is modulated by the user's muscle effort.
In Trigger Mode an increase in the user's effort typically results in a higher pitched sound. Additional sEMG channels can be used to control volume, note duration and other sound parameters to achieve a subtle musical output. The Trigger Mode is used to directly emulate the playing of a traditional musical instrument. As such, training and practice are directly related to the level of skill achieved.
For persons who want to play music, but have physical handicaps that prevent them from doing so (ie. disability in upper limbs), The BodySynth(TM) played in Trigger Mode is an ideal assistive device. Functioning muscles are recruited and trained to take the place of ones that are dysfunctional. Since any surface muscle is capable of triggering the device, it can be customized to fit the abilities of the user. For example, in cases of quadriplegia, neck or facial muscle can be used as triggers.
The BodySynth(TM) in the Trigger Mode is particularly good at emulating drum and percussion sounds which do not demand the degree of pitch control required by say, a flute. The device mimics an acoustic drum by getting louder and slightly higher in pitch as the user "hits" it harder. It is very realistic and intuitive. The bigger the gesture the bigger the sound. The challenge for the BodySynth(TM) drummer lies in training alternative muscles to play to play an accurate rhythm. Theoretically this should be no more difficult than learning how to play any drum from "scratch" using hands and fingers.
Playing a pitched instrument like a guitar or piano in Trigger Mode is more difficult. Significantly more practice and fine motor skills are needed to not only play notes at the correct time but at the correct pitch as well. Based on my own success in learning to play the machine, however, I believe that any disabled individual with the desire and requisite skills can have a realistic artistic experience playing music with The BodySynth(TM).
By contrast, in Autotrigger Mode zero or low effort in the sampled muscle typically results in no sound. As effort increases beyond a threshold, the rhythmic sequence becomes audible. Further increases in effort change the volume and/or pitch of the notes in the sequence. More thresholds may be added, that when crossed, generate entirely new sequences.
Playing The BodySynth(TM) in the Autotrigger Mode is a much less demanding task both physically and mentally than playing in the Trigger Mode; the burden of maintaining a rhythm has been removed. For this reason it is ideal for individuals with developmental and/or physical disabilities and for those who simply don't want to practice. For example, a simple wave of the hand is easily transformed into a woodwind arpeggio. Electrodes placed anywhere around the user's wrist trigger The BodySynth(TM) to play flute in a quarter note sequence limited to a one octave pentatonic scale. These parameters will always produce a pleasing pattern of notes that will repeat if the gesture is accurately repeated. Although accuracy is often a desirable goal in the Autotrigger Mode, it is not essential for an enjoyable musical experience.
In 1996 I contacted RCH Inc., a large recreation center for persons with disabilities in San Francisco. The goal of the institution is to stimulate the development of self-esteem, social interaction, health and physical fitness, and creative expression in its clients. It seemed logical that The BodySynth(TM) could be useful there. The staff welcomed the idea of introducing The BodySynth(TM) into some of their programs. These included several adult developmentally disabled groups, a brainstem injury group and a performing arts troupe. In turn I was provided with the opportunity to explore how The BodySynth(TM) might prove useful for persons with a wide range of disabilities. I have also been working with the Bridge School in Hillsborough, CA., an organization dedicated to ensuring that children with severe speech and physical impairments achieve full participation in their communities through the use of assistive technology. The following discussion is based on many hours of interaction with the staff of these institutions and the individuals attending them.
The work at these institutions revealed that The BodySynth(TM) is an excellent tool for stimulating group interaction. Music is a common reference point for nearly all the participants. Playing in a group teaches them cooperation and social skills and it provides a means of creative expression. Although the original BodySynth(TM) was meant to be played by a single individual, the flexibility of its design allows it to work with up to eight participants at once. This can be done by linking two systems together and synchronizing their Autotrigger clocks. We have found, however, that four participants, each playing one instrument is optimal. This number is sufficient to provide a "critical mass" needed for successful group interactions to occur. Leaders need to be careful in working with larger groups because is difficult to distinguish who is playing what. This tends to dilute the effectiveness of the experience for the participants and their attention spans drop.
A description of a session at the recreation center will serve to illustrate this application. A typical group of adult developmentally disabled individuals has up to eight participants and two staff. The four people chosen to begin the session are seated around a table. The staff attach electrodes to the appropriate limb on each participant. In these groups we usually sample forearm flexors or extensors. The electrode cables are then plugged into the Body Unit which stands in front of the leader on the table. This position allows the leader to monitor the sEMG activity of the participants by observing indicator four lights on the unit (LEDs which shine in proportion to sEMG activity). The leader can further control the session by adjusting different channels with the sensitivity controls on the Body Unit. This is helpful if some of the participants are very active and are "drowning" out the others with their sounds.
The leader selects from a menu of Patches programed to be played in Autotrigger Mode and the session begins. Typically, each participant will have a different instrument sound with its own distinct rhythmic pattern. One at a time, the leader encourages the participants to play alone. This allows them to identify their sound and to fully appreciate their ability to control it. Playing solo for their group helps promote self-esteem in the participant. Those who are not playing must learn to be quiet (relaxed) while others play. Nearly all the participants come to understand the relationship between their gestures and the sound at some level. Some take longer than others. For example, it was not until the third session that one participant who is nonverbal signaled his understanding by repeatedly looking back and forth between the electrodes on his arm and the loud speaker producing the sound. It was a classic "aha!" moment.
In addition to synthesized instrumental timbres, sound effects are very useful in group sessions. They work well for individuals who are not interested in music. Some sounds can be very amusing. Being able to make his or her group laugh can be a very compelling experience for a participant. Sound effects can also be used to tell stories and to role play.
The work at The Bridge School further illustrates how The BodySynth(TM) is able to foster group interaction. Although the children have severe speech and physical disabilities, they have sufficient control in some limbs to play in the Autotrigger Mode. They quickly grasped the concept. The BodySynth(TM) allowed the students to play the music and sound effects for their graduation play last Spring. I have used it in conjunction with visual aids to teach them the names and sounds of instruments as well as elementary music notation. I am continuing to conduct classes there which focus on improving motor skills and providing a means of creative expression. Since much of the academic work at the Bridge School occurs one on one, the group BodySynth(TM) sessions have the added benefit of teaching social skills and cooperation. Their enjoyment of these classes is apparent in their faces and body language. The staff attributes this enthusiasm to the fact that the device has given them a new means of communication and expression. For example one of the boys "went nuts" when he was able to gun the engine of a motorcycle by waving his arm. It was a very powerful experience for a 10 year old with such limited abilities to be able to make so much sound.
I have observed a similar reaction to the machine on the part of George, a nonverbal developmentally disabled participant at RCH Inc. According to his group leader, he rarely if ever gets out of his chair or interacts with other individuals in the group. However, when he enters the room to participate in a BodySynth(TM) session, he immediately thrusts out his arm in anticipation of the electrode strap. As soon as we begin, he stands and takes a dramatic pose, raises his arms above his head and gestures and sways to the sounds he makes. Clearly the machine has provided him with a new means of expression.
Another important use for BodySynth(TM) technology is in applications which are primarily therapeutic rather than recreational or aesthetic. The audio feedback provides both the user and caregiver with information about the activity of the sampled muscle which can be used in physical therapy. The BodySynth(TM) system produces audio output that ranges from dance rhythms to dynamic soundscapes. Since these designs are engaging and are fun to play and hear, they motivate the user to use (or not use) targeted muscles.
Some examples from a brainstem injury group at RCH Inc. will illustrate this point. Jim has a partially paralyzed left arm which he "forgets" to use. During sessions we target his left arm as a trigger. By having him play music in a group with others he was encouraged to use the injured limb. Mary is an elderly woman with a very flat affect and little speech ability. She holds her body motionless and keeps her hands in tight fists. With a bit of coaxing we were able to get her to extend her index finger which in turn triggered the music and brought a rare smile to her face. Once she understood that her movement caused the sound, the process became a motivational tool that helped her release her hand. Jeff's experience is a variation on this. He is a young man who is nonverbal and immobile. He suffers from chronic and nearly constant seizures in his arms. Electrodes placed on his forearms produced a continuous output of sound. Often Jeff was able to respond to his caregiver's verbal cues to "relax his hands" by reducing the pitch of notes and creating periods of silence. The reverse was also true: At times of relaxation he could play notes on command.
In summary, the author has found The BodySynth(TM) to be a versatile assistive device by which persons with a wide range of disabilities can control aspects of their audio environment. For individuals with the requisite skills and desire, The BodySynth(TM) can provide the means for serious musical artistic expression. For those with less physical ability and/or low cognitive skills, the device can be played in Autotrigger Mode and still provide an enjoyable musical experience. The BodySynth(TM) is an excellent tool for stimulating group interactions with music or theatrical sound effects. Lastly, creative sound designs provide a means to motivate physical movement.
Improvements are planned that will eliminate electrode cables, simplify the Processor interface and standardize the Sound Gear. There will be continued improvements to the Autotrigger Mode and a body of music will be developed to be played by it. The BodySynth(TM) has been successful in emulating a computer mouse. Further development of this application is planned to accommodate those individuals who respond better to visual than audio feedback. For example, the device will offer control of video games and other forms of graphic display.