Figures
Abstract
Handwriting – one of the most important developments in human culture – is also a methodological tool in several scientific disciplines, most importantly handwriting recognition methods, graphology and medical diagnostics. Previous studies have relied largely on the analyses of handwritten traces or kinematic analysis of handwriting; whereas electromyographic (EMG) signals associated with handwriting have received little attention. Here we show for the first time, a method in which EMG signals generated by hand and forearm muscles during handwriting activity are reliably translated into both algorithm-generated handwriting traces and font characters using decoding algorithms. Our results demonstrate the feasibility of recreating handwriting solely from EMG signals – the finding that can be utilized in computer peripherals and myoelectric prosthetic devices. Moreover, this approach may provide a rapid and sensitive method for diagnosing a variety of neurogenerative diseases before other symptoms become clear.
Citation: Linderman M, Lebedev MA, Erlichman JS (2009) Recognition of Handwriting from Electromyography. PLoS ONE 4(8): e6791. https://doi.org/10.1371/journal.pone.0006791
Editor: Wen-Liang Zhou, Sun Yat-Sen University, China
Received: April 17, 2009; Accepted: July 8, 2009; Published: August 26, 2009
Copyright: © 2009 Linderman et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This research was sponsored by SBIR grant from National Science Foundation. NSF grant# 0711799. www.nsf.gov. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Michael Linderman is paid employee and the owner of Norconnect Inc. Michael Linderman is going to develop products from this and future research. He is the author of two pending patents with US Patent Office. Recordation of handwriting and hand movement using electromyography 11640954. Handwriting EMG for Medical Diagnosis 61108603. Michael Linderman has 42% stock ownership in Norconnect Inc. He is also the board chairman at Norconnect Inc. This research was supported by the research grant #0711799 to Norconnect from National Science Foundation. Norconnect also won the second award from National Science Foundation to continue this research.
Introduction
The development of systems that can interface bioelectric activity to external devices hold significant clinical promise. For example, neural prosthetics strive to restore limb mobility and communication capacity in disabled subjects by interfacing brain potentials [1], [2], [3] or EMG activity [4], [5], [6] to artificial actuators or devices based on functional electrical stimulation [7], [8]. In addition to clinical applications, it has been suggested that bioelectric interfaces may even be used to enhance certain functions in normal subjects [9]. Although handwriting is one of the most important motor activities, it has received little attention from the designers of bioelectric interfaces due to perceived technical limitations and the paucity of models [10]. An interface that converts human bioelectric activity into text records could have a number of broad applications. First the development of this technology could substitute for computer peripherals or touch screens which have typically been used to record and transmit text messages. Bioelectric interfaces potentially could extract normal handwriting patterns directly from hand and arm EMGs. Clinically, handwriting features have been used for diagnostic purposes for patients with Parkinson's disease [11] and more recently dysgraphia has been shown to be a conserved element in the progression of Alzheimer's disease [12]. Methods that could be used to model handwriting could be used to diagnose diseases with a graphomotor component or be used to grade the progression of the disease or treatment.
The goal of this study was to develop a hardware/software system to record bioelectrical signals from the forearm and hand muscles (Fig. 1) and decode these signals with algorithms to extract and reproduce handwritten characters (Figs. 2 and 3).
A: A photograph of a recording session. B: Electrode placement over the hand (top) and forearm muscles (bottom).
A: Schematics of the Wiener filter. EMG signals (rectified EMGs) from multiple models were fed into two independent Wiener filters which reconstructed X and Y coordinates of the pen, respectively. Each filter represented reconstructed coordinate as a weighted sum of EMGs. B: Examples of reconstructed traces from one recording session. Actual traces are shown in blue; reconstructed traces are shown in red. The first two columns show X(t) and Y(t), respectively. The third column shows X-Y plots.
A: Schematics of the algorithm. Compound EMG (the sum of all rectified EMGs) was first used to detect the periods during which handwriting occurred. Compound EMG was first segmented into epochs corresponding to individual characters using a threshold that detected EMG bursts. Then, a generic compound EMG template was calculated by averaging these epochs. Template matching was used to refine the EMG segments, which were then classified using linear discriminant analysis. B: Example of discrimination for a representative recording session. From top to bottom: Eight EMGs were used for character recognition. 3.5-s segments corresponding to individual characters are highlighted as blue bars which are aligned on peak correlation coefficient, R, for template matching. Posterior probabilities for character recognition which were computed by discriminant analysis are shown as color plots. Recognized font character which corresponds to the highest probability is shown near each plot. Original handwriting is shown at the bottom.
Results
We implemented two fundamental approaches for decoding handwriting from the EMGs. In the first approach, we reconstructed pen traces using linear decoding algorithm, the Wiener filter [13], [14] (Fig. 2). In the second approach, we recognized handwritten characters from the EMG patterns and displayed them as textual fonts. Thus, EMG patterns were mapped to discrete font characters. Both the reconstruction algorithms and the recognition algorithms had to be trained on the data from individual subjects and did not generalize to other subjects because of inter-subject variability.
As shown in Fig. 1B, bipolar surface EMG electrodes were placed on the skin overlying four forearm muscles and four hand muscles. Each of the muscles recorded exhibited EMG bursts during handwriting (Fig. 1A). Following conventional methodology [15], the intensity of EMG modulations was quantified as rectified EMG. To reconstruct the pen trace, the Wiener filters expressed X (left-right dimension) and Y (bottom-top) coordinates of the pen with respect to the writing surface as weighted sums of the rectified EMGs (Fig. 2A). The results of such reconstruction are shown in Fig. 2B. Pen traces recorded by the digitizing tablet are shown in blue, and the traces reconstructed from the EMGs are shown in red. The reconstructed traces followed the original handwriting with accuracy comparable to other bioelectrical interfaces [2].
The quality of reconstruction was evaluated as coefficient of determination, R2. R2 values for individual subjects and statistics for the whole group are presented in Table 1. For the 6 subjects involved in these experiments, R2 was 0.47±0.20 (mean±standard deviation across subjects) for X and 0.63±0.15 for Y. (R2 can range from 0 to 1, and it reflects the proportion of variance in the original data captured by the reconstruction.) Table 1 also shows R2 values for hand and forearm muscles. When only hand-muscle recordings were used for the reconstruction, R2 was 0.26±0.10 for X and 0.50±0.12 for Y. When only forearm-muscle recordings were used, R2 was 0.43±0.21 for X and 0.51±0.13 for Y. Pen position could be reconstructed even from EMGs of single muscles, although the accuracy was less compared to multiple-muscle reconstructions (Table 2). When the best reconstructing muscle was selected, R2 was 0.31±0.17 for X and 0.32±0.10 for Y (Table 1).
In the EMG recognition approach, we used linear discriminant analysis [16] to translate EMG patterns into font characters. Figure 3A illustrates the operation of written-character discrimination algorithm. The subjects were asked to write characters, numbers from “0” to “9” (50 repetitions per characters). A half of the records (250 randomly selected trials) were used as the training set for the discriminant analysis, and the remainder of the records was used as a sample set. During the training phase, 3.5 s epochs corresponding to single characters were selected using an algorithm in which bursts in compound EMG (the sum of rectified EMGs from all muscles) were detected which crossed a threshold (0.5 standard deviation from the intertrial level) designated the epoch onset. Averaging the EMGs over all detected epochs yielded a generic template. This template was entered in the template matching analysis in which a 3.5 s sliding window was moved along the EMG records, and the correlation coefficient between the EMGs and the template was continuously correlated. Character writing epochs were then refined using the occurrences of peak correlations as epoch onsets. These epochs were then entered in the linear discriminant analysis that recognized the characters. The quality of recognition was evaluated across the 6 subjects using the percent of correct recognitions as the measure. This percent was 90.4±7.0 (mean±standard deviation across subjects). When hand muscle EMGs were analyzed separately (Table 1) the percent was 79.2±10.6, and for the forearm muscles we obtained the value 83.5±10.0. When a single best reconstructing muscle was selected, the percent of correct reconstructions was 65.9±11.4 (Table 1), and the average percent correct for any single muscle was 51.6±12.5 (Table 2).
As shown in Fig. 4, the performance of both the reconstruction algorithm (Fig. 4A,B) and the recognition algorithm (Fig. 4C) benefited from the EMG recordings from multiple muscles. Further, we estimated the performance of discriminate analysis for different amounts of training data. In this analysis, different amounts of data were taken from the recordings as the training set, and the rest of the data was used as a sample set. Analysis of a representative experimental session is shown in Fig. 4D. A minimum of five repetitions per font character were needed for the discriminant analysis to work. Recognition accuracy was 63% correct for this amount of training data. As the number of repetitions increased to 35 per character, recognition improved to 97% correct.
The analyses were conducted were all muscles and hand and forearm muscles only (see key on top). A: Reconstruction accuracy of the X-coordinate of the pen as the function of number of muscles recorded. Muscles were taken in different combinations, and R2 was averaged across these combinations and across subjects. B: Recognition accuracy of the Y-coordinate. C: Recognition accuracy as the function of the number of recorded muscles. D: Improvement in recognition accuracy as the function of training set size.
Discussion
Thus, we have shown that EMGs of hand and arm muscles can be converted into handwriting patterns: either the actual handwriting traces or font characters. This demonstration opens a number of directions for future research and practical applications. First, we have shown that EMG-based technology is a viable alternative to traditional methods of record taking. We envision a computer peripheral, an EMG glove, in which electrical activity of hand and/or forearm muscles are streamed directly to a computer where mathematical algorithms transform it into font characters. Such technology can employ dry electrode attachments and can be interfaced to the computer using wireless technology. As such, it may both offer certain advantages to conventional digitizing technologies, such as digitizing tablets, and become a useful supplement to these technologies. Our methodology can be also applied to clinical studies. While handwriting is impaired in dementia [17], Parkinson's disease [18], writing tremor [19], and attention deficit hyperactivity disorder [20], EMG changes during these handwriting impairments are poorly understood. Our techniques for extracting handwriting patterns from hand and forearm EMGs may contribute to both clinical research and development of clinical devices that would assist patients with impaired handwriting.
Methods
EMG and handwriting recordings
This study was approved by the Institutional Review Board of St. Lawrence University, Canton, NY. The data were recorded from 6 subjects. No personal information was recorded during sessions and all data were analyzed anonymously. Written informed consent was obtained from the subjects prior to the EMG recording sessions.
Each subject was comfortably seated at a desk in front of a computer monitor and wrote on a digitizing table with a pen while looking at the computer monitor that displayed the written traces. EMGs of 8 muscles were simultaneously recorded (Fig. 1a). Since the handwriting involves both the finger and wrist movements, surface EMGs were recorded from intrinsic hand and forearm muscles that produce these movements (Fig. 1b). Bipolar surface EMG electrodes were placed on four forearm muscles: flexor carpi radialis (FCR), extensor digitorum (ED), extensor carpi ulnaris (ECU), extensor carpi radialis (ECR) and four intrinsic hand muscles: opponens pollicis (OP), abductor pollicis brevis (APB), and medial (mFDI) and lateral (lFDI) heads of first dorsal interrosseus). The grounding electrode was placed on the subjects forehead. The skin surface overlying the muscles of interest was first cleaned with alcohol and the electrodes were prepped with electrode paste, firmly pressed to the skin, and fixed in place with hypoallergenic tape. After the electrodes were attached, the whole assembly was wrapped with an elastic bandage (Fig. 1a) to fix the electrode leads to minimize the occurrence of mechanical artifacts in the recordings.
Our system acquired data from three sources: EMG signals were obtained using a amplified recording system (Grass, model 15LT/15A54-2 quad modules) with a digitizer (Polyview/16SYS), pen tracking was accomplished using an Intuos 3/Wacom tablet, and pen contact was detected with pressure sensitive piezo film attached to the tablet. The tablet data were sampled at 100 Hz, which included mouse click events generated when the pen touched the tablet or was lifted from the tablet. EMG signals were differentially amplified (1000X gain), band-pass filtered (-6 dB cutoff points at 5 Hz and 500 Hz) and sampled at 1 kHz per channel. The pressure sensitive piezo film was placed on the writing surface of the tablet and the purpose of the film was simply to indicate the onset of the writing session by generating a triggering pulse (5 V, 0.05 s) sent to one channel of the EMG amplifier. EMG acquisition was handled by Grass software, and a MATLAB script controlled the acquisition of the data from the tablet.
Each subject was instructed to write numeric characters from “0” to “9”. The handwritten traces were displayed on the computer monitor mounted in front of the subject. The subjects were instructed to write the numbers within an 8×8 cm writing area on the digitizing tablet using their individual handwriting patterns. Each subject held the pen in the hand according to his/her individual habits. By a trial we define a recording epoch during which a subject wrote a single character. Each character was successively written 50 times. Therefore, each subject wrote 500 characters (i.e. performed 500 trials) during a daily recording session. Subjects could rest for a few minutes in between the recordings of individual characters, but the electrodes were not removed. Since in this study we sought to recognize individual characters, the subjects were asked to make pauses in between the characters. The trials were paced by the computer software which displayed a fresh writing area in the beginning of each trial. The duration of each trial was 7 s of which 2–3 s corresponded to character writing. Representative examples of the EMG signals and handwriting traces are shown in Fig. 3a.
Data analysis
Handwriting patterns were extracted from rectified EMGs. Rectified EMGs were calculated by full-wave rectification of the original EMG signals followed by low-pass filtering with a cutoff frequency of 5 Hz (second order Butterworth filter).
In the handwriting reconstruction algorithm, we reconstructed pen traces from the EMGs of eight muscles of the arm. Thus, the end result of this method was the trace of the character with the only difference that it was not actually written by a pen, but rather derived from the EMGs (Fig. 2b). Handwriting traces were extracted from the EMGs using a linear method, the Wiener [14]. In the recognition algorithm, we recognized the characters written by the subjects by comparing the EMG patterns to a set of previously obtained EMGs. The recognition choice was the character whose previously recorded EMG patterns most closely matched the examined EMG pattern.
In both the reconstruction and recognition methods, the analysis consisted of two steps: (1) training the algorithm and (2) decoding using the trained algorithm. Accordingly, the experimental data (50 trials per each of 10 numerical characters) were split into two separate parts: (1) training data and (2) decoding data. To split the data into these parts, we simply used the first half of the record for each character for training and the second part for decoding. Thus, 250 trials (25 trials per character) were used for training and separate 250 trials were used for decoding (cross-validation) within one recording session for one subject.
To reconstruct handwriting into traces, we started with the application of a linear model that decoded pen coordinates x(t) and y(t) as a weighted linear combination of the EMG inputs:(1)where x(t) is X-coordinate at time
,
is a vector of input signals (rectified EMGs on 8 channels), at time
and time-shift
(negative shifts correspond to past values, positive shifts correspond to future values), T is the time window for the lags were,
is a vector of weights for each input at time-lag
,
is the y-intercept, and
is the residual error.
Wiener filter defined by equation 1 reconstructs pen position from the EMG activity of several muscles sampled in the interval –T to T centered on the time point for which the reconstructed variable is calculated. EMG samples in this window are given different weights to optimally map time-varying EMG activity pen coordinates.
Equation 1 was solved using linear least squares regression (MATLAB routine regress). This equation can be recast in matrix form as(2)where x, w and
are column vectors, N is a matrix and b is a scalar. Rows in x and N correspond to time t = {tstart, tstart + step, tstart + 2step, …, tend}, and rows in w correspond to lags
= {−T, −T + step, −T + 2step, … T} and recording channels. In this notation, matrix N contains lagged data and thus has a column for each lag and each channel. The y-intercept is handled by prepending a column of ones to matrix N. The weights w are solved by
(3)where N and x are taken from model training datasets.
Linear model decoding for the y-coordinate of the pen was performed the same way as the decoding for the x-coordinate (equations 1–3).
For character recognition, we used Fischer linear discriminant analysis [16] to translate EMGs into the text characters represented by a computer (Fig. 3). To detect the onset of handwriting for each character, we used the compound EMG calculated as the sum of all analyzed rectified EMGs. Compound EMG was then detected as the threshold crossing set to 0.5 standard deviations from the intertribal level. After these onsets were determined, the EMG record was segmented into 3.5 s epochs which represented the writing of each character. Further, these epochs were downsampled to 10 samples per second (or 100 ms bins). A generic EMG template was calculated by averaging the EMG records across the epochs representing individual characters. This template consisted of the templates for individual muscles (35 bins per muscle) stacked together. Then, the template was slid across the EMG records, and correlation coefficient, R, between the EMGs and the template was calculated. R was high when the template was aligned with character writing episodes. Peak R values were then used to better segment the EMGs into 3.5 s epochs corresponding to each character: the onsets of these epochs were set to the occurrences of peak R. These segments were entered in the discriminant analysis (MATLAB function classify) as the training data. To reduce data dimensionality, principal component analysis was used to preprocess the EMG data before the discriminant analysis step. Empirically, the best results were obtained when the number of parameters was reduced from 280 (35 bins for each of eight muscles) to 50 principal components.
Acknowledgments
We would like to thank the following people for their insight and contributions to this project: Dr. Carlo De Luca (mentoring), Dr. Andrew Fuglevand (guidance), Dr. Levent Burak Kara (tablet programming), Eamon Egan (design for recording synchronization), Dr. Mitch Thompson (piezoelectric application), Dr. Berrin Yanikoglu (pattern recognition), Dr. Ian Mc Keague (statistical analysis), Dr. Bjørg Elisabeth Kilavik (programming approaches), Dr. Fred Schlereth (comments), Dr. Arkady M. Pertsov (comments), Dr. Danil Prokhorov (comments).
All research involving human participants have been approved by the Institutional Review Board of St. Lawrence University, Canton, NY.
Author Contributions
Conceived and designed the experiments: ML MAL JSE. Performed the experiments: ML JSE. Analyzed the data: ML MAL. Wrote the paper: ML MAL JSE.
References
- 1. Birbaumer N, Murguialday AR, Cohen L (2008) Brain-computer interface in paralysis. Curr Opin Neurol 21: 634–638.
- 2. Lebedev MA, Nicolelis MA (2006) Brain-machine interfaces: past, present and future. Trends Neurosci 29: 536–546.
- 3. Wolpaw JR, McFarland DJ (2004) Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. PNAS 101: 17849–17854.
- 4. Ohnishi K, Weir RF, Kuiken TA (2007) Neural machine interfaces for controlling multifunctional powered upper-limb prostheses. Expert Rev Med Devices 4: 43–53.
- 5. Parker P, Englehart K, Hudgins B (2006) Myoelectric signal processing for control of powered limb prostheses. J Electromyogr Kinesiol 16: 541–548.
- 6. Uellendahl JE (2000) Upper extremity myoelectric prosthetics. Phys Med Rehabil Clin N Am 11: 639–652.
- 7. Seifert HM, Fuglevand AJ (2002) Restoration of movement using functional electrical stimulation and Bayes' theorem. J Neurosci 22: 9465–9474.
- 8. Anderson CV, Fuglevand AJ (2008) Probability-based prediction of activity in multiple arm muscles: implications for functional electrical stimulation. J Neurophysiol 100: 482–494.
- 9. McGee EM, Maguire GQ Jr (2007) Becoming borg to become immortal: regulating brain implant technologies. Camb Q Healthc Ethics 16: 291–302.
- 10. McKeague IW (2005) A Statistical model for signature verification. J Amer Statistical Assoc 100: 231–241.
- 11. Van Gemmert AW, Teulings HL, Contreras-Vidal JL, Stelmach GE (1999) Parkinson's disease and the control of size and speed in handwriting. Neuropsychologia 37: 685–694.
- 12. Silveri MC, Corda F, Di Nardo M (2007) Central and peripheral aspects of writing disorders in Alzheimer's disease. J Clin Exp Neuropsychol 29: 179–186.
- 13.
Wiener N (1949) Extrapolation, Interpolation, and Smoothing of Stationary Time Series. New York: Wiley.
- 14.
Haykin S (2002) Adaptive Filter Theory 4th ed. Prentice Hall. Upper Saddle River, NJ.
- 15.
Basmajian JV, de Luca CJ (1985) Muscles Alive – The Functions Revealed by Electromyography. The Williams & Wilkins Company; Baltimore.
- 16. Fisher RA (1936) The use of multiple measurements in taxonomic problems. Annals of Eugenics 7: 179–188.
- 17. Luzzatti C, Laiacona M, Agazzi D (2003) Multiple patterns of writing disorders in dementia of the Alzheimer type and their evolution. Neuropsychologia 41: 759–772.
- 18. Tucha O, Mecklinger L, Thome J, Reiter A, Alders GL, et al. (2006) Kinematic analysis of dopaminergic effects on skilled handwriting movements in Parkinson's disease. J Neural Transm 113: 609–623.
- 19. Elble RJ, Moody C, Higgins C (1990) Primary writing tremor. A form of focal dystonia? Mov Disord 5: 118–126.
- 20. Racine MB, Majnemer A, Shevell M, Snider L (2008) Handwriting performance in children with attention deficit hyperactivity disorder (ADHD). J Child Neurol 23: 399–406.