https://orcid.org/0000-0001-8051-3751
Figures
Abstract
Upper limb robotic (myoelectric) prostheses are technologically advanced, but challenging to use. In response, substantial research is being done to develop person-specific prosthesis controllers that can predict a user’s intended movements. Most studies that test and compare new controllers rely on simple assessment measures such as task scores (e.g., number of objects moved across a barrier) or duration-based measures (e.g., overall task completion time). These assessment measures, however, fail to capture valuable details about: the quality of device arm movements; whether these movements match users’ intentions; the timing of specific wrist and hand control functions; and users’ opinions regarding overall device reliability and controller training requirements. In this work, we present a comprehensive and novel suite of myoelectric prosthesis control evaluation metrics that better facilitates analysis of device movement details—spanning measures of task performance, control characteristics, and user experience. As a case example of their use and research viability, we applied these metrics in real-time control experimentation. Here, eight participants without upper limb impairment compared device control offered by a deep learning-based controller (recurrent convolutional neural network-based classification with transfer learning, or RCNN-TL) to that of a commonly used controller (linear discriminant analysis, or LDA). The participants wore a simulated prosthesis and performed complex functional tasks across multiple limb positions. Analysis resulting from our suite of metrics identified 16 instances of a user-facing problem known as the “limb position effect”. We determined that RCNN-TL performed the same as or significantly better than LDA in four such problem instances. We also confirmed that transfer learning can minimize user training burden. Overall, this study contributes a multifaceted new suite of control evaluation metrics, along with a guide to their application, for use in research and testing of myoelectric controllers today, and potentially for use in broader rehabilitation technologies of the future.
Citation: Williams HE, Shehata AW, Cheng KY, Hebert JS, Pilarski PM (2024) A multifaceted suite of metrics for comparative myoelectric prosthesis controller research. PLoS ONE 19(5): e0291279. https://doi.org/10.1371/journal.pone.0291279
Editor: Andrea Tigrini, Polytechnic University of Marche: Universita Politecnica delle Marche, ITALY
Received: August 24, 2023; Accepted: February 15, 2024; Published: May 13, 2024
Copyright: © 2024 Williams et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data that support the findings of this study are available on Dryad at http://doi.org/10.5061/dryad.18931zd31.
Funding: This work was supported supported by the Sensory Motor Adaptive Rehabilitation Technology (SMART) Network at the University of Alberta, the Alberta Machine Intelligence Institute (Amii), and the Canada CIFAR AI Chairs program. HW was supported by the Natural Science and Engineering Research Council (NSERC) CGS-D, the Killam Trusts, and P.E.O. International. JH was supported by NSERC DG RGPIN-2019-05961, and PP was supported by NSERC DG RGPIN-2015-03646. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Below elbow (transradial) is the most prevalent of major upper limb amputations [1]. A myoelectric prosthesis offers a means of restoring complex limb function to those with transradial amputation, ideally across a wide range of arm positions [2]. Conventional myoelectric device control is based on electromyography (EMG) [3]. Here, signals are typically detected by surface electrodes that are housed within a donned prosthesis socket and then transmitted to the device’s onboard controller. The controller decodes user-specific muscle contractions and sends corresponding instructions to appropriate prosthesis wrist and hand motors.
Myoelectric prostheses that employ pattern recognition offer predictive device control that is capable of learning a user’s intended movements [4, 5]. Despite the potential of such machine learning-based control solutions, device performance challenges persist for users, particularly when various limb positions are necessary [6]. In these instances, EMG signals change due to gravity, supplemental muscle activities, and electrode shifts resulting from changes in muscle topology [7]. Resulting control can be unpredictable and therefore frustrating for users [6]. This control challenge is well documented and referred to as the “limb position effect” [7]. Several pattern recognition-based control methods have been investigated to minimize the limb position effect [8–22]. These methods require a user to perform a training routine across multiple limb positions, prior to daily device use. A training routine involves execution of a specific sequence of forearm muscle contractions. EMG signals resulting from the muscle contractions are captured for use by the device controller’s model. The model learns to recognize various patterns of consistent and repeated muscle signal features [4], including patterns involved in wrist rotation and hand open/close. Learned features are subsequently classified during device use, with classifications informing motor instructions.
Inertial measurement unit (IMU) data can provide a classification control model with additional and informative limb position-related data [7, 23]. Deep learning control methods, such as recurrent convolutional neural networks (RCNNs), can combine high volumes of EMG and IMU data from multiple limb positions. However, to capture all required muscle and limb position data (in low to high arm positions), lengthy and burdensome training routines must be performed by users [2, 24, 25]. Control model retraining is also required in instances when device control degrades, such as due to muscle fatigue or electrode shifts. The overall training burden poses drawbacks to position-aware myoelectric control methods.
Our earlier study uncovered that the addition of transfer learning (TL) can alleviate the training burden necessitated by data intensive RCNN-based solutions [26]. In this previous work, an RCNN classification control model (classifier) was trained using a large dataset of EMG and IMU signals obtained from numerous individuals with intact upper limbs, to become the starting point of new users’ control. Each new user required a reduced amount of personal data for training thereafter. To test control, participants wearing a simulated prosthesis [27, 28] performed two functional tasks: the Pasta Box Task [29] and the Refined Clothespin Relocation Test [30]. Control assessments were based on metrics established in the literature [27–29, 31, 32]. This research showed that RCNN-based classification control with TL reduces training burden and offers a control solution with a tendency towards better functional task performance across multiple limb positions (versus a linear discriminant analysis, or LDA, classification controller). Interestingly, the research also identified possible instances of the limb position effect during high grasping movements, however it was noted that more detailed measures of control were needed to confirm this [26]. As a corollary to the TL-based findings resulting from this work, metrics deficiencies were uncovered—control characteristics outcomes evidenced during task performance could not be fully understood, and user-reported control experiences were not considered [26].
Without question, our earlier work identified important omissions in the collective of metrics commonly used for prosthesis control appraisal, even amongst those of established task-based assessments meant to mimic activities of daily living. Such functional assessments include the Box and Blocks Test [33], Jebsen-Taylor Hand Function Test [34], Activities Measure for Upper Limb Amputees [35], Southampton Hand Assessment Procedure [36], and Assessment of Capacity for Myoelectric Control [37, 38]. These assessments typically require users to interact with a variety of objects (e.g., grasp, move, rotate, release of objects), but the scores used to summarize arm function are limited—either based on task completion durations [34, 36], number of objects moved [33], or a trained rater’s assessment [35, 37, 38]. Such scores cannot yield a complete understanding of the quality of participants’ hand, wrist, and arm movements [29]. Furthermore, they cannot adequately characterize the nature of device control, such as the identification of unnecessary grip aperture adjustments [26]. Even a recently introduced take-home assessment method left researchers unable to distinguish how available grips were used and how the effects of usage conditions affected control [39]. For all such reasons, we determined that RCNN-based classification control with TL (RCNN-TL) not be judged in future work by task performance alone, but rather, that control characteristics also be considered. Then collectively, the task performance and control characteristics can be weighed against subjective user experience, to provide a full complement of data-driven prosthesis control outcomes.
This current work re-examines RCNN-TL control through the lens of comprehensive assessment metrics. As a primary contribution, it introduces a novel suite of metrics that aims to address the issue of inconclusive myoelectric controller assessment outcomes. The suite includes three broad categories of metrics: task performance, control characteristics, and user experience. As a secondary contribution, this work showcases the use and thoroughness of the suite by deploying these metrics to reinvestigate our earlier controller research findings [26]—examining whether RCNN-TL can indeed reduce training burden, offer improved device control over a comparative LDA baseline classification controller (LDA-Baseline), and if instances of the limb position effect can be pinpointed. In using the suite of metrics, this work provides a data-driven understanding of when and why TL-based neural network control solutions show great promise towards solving the limb position effect challenge. It is expected that the suite introduced by this research will guide future rehabilitation device control experimentation.
What follows is a presentation of our collective of metrics, an overview of our reinvestigation research, and a detailed presentation of its methods, results, discussion, and conclusion.
Introducing the suite of myoelectric control evaluation metrics
Performance-based assessments for the evaluation of real-time upper limb prosthesis control often require participants to either move a virtual arm to a target posture or a cursor to a target position, by performing appropriate muscle contractions in their residual limb [8, 14, 40–50]. For example, a virtual arm is presented on a computer screen in the Target Achievement Control test [51], and a cursor is presented on-screen in Fitts’ Law tests [52]. EMG sensors placed on participants’ limbs record the corresponding muscle signals in such assessments. However, limb kinematics and factors that change EMG signals (including the limb position effect) are not taken into consideration [43]. Consequently, research that employs on-screen assessments often recommend that future work be undertaken using alternative real-time methods [8, 42].
Other studies have taken a next step towards a deeper understanding of control, through the introduction of functional task assessments [50, 53–63]. Here, either non-disabled participants wearing a simulated prosthesis or actual myoelectric prosthesis users, are required to perform upper limb tasks that mimic activities of daily living. One common clinical control assessment technique, known as the Assessment of Capacity for Myoelectric Control, has a trained rater assess control during functional task execution [37, 38]. However, a trained rater may not always be available. Alternative non-rater-based assessment approaches record the movement of participants’ upper limbs during task execution, using motion capture technology. From the resulting data, hand movement metrics, including hand velocity, hand distance travelled, and hand trajectory variability can be calculated (rather than rated) [29, 32, 64, 65]. As we had motion capture technology available to us, we used these three hand movement metrics, plus common task success rate and duration-based measures—collectively presenting them as Task Performance Metrics. All such metrics can be found in Table 1, with calculations derived from the work of Valevicius et al. [29]. Note that these metrics have already been validated [32] and have been employed to examine both myoelectric and body-powered prosthesis use [31, 66]. Furthermore, they have been compared to common clinical assessments with prosthesis users [27, 31]. S1 Table offers select examples of strong and weak outcomes resulting from these metrics.
Task performance metrics alone, however, do not capture the nuances of device control characteristics, such as instances where a user introduces unnecessary hand/wrist movements when grasping or releasing an object [26]. Some studies have introduced metrics that quantify specific myoelectric prosthesis control characteristics, including misclassification rates [67], grasp force [67, 68], grip aperture plateau time [69], wrist rotation range of motion [67], workload (assessed via pupil size) [70], and measures of muscle activations [71]. We selected and derived metrics from the literature, plus developed additional novel metrics, and collectively present them as Control Characteristic Metrics. All such metrics can be found in Table 2, with select examples in S1 Table.
Finally, whether any proposed control solution yields noticeable improvement depends on users’ assessments. The National Aeronautics and Space Administration Task Load Index (NASA-TLX) is a survey tool, which measures subjective mental workload [72]. A recent literature review confirmed that the NASA-TLX has been widely employed in prosthesis use assessments [73]. Usability surveys offer yet another assessment approach and capture the users’ opinions of alternative device control solutions [74]. We selected relevant survey questions from the literature and present them as User Experience Metrics. All such metrics can be found in Table 3.
Tables 1–3, collectively describe the Suite of Myoelectric Control Evaluation Metrics introduced in this work. The Control Improvement Indicator column in each of these tables uses unimpaired limb movement as a yardstick for control assessment. To conduct such assessments, data collection protocols should include the following:
- Control models under investigation should be trained by participants using a research-specific series of hand/wrist movements that elicit forearm muscle signals for capture
- Participants must either wear a myoelectric prosthesis or a simulated prosthesis
- Participants must perform functional task(s) that can be split into the distinct phases of Reaches, Grasps, Transports, and Releases. All such task(s) should be standardized—using specific object sizes and locations for grasp and release actions. Examples of suitable functional tasks include, but are not limited to, the Pasta Box Task [29], Cup Transfer Task [29], Refined Clothespin Relocation Test [30], and the modified Box and Blocks test [75, 76].
- Motion capture data, muscle signals, and device motor data should be collected during functional task execution
- User experience survey responses should be collected at the end of each testing session
- Data streams of interest (for each functional task trial) could include: the number of error-free trials executed; trial time stamps; the 3D position of the device hand, its grip aperture and wrist rotation angles; plus the participants’ shoulder flexion/extension angles, EMG signal data, and post-testing session survey scores.
- Given that standardized tasks are to be used, results calculated with the suite of metrics should only be compared within each task, not across tasks
With adherence to the above-mentioned data collection requirements, the suite of control evaluation metrics presented in this work facilitates in-depth analysis that will uncover numerous upper limb prosthesis control insights. These insights are expected to be particularly beneficial in investigations that compare myoelectric device controllers.
Overview: Reinvestigation using our suite of metrics
As an example of how the suite of evaluation metrics introduced in this work can be used to advance prosthesis control research, we deployed them in a deliberately challenging experiment—reinvestigating our earlier comparative classifier research findings [26]. Here, device control offered by two classifiers was compared: a proposed deep learning-based controller (RCNN-TL) intent on mitigating the limb position effect, versus a commonly used and commercially available controller (LDA-Baseline). Given that LDA-based control is commonly used in myoelectric prostheses, it has been adopted in research as a baseline for comparison to other controllers [23, 77, 78]. Fig 1 presents an overview of how each control model was trained and tested, using two distinct groups of participants without upper limb impairment who wore an EMG and IMU data capture armband.
The blue panel (A) illustrates the step that the General Participant Group performed (training routine that yielded RCNN-TL’s pre-trained model) while wearing an EMG and IMU armband. The yellow panels illustrate the steps that the Simulated Prosthesis (SP) Participant Group performed: (B) respective training routines that yielded RCNN-TL’s retrained model and LDA-Baseline’s trained model, and (C) subsequent controller testing using functional tasks, all while wearing an EMG and IMU armband plus a simulated prosthesis.
(1) A large General Participant Group’s data created a control starting point for new users.
RCNN-TL’s Model Pre-Training—Each General Participant Group member performed a training routine (isotonic forearm muscle contractions were executed in four limb positions), during which their forearm EMG and IMU signals were collected. Their collective resulting signal data, along with the corresponding classes of muscle contractions, informed RCNN-TL’s pre-trained control model.
(2) A new, smaller Simulated Prosthesis (SP) Participant Group wore a simulated prosthesis.
RCNN-TL’s Model Retraining & Testing—Each SP Participant Group member performed a brief training routine (isometric contractions were held in three limb positions). The resulting participant-specific EMG and IMU data, plus classes of muscle contractions, were used to calibrate RCNN-TL’s model. Participants tested RCNN-TL by performing functional tasks across multiple limb positions—the Pasta Box Task [29] and the Refined Clothespin Relocation Test (RCRT) [30].
LDA-Baseline’s Model Training & Testing—The forearm muscle signals of each SP Participant Group member were also captured using a standard pattern recognition training routine that was not designed to mitigate the limb position effect (isometric contractions were held in one limb position) [23]. The resulting EMG data, plus classes of muscle contractions, were used to train LDA-Baseline’s model. Each SP Participant Group member tested LDA-Baseline by performing the Pasta Box Task [29] and RCRT [30].
Methods
What follows are details about our reinvestigation research methods, including: participant descriptors; muscle signal data collection and processing techniques; a description of the simulated prosthesis donned by participants; specifications of the control models under investigation and their training requirements; setup of the testing environment; the functional tasks used to assess control; the participant survey administration process; control data processing techniques to yield the suite of metrics; the statistical analysis of such metrics; and the identification of instances of the limb position effect resulting from this analysis.
Participants
Participants recruitment took place from March 2, 2022, to March 31, 2022. All participants provided written informed consent, as approved by the University of Alberta Health Research Ethics Board (Pro00086557).
General participant group (without simulated prosthesis).
Nineteen participants without upper limb impairment were recruited. All had normal or corrected vision, 10 were male, nine were female, 17 were right-handed. They had a median age of 25 years (range: 19–58 years) and median height of 170 cm (range: 159–193 cm). Each of the 19 participants completed one data collection session.
SP participant group (with donned simulated prosthesis).
A total of nine new participants without upper limb impairment were recruited. One participant was removed due to their inability to reliably control the donned simulated prosthesis even after control practice. Of the remaining eight participants, all had normal or corrected vision, five were male, three were female, seven were right-handed. They had a median age of 22 years (range: 20–56 years) and median height of 181 cm (range: 169–185 cm). No participants had experience with EMG pattern recognition control using a simulated prosthesis. The eight participants completed two data collection sessions on different days, with a median of 24 days between sessions (range: 18–45 days). Half of the participants retrained/tested RCNN-TL in their first session (as shown in Fig 1B and 1C), and the other half trained/tested LDA-Baseline in their first session (also shown in Fig 1B and 1C). Each trained/tested the other controller in their second session.
Signal data collection & processing procedure
Participants in both groups wore a Myo gesture control armband (Thalmic Labs, Kitchener, Canada) over their largest forearm muscle bulk [79]. That is, at approximately the upper third of their forearm, as shown in Fig 2A (with the top of the armband at a median of 28% of the way down the forearm from the medial epicondyle to the ulnar styloid process). The Myo armband contained eight surface electrodes to collect EMG data at a sampling rate of 200 Hz. The Myo armband also contained one IMU to collect limb position data (three accelerometer, three gyroscope, and four quaternion data streams) at 50 Hz. Myo Connect software was used to stream and record EMG and IMU data in Matlab.
A) Myo armband on a participant’s forearm and B) simulated prosthesis on a participant’s forearm, with labels indicating the sleeve, two pieces of liner, hand brace, distal ring, cushions, wrist motor, and hand motor. Adapted from Williams et al. [26].
The EMG data from the Myo armband were filtered using a high pass filter with a cutoff frequency of 20 Hz (to remove movement artifacts), as well as a notch filter at 60 Hz (to remove electrical noise). The accelerometer data streams were upsampled to 200 Hz (using previous neighbour interpolation) to align them with the corresponding EMG data. Data were then segmented into windows (160-millisecond with a 40-millisecond offset).
Simulated prosthesis & donning procedure
The simulated prosthesis used in this study was the 3D-printed Modular-Adaptable Prosthetic Platform (MAPP) [80] (shown in Fig 2B). It was fitted to each SP Participant Group member’s right arm for simulation of transradial prosthesis use. The MAPP’s previously-published design [80] was altered to improve wearer comfort in our study—the distal ring was made to resemble the oval shape of a wrist and the hand brace was elongated so that the distal ring would sit more proximally on the wearer’s wrist. A nonproprietary 3D-printed robotic hand [81] was affixed to the MAPP beneath the participant’s hand. Wrist rotation capabilities were also added to the device. Hand and wrist movements (that is, with two degrees of freedom) were powered by two Dynamixel MX Series motors (Robotis Inc., Seoul, South Korea).
After placement of the Myo armband, each SP Participant Group member donned a thin, protective sleeve and then the simulated prosthesis. To increase participant comfort, pieces of thermoplastic elastomer liner were placed inside the distal ring and just above the participant’s elbow, and 3D-printed cushions, made of Ninjaflex Cheetah filament (Ninjatek, Inc.), were placed throughout the device socket (shown in Fig 2B). The secureness of the device and the participants’ comfort were checked before proceeding with controller training.
Control model descriptions & training routines
RCNN-TL’s model.
Bayesian optimization automatically determined the number of convolution layers, number of filters, filter size, pooling size, and patience required for the classifier used in this controller. Optimization was performed in two steps: first, the number of layers along with each hyperparameter being optimized were determined using a broad range of values; thereafter, values were refined using a narrower range (centered at earlier optimized values). RCNN-TL’s model architecture consisted of 19 layers, as illustrated in Fig 3. In this model, a sequence input layer first received and normalized the training data. Then, a sequence folding layer was used, allowing convolution operations to be performed independently on each window of EMG and accelerometer data. This was followed by a block of four layers: a 2D convolution, a batch normalization, a rectified linear unit (ReLU), and an average pooling layer. This block of layers was repeated once more. Each of the two average pooling layers had a pooling size of 1x4. A block of three layers followed: a 2D convolution, a batch normalization, and a ReLU layer. The optimal number of filters in the convolution layers were determined to be 4, 16, and 32, respectively, and each had a filter window size of 1x3. The next layers included a sequence unfolding layer (to restore the sequence structure), a flatten layer, a long short-term memory (LSTM) layer, and a fully connected layer. Finally, a softmax layer and classification layer were used. To prevent overfitting, a patience parameter was set to trigger early stopping when the validation loss increased five times (similar to methods used in other works, including Côté-Allard et al. [82]).
Adapted from Williams et al. [26].
RCNN-TL’s model pre-training routine.
General Participant Group members followed onscreen instructions, performing muscle contractions in 5 wrist positions (rest, flexion, extension, pronation, and supination; shown in Fig 1A), for 5 seconds each. The muscle contractions were performed twice in 4 limb positions: arm at side, elbow bent at 90°, arm out in front at 90°, and arm up 45° from vertical (shown in Fig 1A). This position-aware routine was similar to those used in other real-time control studies aiming to mitigate the limb position effect [8, 23, 79]) and took approximately 200 seconds. The resulting EMG and accelerometer data, plus corresponding classes of muscle contractions, were used to pre-train RCNN-TL’s model.
RCNN-TL’s model retraining routine.
Our previous offline research [66] examined methods of reducing user training burden and uncovered a shortened/optimized routine that still yielded high predictive accuracy. In keeping with this, the SP Participant Group members followed onscreen instructions, performing muscle contractions in the same 5 wrist positions (shown in Fig 1B), for only 2 (rather than 5) seconds each. The muscle contractions were performed twice in only 3 (not 4) limb positions: arm at side, elbow bent at 90°, and arm up 45° from vertical (shown in Fig 1B). The resulting EMG and accelerometer data, plus corresponding classes of muscle contractions, were used to retrain RCNN-TL’s model.
LDA-baseline’s model.
Four commonly used EMG features were chosen for implementation of this controller’s classifier: mean absolute value, waveform length, Willison amplitude, and zero crossings [83]. These features were calculated for each channel within each window of EMG data. A pseudo-linear LDA discriminant type was used, given that columns of zeros were occasionally present in some classes for some features (including Willison amplitude and zero crossings).
LDA-baseline’s model training routine.
SP Participant Group members followed onscreen instructions, performing muscle contractions in the same 5 wrist positions (shown in Fig 1B), for 5 seconds each. The muscle contractions were performed twice, with the participants’ elbow bent at 90° (shown in Fig 1B). This single-position routine mimicked standard myoelectric prosthesis training [3] and took approximately 50 seconds. The resulting EMG data and corresponding classes of muscle contractions were used to train LDA-Baseline’s model.
RCNN-TL & LDA-baseline implementation.
Each model was trained using Matlab software running on a computer with an Intel Core i9–10900K CPU (3.70 GHz) with 128 GB of RAM. RCNN-TL’s and LDA-Baseline’s models were retrained/trained in median times of 3.41 and 0.39 seconds, respectively. For both controllers, Matlab code was written to receive signal data and subsequently classify wrist movements. Code was also written to send motor instructions, based on the resulting classifications, to brachI/Oplexus software [84] (flexion controls hand close, extension controls hand open, pronation controls wrist counter-clockwise rotation, and supination controls wrist clockwise rotation). brachI/Oplexus relayed the corresponding control signals to the simulated prosthesis’ motors. The positions of the motors were recorded with a sampling rate of 50 Hz.
Simulated device control practice & testing eligibility
During each testing session, SP Participant Group members took part in a control practice period (approximately 40 minutes), during which they were taught how to operate the simulated prosthesis using isometric muscle contractions, under three conditions:
- Controlling the hand open/close while the wrist rotation function was disabled. They practiced grasping, transporting, and releasing objects at varying heights.
- Controlling wrist rotation while the hand open/close function was disabled. They practiced rotating objects at varying heights.
- Controlling the hand open/close function in concert with the wrist rotation function. They practiced tasks that involved grasping, transporting, rotating, and releasing objects at varying heights.
Following their practice period, participants were tested to determine whether they could reliably control the simulated prosthesis. Two cups were situated in front of them at two different heights, with a ball in one of the cups. Participants were asked to pour the ball between the two cups, and instances when they dropped the ball or a cup were recorded. If participants could not complete at least 10 pours with a success rate of at least 75% within 10 minutes, the session was ended, and they were removed from the study. Recall that one participant was removed (as stated in the Participants section), given that they could not complete this activity with LDA-Baseline in their first session.
Motion capture setup & kinematic calibrations
For participants who were deemed eligible for controller testing, the following motion capture steps were undertaken:
Step 1: Motion capture setup.
An 8-camera OptiTrack Flex 13 motion capture system (Natural Point, OR, USA) was used to capture participant movements and task objects at a sampling rate of 120 Hz. Eight individual markers were placed on the simulated prosthesis hand, circled in Fig 4 (one on the thumb, one on the index finger, and the remaining six throughout the back and side of the hand to ensure reliable rigid body tracking). Rigid marker plates were also placed on each participant’s right forearm (affixed to the simulated prosthesis socket), upper arm, and thorax, in accordance with Boser et al.’s cluster-based marker model [85].
The eight motion capture markers that remained attached to the hand are circled, and the three additional individual markers for the ski pose calibration are labelled.
Step 2: Kinematic calibrations
Each participant was required to perform two kinematics calibrations. As per Boser et al., the first calibration called for participants to hold an anatomical pose [85], for capture of the relative positions of the hand markers and motion capture marker plates when wrist rotation and shoulder flexion/extension angles were at 0°. The second calibration required participants to hold a ski pose [85], for the purpose of refining wrist rotation angles. Here, three additional individual markers were affixed to the simulated prosthesis, as shown in Fig 4:
- One marker placed on the top of the prosthesis’ hand motor, with the device hand closed
- One marker placed on the bottom of the prosthesis’ wrist motor, forming a line with the first marker (to represent the axis about which the wrist rotation occurred)
- One marker placed on the side of the prosthesis’ wrist motor (to create a second axis, perpendicular to the axis of wrist rotation)
Upon completion of the two kinematics calibrations, all Step 2 markers were removed. What remained were only those markers affixed during Step 1 for data collection purposes.
Functional tasks & data collection
Motion capture data were collected during the execution of the following functional tasks: Pasta Box Task (Pasta)—Participants were required to perform three distinct movements, where they transported a pasta box between a 1st, 2nd, and 3rd location (a side table and two shelves at varying heights on a cart, including across their midline) [29]. The task setup is shown in Fig 5A. Motion capture markers were placed on all task objects, as per Valevicius et al. Participants performed a total of 10 Pasta trials. If participants dropped the pasta box, placed it incorrectly, performed an incorrect movement sequence, or hit the frame of the task cart, the trial was not analyzed. Pasta was the first of two functional tasks performed as it was considered easier.
In panel (A), the 1st, 2nd, and 3rd pasta box locations are labelled. The pasta box movement sequence is 1st —>2nd —>3rd —>1st locations. In panel (B), the 1st, 2nd, and 3rd clothespin locations on the horizontal and vertical bars are labelled. The clothespin movement sequences in RCRT Up are horizontal 1st —>vertical 1st, horizontal 2nd —>vertical 2nd, and horizontal 3rd —>vertical 3rd locations. The clothespin movement sequence in RCRT Down follows the same order, but with each clothespin moved from vertical to horizontal locations.
RCRT.
Participants were required to perform three distinct movements using clothespins. They moved three clothespins between 1st, 2nd, and 3rd locations on horizontal and vertical bars [30]. To simplify trial execution, RCRT was split into RCRT Up and RCRT Down trials. The task setup for these trials is shown in Fig 5B. During Up trials, participants moved the three clothespins from the horizontal bar to the vertical bar, and during Down trials, they moved the clothespins from the vertical bar to the horizontal bar. A height adjustable cart was set such that the top of each participants’ shoulder was aligned with the midpoint between the top two targets on the vertical bar. Motion capture markers were placed on all task objects, as per our earlier research [26]. Participants performed a total of 10 Up trials and 10 Down trials. If participants dropped a clothespin, placed it incorrectly, or performed an incorrect movement sequence, the trial was not analyzed. Performance of RCRT Up and Down trials were alternated, and started with RCRT Up.
Survey administration
At the end of each session, each participant completed two surveys: the NASA-TLX [72] and a usability survey [74]. The former was administered using the official NASA-TLX iPad application, where participants scored their device control workload demand on a continuous rating scale with endpoint anchors of low and high. The usability survey was administered on paper, where participants marked their usability scores on a continuous rating scale with endpoint anchors of 0 and 5. In their second session, participants were not reminded of their survey responses from their first session.
Data processing & calculation procedures
Motion capture data cleaning & calculations.
Motion capture marker trajectory data were cleaned and filtered. As per Valevicius et al. [29], grip aperture was calculated as the distance between the motion capture markers on the simulated prosthesis’ index and thumb, and a 3D object representing the simulated prosthesis’ hand was generated using the remaining 6 hand motion capture markers. Then, through calculations modified from Boser et al. [85], wrist rotation was calculated using the forearm and hand motion capture markers, and shoulder flexion/extension was calculated using the upper arm and thorax motion capture markers.
Data segmentation.
The task data were segmented in accordance with Valevicius et al. [29], as follows:
- For each task, the data from each trial were first divided into distinct movements 1, 2, and 3 based on hand velocity and the velocity of the pasta box/clothespins during transport.
Pasta Movements 1, 2, and 3 differentiated between: (1) reaching for the pasta box at its 1st location, grasping it, transporting it to its 2nd location, releasing it, and moving their hand back to a home position; (2) reaching for the pasta box at the 2nd location, grasping it, transporting it to its 3rd location, releasing it, and moving their hand back to the home position; and (3) reaching for the pasta box at the 3rd location, grasping it, transporting it back to the 1st location, releasing it, and moving their hand back to the home position.
RCRT Up Movements 1, 2, and 3 differentiated between: (1) reaching for the 1st clothespin at its 1st horizontal location, grasping it, transporting it to its 1st vertical location, releasing it, and moving their hand back to a home position; (2) reaching for the 2nd clothespin at its 2nd horizontal location, grasping it, transporting it to its 2nd vertical location, releasing it, and moving their hand back to the home position; and (3) reaching for the 3rd clothespin at its 3rd horizontal location, grasping it, transporting it to its 3rd vertical location, releasing it, and moving their hand back to the home position.
RCRT Down Movements 1, 2, and 3 differentiated between: (1) reaching for the 1st clothespin at its 1st vertical location, grasping it, transporting it to its 1st horizontal location, releasing it, and moving their hand back to a home position; (2) reaching for the 2nd clothespin at its 2nd vertical location, grasping it, transporting it to its 2nd horizontal location, releasing it, and moving their hand back to the home position; and (3) reaching for the 3rd clothespin at its 3rd vertical location, grasping it, transporting it to its 3rd horizontal location, releasing it, and moving their hand back to the home position. - Then, the data from each of the three movements were further segmented into five phases of (1) Reach, (2) Grasp, (3) Transport, (4) Release, and (5) Home (note that the Home phase was not used for data analysis)
- Finally, two movement segments of (1) Reach-Grasp and (2) Transport-Release were defined for select metrics analysis
- Six final levels for data analysis resulted: controller (either RCNN-TL or LDA-Baseline), task (either Pasta, RCRT Up, or RCRT Down), trial (1–10), movement (1–3), movement segment (Reach-Grasp or Transport-Release), and phase (Reach, Grasp, Transport, or Release)
Grip aperture & wrist rotation re-calculations.
The grip aperture and wrist rotation angle were re-calculated using the data from the simulated prosthesis’ two motors, given that small (yet informative) adjustments in the positions of these motors may not have been detected by motion capture cameras. The positions of these motors were first upsampled to 120 Hz using linear interpolation. Grip aperture re-calculation: motion-capture-calculated grip aperture was used to fit a trinomial curve to transform the hand motor data to grip aperture. Wrist motor angle re-calculation: motion-capture-calculated wrist rotation was used to fit a binomial curve to transform the wrist motor data to wrist rotation angles.
Final suite of metrics calculations.
The final suite of metrics was calculated using the procedures described in Tables 1–3. Note that the simultaneous wrist-shoulder movements metric was calculated only for Reach and Transport phases of RCRT Up and RCRT Down trials, because these were the only phases that required the participant to rotate the device wrist while moving their arm to a different height.
Statistical analysis
To investigate task performance difference between RCNN-TL and LDA-Baseline, the following statistical analyses were performed:
For metrics that were analyzed at the phase or movement segment level.
Participants’ results were first averaged across trials and movements. If results were normally-distributed, a two-factor repeated-measures analysis of variance (RMANOVA) was conducted using the factors of controller and phase/movement segment. When the resulting controller effects or controller-phase/movement segment interactions were deemed significant (that is, when the Greenhouse-Geisser corrected p value was less than 0.05), pairwise comparisons between the controllers were conducted. If results were not normally-distributed, the Friedman test was conducted. When the resulting p value was less than 0.05, pairwise comparisons between the controllers were conducted. Pairwise comparisons (t-test/Wilcoxon sign rank test) were deemed significant if the p value was less than 0.05.
Limb position effect identification
The limb position effect has been shown to cause control accuracy degradation and large between-participant control variation in offline research [40]. However, earlier works have not pinpointed specific instances of the effect in functional task execution data. Using the novel control characteristics metrics introduced in this work, identification of such instances is possible—larger medians and/or larger interquartile ranges (IQRs) can provide evidence of degraded control. To identify the limb position effect in this study, metrics’ medians and IQRs for Reach, Grasp, Transport, and Release phases were considered separately across movements 1, 2, and 3 of Pasta, RCRT Up and RCRT Down. An occurrence where movement variation was not due to the limb position effect is illustrated in Fig 6A, where the number of wrist rotation adjustments metric in RCRT Down Release phases have medians and IQRs that remain relatively constant at different limb positions. Conversely, an occurrence where movement variation was due to the limb position effect can be seen in Fig 6B. Here, the same metric in RCRT Down Grasp phases shows its medians and IQRs both increasing as the limb position changed.
in each (A) RCRT Down Release and (B) RCRT Down Grasp of each task movement (Mvmt). Medians are indicated with thick lines, and interquartile ranges are indicated with boxes.
The following limb position identification process was used to examine all control characteristics metrics for Reach, Grasp, Transport, and Release phases across the three movements of Pasta, RCRT Up, and RCRT Down:
- First, the three medians were rescaled as percentages of the maximum of the three medians
- Next, the three IQRs were rescaled as percentages of the maximum of the three IQRs
- Then, the limb position effect identification rules outlined in Table 4 were developed—through iterative trial-and-error comparisons of potential rules to visual representations of metrics’ medians and IQRs (as in Fig 6A and 6B)
- The resulting rules were subsequently used to identify instances of the limb position effect. Note that for Pasta, two rule options were used, given that the limb position effect was most likely to be present in that task’s movements 2 or 3 (at the highest shelf location). For RCRT Up and RCRT Down, only one rule option was necessary, given that the limb position effect was most likely to occur in movement 3 (at the top clothespin location). For each rule, the limb position effect was identified only in instances where all conditions were met (that is, when all rules in a row of Table 4 were true).
The abovementioned rules are applicable to metrics where smaller values are indicative of control improvements, as was the case with most control characteristics metrics in this work. The exception was the simultaneous wrist-shoulder movements metric, where larger values were indicative of improved control. To adjust for this exception, each of the three movements’ rescaled medians (represented as percentages) were subtracted from 100%. In doing so, these medians were altered to represent the percent of the phase in which simultaneous movements did not occur. After this adjustment, the limb position identification process could be followed.
Results
Task performance
The significant differences across the task performance metrics are reported in Table 5. Task specific outcomes derived from the table include:
Pasta.
RCNN-TL performed significantly better than LDA-Baseline in one metric: Release phase duration.
Control characteristics
The significant differences across control characteristics metrics are reported in Table 6. Task specific outcomes derived from the table include:
Pasta.
RCNN-TL performed significantly better than LDA-Baseline in seven metrics. Examples of one such metric are illustrated in S1 Table (for Reach-Grasp grip aperture plateau). The limb position effect was identified in 12 metrics under LDA-Baseline control, and in one metric (Grasp total muscle activity) under RCNN-TL control. Four of the seven significant RCNN-TL versus LDA-Baseline differences were in metrics that showed evidence of the limb position effect.
RCRT Up.
No significant differences were identified, and no metrics showed evidence of the limb position effect.
RCRT Down.
RCNN-TL performed significantly better than LDA-Baseline in two metrics. Examples of one such metric are illustrated in S1 Table (for Reach simultaneous wrist-shoulder movements). The limb position effect was identified in four other metrics, with one such instance illustrated in Fig 6B (Grasp number of wrist rotation adjustments).
Summary.
9 of the 81 Control Characteristics metrics showed significant differences, all of which demonstrated that RCNN-TL performed better than LDA-Baseline. Furthermore, 16 metrics showed evidence of the limb position effect. All such metrics were only identified in Pasta and RCRT Down, suggesting that these outcomes are likely influenced by the position-aware nature of RCNN-TL control.
User experience
User experience metrics were calculated at the controller level, rather than for each task (detailed in Table 3). There were no significant differences between RCNN-TL and LDA-Baseline. Box plots illustrating the median controller-level scores across participants can be found in Fig 7.
Medians are indicated with thick lines, interquartile ranges are indicated with boxes, and outliers are indicated with circles.
Of note, RCNN-TL scored better than LDA-Baseline in the NASA-TLX’s Mental Demand dimension and in the usability survey’s Intuitiveness dimension. These results suggest that RCNN-TL offered more intuitive control. The two controllers had equal median scores in NASA-TLX’s Effort dimension. LDA-Baseline scored better than RCNN-TL in all other dimensions.
Discussion
The suite of myoelectric prosthesis control evaluation metrics introduced in this work (detailed in Tables 1, 2 and 3) yielded informative limb position effect-related outcomes that could only be speculated upon in our earlier work [26]. What follows is a discussion about the metrics-driven findings from this current work, with a focus on understanding when and why limb position variations caused control challenges during participants’ execution of the Pasta Box Task (Pasta) and the Refined Clothespin Relocation Test (RCRT Up and RCRT Down).
Findings from control characteristics metrics
Limb position effect identification.
To our knowledge, no other study has identified occurrences of the limb position effect using functional task assessment outcomes. In this work, we used functional tasks to assess device control and found that analysis of our control characteristics metrics did uncover instances of the limb position effect. In LDA-Baseline results, instances were identified in Pasta and RCRT Down (16 of 81 cells with dark borders in Table 6), and never in RCRT Up. Note that for RCNN-TL, one instance of the limb position effect was uncovered—Grasp total muscle activity in Pasta. However, this instance may simply be due to inevitable positional EMG signal variations, rather than due to control degradation. Based solely on LDA-Baseline results, we surmised the following:
- Raised arm positions in the sagittal plane caused grasp challenges for participants, as evidenced by the identification of the limb position effect only during the Grasp phases of RCRT Down (in four metrics).
- Raised arm positions in the sagittal plane did not cause release challenges for participants, as evidenced by the absence of limb position effect identification in RCRT Up. Logically, as hand opening during object release phases is controlled by wrist extension muscle activation, classification of wrist extension was not affected by the limb position effect.
- Arm movements along the frontal plane caused further control challenges for participants. Not only did Pasta require participants to perform arm raises in the sagittal plane, large cross-body and away-from-body movements had to be introduced to accomplish this task. The limb position effect was detected in three of four Pasta phases (four times in Reach, once in Reach-Grasp, three times in Grasp, and four times in Release)
So overall, LDA-Baseline control often appeared to be impeded by shoulder position fluctuations in the frontal plane. Furthermore, arm raises limited to the sagittal plane caused only grasp control deterioration for this controller. Both such circumstances identify catalysts for limb position effect control challenges.
Evidence of limb position effect mitigation.
Recall that Table 6 also identified significant differences in control characteristics metrics, where green cells indicated instances where RCNN-TL performed significantly better than LDA-Baseline. Nine of the 81 metrics (cells) were significant (shaded in green), indicating that RCNN-TL always performed the same as, or significantly better than, LDA-Baseline for these metrics. Furthermore, all such significant differences presented in Table 6 occurred in Pasta and RCRT Down, and never in RCRT Up. This coincides with those tasks where instances of the limb position effect were identified, suggesting that RCNN-TL successfully mitigated such occurrences.
Interestingly, RCNN-TL performed significantly better than LDA-Baseline in several Pasta metrics, even though Pasta involves numerous limb positions that were not included in RCNN-TL’s pre-training/retraining routines. We speculate that the pre-training data from 19 individuals provided sufficient variety to result in a controller that is robust to limb positions not included in its training routines.
Significant control characteristics differences between RCNN-TL and LDA-Baseline were not identified for RCRT Up. Recall that the problem was not identified in RCRT Up, despite this task’s requirement for varied limb positions. So, if the limb position effect did not cause control degradation for either controller, then perhaps: (a) LDA-Baseline simply performed well during this task and control improvements were not necessary, or (b) RCNN-TL control should be improved in instances when the limb position effect is not evident.
Merits of control characteristics metrics.
Significant differences between RCNN-TL and LDA-Baseline were identified in at least one phase/movement segment for all control characteristics metrics analyzed, with the exceptions of total grip aperture movement and total muscle activity. Still, these two metric exceptions might yield outcomes beneficial to other controller comparisons and should not be discounted from the metrics introduced in this work. Total grip aperture movement, for instance, might help to identify grasping efficiency during task execution, and total muscle activity might help to identify muscle exertion required for task completion. Future controller comparisons are expected to determine whether these metrics are sensitive to controller variations.
Findings from task performance metrics
Table 5 identified 2 of 48 metrics that showed RCNN-TL performing significantly better than LDA-Baseline (for Release phase duration in Pasta and Grasp relative phase duration in RCRT Down), and 2 of 48 metrics that showed the contrary (for success rate in RCRT Up and Release relative phase duration in RCRT Down). These outcomes coincided with those of our earlier work [26], however, the control characteristics metrics introduced in this study facilitated a deeper understanding of why task performance deteriorated at times—specifically, when instances of the limb position effect hampered control. The following task performance insights were uncovered in this work:
- RCNN-TL successfully mitigated the limb position effect, as evidenced by the two specific instances when its control was significantly better than that of LDA-Baseline—in Pasta Release and RCRT Down Grasp phases. Our control characteristics analysis revealed that participants struggled during these phases (identified as instances where the limb position effect occurred). Such struggles were apparent when participants used LDA-Baseline, but not so when using RCNN-TL. So, RCNN-TL likely remedied control degradation introduced by the limb position effect.
- RCNN-TL may not have performed well in instances where the limb position effect was not evidenced. Consider that two task performance metrics showed that RCNN-TL performed significantly worse than LDA-Baseline: (1) the relative duration of the RCRT Down Release phases, and (2) the RCRT Up success rate. Furthermore, consider that analysis of control characteristics revealed that device control was not affected by the limb position effect in either the RCRT Down Release phase, or any phase of RCRT Up. Together, these outcomes support the hypothesis that RCNN-TL may not have performed well in instances where the limb position effect was not evidenced.
- Of note, the lower RCRT Up success rate was due to clothespins being dropped by participants. This tendency towards unintended hand opening is in keeping with RCNN-TL’s control characteristics results versus those of LDA-Baseline, with higher medians in grip aperture total movement, higher medians in number of grip aperture adjustments, along with larger IQRs in Reach and Grasp phases of these same metrics.
Findings from user experience metrics
Fig 7 presented the user experience metrics. Despite this work’s improved control outcomes, no significant differences were identified in the NASA-TLX and usability surveys. This work uncovered the following insights, to guide future use of user experience metrics:
- Without the provision of participants’ scores from their first testing session upon return for their second session (following the washout period of at least one week), their initial anchor scores of “good” and “poor” were not likely to have been precisely recalled.
- Given that participants were without limb loss, they only had a perception of fully functional control using their intact hand and wrist. They did not have a baseline perception of poor or diminished control, as none had prior experience with a simulated prosthesis. As a result, their subjective anchor scores of “good” and “poor” in their first session were likely influenced by their perception of perfect control, whereas in their second session, they might have been further influenced by their first session’s simulated prosthesis control experience.
Overall, a specific question about which controller each participant preferred (asked at the end of their second session) would better gauge their controller partiality. In addition, reminding participants of their first-session scores immediately prior to their second session survey completion, might address expectation-related variability in anchor scores [86]. After this current study was conducted, a new Prosthesis Task Load Index (PROS-TLX) was developed and validated [87], and should be considered in future comparative prosthesis control research.
Evidence of training routine reduction
The General Participant Group performed a long (200-second) pre-training routine prior to RCNN-TL use. This pre-training duration is similar to that of position-aware controller solutions in the literature [8, 12, 20, 79, 88–90]. RCNN-TL retraining, as performed by the SP Participant Group, was accomplished using a shortened (60-second) routine. That is, a 70% decrease in model training duration resulted due to the introduction of transfer learning. This current research, therefore, confirms that TL is a valuable adjunct to RCNN-based classification control, as it offers a model starting point that needs only to be calibrated using a smaller amount of individual-specific data. Notably, a TL solution is not possible with LDA-based control. Overall, this training routine reduction solution shows promise towards solving the limb position effect challenge—without the requirement of a burdensome training routine.
Limitations
As a first limitation of this study, participants without upper limb impairment were recruited rather than myoelectric prosthesis users. Although these participants learned how to control a simulated prosthesis, further practice may have been necessary to accurately represent the control capabilities of myoelectric prosthesis users. Secondly, the implementation of the surveys may not have adequately captured user experience data. Thirdly, previous work had suggested that the conditions under which the RCNN-TL pre-training data was collected could result in control flaws [26]—because pre-training data from participants without a donned simulated prosthesis were too dissimilar from those with a donned simulated prosthesis (the conditions under which the retraining and device use occurred). This testing condition dissimilarity consideration was not examined in the current study. Finally, although our experimentation used optical motion capture technology to gather movement data, the use of this costly equipment might not always be feasible. As such, our suite of control metrics is not solely reliant on motion capture data for metrics calculations. Alternative sources that capture grip aperture and wrist rotation data (such as from device motor positions), along with shoulder flexion/extension data (such as from markerless motion capture technology or IMUs [91]) can be used for these calculations. Furthermore, other methods of segmenting functional tasks into Reach, Grasp, Transport, and Release phases (such as segmenting markerless motion capture data or IMU-captured data) can be implemented in preparation for metrics calculations.
Future work
RCNN-TL future work.
Next steps for RCNN-TL should focus on improvements to device control in instances when the limb position effect is least likely to occur. Improvements to pre-training data collection conditions need to be studied. Examination of RCNN-TL using myoelectric prosthesis users is also a necessary step.
Suite of control evaluation metrics future work.
To verify whether the suite of metrics introduced in this study are beneficial, future work should examine different controllers, using both participants without limb loss wearing a simulated prosthesis and actual myoelectric prosthesis users operating a commercial device. Further use of the suite of metrics in control research is expected to uncover recommended subsets of discriminant metrics, based on each study’s goals and experimentation methods—for instance, studies with tasks that do not require wrist rotation need not examine simultaneous wrist-shoulder movements; they should instead focus on metrics such as the number of grip aperture adjustments and grip aperture plateau time.
Conclusion
This work reinvestigated earlier comparative RCNN-TL versus LDA-Baseline research, which recommended that pattern recognition-based control not be judged by task performance alone, but rather, that control characteristics also be measured [26]. Then collectively, the task performance and control characteristics should be weighed against qualitative user experience [26]. The current study heeded these recommendations, and in doing so, contributed and tested a viable suite of myoelectric prosthesis control evaluation metrics for use in future comparative control model research. Using these metrics, this study has contributed insights into occurrences and implications of the limb position effect challenge and offered validation that TL-based neural network control solutions show promise towards solving this pervasive problem. The suite of metrics introduced and subsequently used in this work is expected to benefit future research intent on improving rehabilitation device control.
Supporting information
S1 Table. Metrics examples.
This table contains figures that exemplify “good” and “poor” results for selected metrics. For each metric presented, a description of the example figure precedes that associated good/poor graphs.
https://doi.org/10.1371/journal.pone.0291279.s001
(PDF)
Acknowledgments
We thank Quinn Boser, Thomas R. Dawson, Michael R. Dawson, and Albert Vette for experimental design and data processing assistance.
References
- 1.
Braza DW, Yacub Martin JN. Upper Limb Amputations. In: Essentials of Physical Medicine and Rehabilitation: Musculoskeletal Disorders, Pain, and Rehabilitation. 4th ed. Elsevier Inc.; 2018. p. 651–657.
- 2. Cordella F, Ciancio AL, Sacchetti R, Davalli A, Cutti AG, Guglielmelli E, et al. Literature review on needs of upper limb prosthesis users. Frontiers in Neuroscience. 2016;10:209. pmid:27242413
- 3. Smith LH, Kuiken TA, Hargrove LJ. Myoelectric control system and task-specific characteristics affect voluntary use of simultaneous control. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2016;24(1):109–16. pmid:25769167
- 4. Scheme E, Englehart K. Electromyogram pattern recognition for control of powered upper-limb prostheses: State of the art and challenges for clinical use. Journal of Rehabilitation Research and Development. 2011;48(6):643–59. pmid:21938652
- 5. Shehata AW, Williams HE, Hebert JS, Pilarski PM. Machine learning for the control of prosthetic arms. IEEE Signal Processing Magazine. 2021;38(4):46–53.
- 6.
Scheme E, Fougner A, Stavdahl O, Chan ADC, Englehart K. Examining the adverse effects of limb position on pattern recognition based myoelectric control. Proceedings of the 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2010; p. 6337–40.
- 7. Campbell E, Phinyomark A, Scheme E. Current trends and confounding factors in myoelectric control: Limb position and contraction intensity. Sensors (Switzerland). 2020;20(6):1613. pmid:32183215
- 8. Geng Y, Samuel OW, Wei Y, Li G. Improving the Robustness of Real-Time Myoelectric Pattern Recognition against Arm Position Changes in Transradial Amputees. BioMed Research International. 2017;2017:5090454. pmid:28523276
- 9. Du Y, Jin W, Wei W, Hu Y, Geng W. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors (Switzerland). 2017;17(3):458. pmid:28245586
- 10. Koppolu PK, Chemmangat K. A two-stage classification strategy to reduce the effect of wrist orientation in surface myoelectric pattern recognition. The Proceedings of the IEEE International Conference on Signal Processing and Communications (SPCOM). 2022.
- 11. Gharibo JS, Naish MD. Multi-modal Prosthesis Control using sEMG, FMG and IMU Sensors. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2022; p. 2983–7. pmid:36086300
- 12. Geng Y, Zhou P, Li G. Toward attenuating the impact of arm positions on electromyography pattern-recognition based motion classification in transradial amputees. Journal of NeuroEngineering and Rehabilitation. 2012;9:74. pmid:23036049
- 13. Gigli A, Gijsberts A, Castellini C. Natural myocontrol in a realistic setting: A comparison between static and dynamic data acquisition. Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR). 2019; p. 1061–6.
- 14. Spieker V, Ganguly A, Haddadin S, Piazza C. An adaptive multi-modal control strategy to attenuate the limb position effect in myoelectric pattern recognition. Sensors. 2021;21(21):7404. pmid:34770709
- 15. Khushaba RN, Takruri M, Miro JV, Kodagoda S. Towards limb position invariant myoelectric pattern recognition using time-dependent spectral features. Neural Networks. 2014;55:42–58. pmid:24721224
- 16. Park KH, Suk HI, Lee SW. Position-independent decoding of movement intention for proportional myoelectric interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2016;24(9):928–39. pmid:26415203
- 17.
Yu Y, Sheng X, Guo W, Zhu X. Attenuating the impact of limb position on surface EMG pattern recognition using a mixed-LDA classifier. Proceedings of the 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO). 2017; p. 1497–502.
- 18. Beaulieu RJ, Masters MR, Betthauser J, Smith RJ, Kaliki R, Thakor NV, et al. Multi-position Training Improves Robustness of Pattern Recognition and Reduces Limb-Position Effect in Prosthetic Control. Journal of Prosthetics and Orthotics. 2017;29(2):54–62. pmid:28983183
- 19. Betthauser JL, Hunt CL, Osborn LE, Masters MR, Levay G, Kaliki RR, et al. Limb Position Tolerant Pattern Recognition for Myoelectric Prosthesis Control with Adaptive Sparse Representations from Extreme Learning. IEEE Transactions on Biomedical Engineering. 2018;65(4):770–8. pmid:28650804
- 20. Shahzad W, Ayaz Y, Khan MJ, Naseer N, Khan M. Enhanced performance for multi-forearm movement decoding using hybrid IMU–SEMG interface. Frontiers in Neurorobotics. 2019;13:43. pmid:31333441
- 21. Mukhopadhyay AK, Samui S. An experimental study on upper limb position invariant EMG signal classification based on deep neural network. Biomedical Signal Processing and Control. 2020;55:101669.
- 22. Shair EF, Suleiman MFA, Abdullah AR, Nazmi N, Saharuddin NZ. Reduction of Limb Position Invariant of SEMG Signals for Improved Prosthetic Control Using Spectrogram. International Journal of Integrated Engineering. 2021;13(5):79–88.
- 23. Williams HE, Shehata AW, Dawson MR, Scheme E, Hebert JS, Pilarski PM. Recurrent convolutional neural networks as an approach to position-aware myoelectric prosthesis control. IEEE Transactions on Biomedical Engineering. 2022;69(7):2243–55. pmid:34986093
- 24. Geethanjali P. Myoelectric control of prosthetic hands: State-of-the-art review. Medical Devices: Evidence and Research. 2016;9:247–55. pmid:27555799
- 25.
Williams HE, Günther J, Hebert JS, Pilarski PM, Shehata AW. Composite Recurrent Convolutional Neural Networks Offer a Position-Aware Prosthesis Control Alternative While Balancing Predictive Accuracy with Training Burden. Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR). 2022.
- 26.
Williams HE, Hebert JS, Pilarski PM, Shehata AW. A Case Series in Position-Aware Myoelectric Prosthesis Control using Recurrent Convolutional Neural Network Classification with Transfer Learning. Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR). 2023;.
- 27. Williams HE, Chapman CS, Pilarski PM, Vette AH, Hebert JS. Myoelectric prosthesis users and non-disabled individuals wearing a simulated prosthesis exhibit similar compensatory movement strategies. Journal of NeuroEngineering and Rehabilitation. 2021;18:72. pmid:33933105
- 28.
Williams HE, Boser QA, Pilarski PM, Chapman CS, Vette AH, Hebert JS. Hand Function Kinematics when using a Simulated Myoelectric Prosthesis. In: Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR). vol. 2019-June. Toronto, Canada; 2019. p. 169–74.
- 29. Valevicius AM, Boser QA, Lavoie EB, Murgatroyd GS, Pilarski PM, Chapman CS, et al. Characterization of normative hand movements during two functional upper limb tasks. PLoS ONE. 2018;13(6):e0199549. pmid:29928022
- 30. Hussaini A, Kyberd P. Refined clothespin relocation test and assessment of motion. Prosthetics and Orthotics International. 2017;41(3):294–302. pmid:27473641
- 31.
Boser QA. Characterizing the Visuomotor Behaviour of Upper Limb Body-Powered Prosthesis Users. University of Alberta; 2019.
- 32. Williams HE, Chapman CS, Pilarski PM, Vette AH, Hebert JS. Gaze and Movement Assessment (GaMA): Inter-site validation of a visuomotor upper limb functional protocol. PLoS ONE. 2019;14(12):e0219333. pmid:31887218
- 33. Mathiowetz V, Volland G, Kashman N, Weber K. Adult norms for the Box and Block Test of manual dexterity. The American journal of occupational therapy. 1985;39(6):386–391. pmid:3160243
- 34. Jebsen RH, Taylor N, Trieschmann RB, Trotter MJ, Howard LA. An objective and standardized test of hand function. Archives of Physical Medicine and Rehabilitation. 1969;50(6):311–9. pmid:5788487
- 35. Resnik L, Adams L, Borgia M, Delikat J, Disla R, Ebner C, et al. Development and evaluation of the activities measure for upper limb amputees. Archives of Physical Medicine and Rehabilitation. 2013;94(3):488–94. pmid:23085376
- 36. Light CM, Chappell PH, Kyberd PJ. Establishing a standardized clinical assessment tool of pathologic and prosthetic hand function: Normative data, reliability, and validity. Archives of Physical Medicine and Rehabilitation. 2002;83(6):776–83. pmid:12048655
- 37. Capsi-Morales P, Piazza C, Sjoberg L, Catalano MG, Grioli G, Bicchi A, et al. Functional assessment of current upper limb prostheses: An integrated clinical and technological perspective. PLoS ONE. 2023;18(8):e0289978. pmid:37585427
- 38. Hermansson LM, Fisher AG, Bernspång B, Eliasson A. Assessmet of Capacity for Myoelectric Control: A new Rasch-built measure of prosthetic hand control. Journal of Rehabilitation Medicine. 2005;37(3):166–71. pmid:16040474
- 39. Simon AM, Turner KL, Miller LA, Potter BK, Beachler MD, Dumanian GA, et al. User Performance With a Transradial Multi-Articulating Hand Prosthesis During Pattern Recognition and Direct Control Home Use. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2023;31:271–81. pmid:36355739
- 40. Teh Y, Hargrove LJ. Understanding limb position and external load effects on real-time pattern recognition control in amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2020;28(7):1605–13. pmid:32396094
- 41. Ortiz-Catalan M, Hakansson B, Branemark R. Real-time and simultaneous control of artificial limbs based on pattern recognition algorithms. IEEE Trans Neural Syst Rehabil Eng. 2014;22(4):756–64. pmid:24710833
- 42. Young AJ, Smith LH, Rouse EJ, Hargrove LJ. A comparison of the real-time controllability of pattern recognition to conventional myoelectric control for discrete and simultaneous movements. Journal of NeuroEngineering and Rehabilitation. 2014;11:5. pmid:24410948
- 43. Vujaklija I, Shalchyan V, Kamavuako EN, Jiang N, Marateb HR, Farina D. Online mapping of EMG signals into kinematics by autoencoding. Journal of NeuroEngineering and Rehabilitation. 2018;15:21. pmid:29534764
- 44. Jiang N, Vujaklija I, Rehbaum H, Graimann B, Farina D. Is accurate mapping of EMG signals on kinematics needed for precise online myoelectric control? IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014;22(3):549–58. pmid:24235278
- 45. Jiang N, Rehbaum H, Vujaklija I, Graimann B, Farina D. Intuitive, online, simultaneous, and proportional myoelectric control over two degrees-of-freedom in upper limb amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014;22(3):501–10. pmid:23996582
- 46. Ameri A, Kamavuako EN, Scheme EJ, Englehart KB, Parker PA. Support vector regression for improved real-time, simultaneous myoelectric control. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014;22(6):1198–209. pmid:24846649
- 47. Ameri A, Akhaee MA, Scheme E, Englehart K. Real-time, simultaneous myoelectric control using a convolutional neural network. PLoS ONE. 2018;13(9):e0203835. pmid:30212573
- 48. Ameri A, Akhaee MA, Scheme E, Englehart K. Regression convolutional neural network for improved simultaneous EMG control. Journal of Neural Engineering. 2019;16(3):036015. pmid:30849774
- 49. Lukyanenko P, Dewald HA, Lambrecht J, Kirsch RF, Tyler DJ, Williams MR. Stable, simultaneous and proportional 4-DoF prosthetic hand control via synergy-inspired linear interpolation: a case series. Journal of NeuroEngineering and Rehabilitation. 2021;18:50. pmid:33736656
- 50. Piazza C, Rossi M, Catalano MG, Bicchi A, Hargrove LJ. Evaluation of a Simultaneous Myoelectric Control Strategy for a Multi-DoF Transradial Prosthesis. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2020. pmid:32804650
- 51. Simon AM, Hargrove LJ, Lock BA, Kuiken TA. Target Achievement Control Test: evaluating real-time myoelectric pattern-recognition control of multifunctional upper-limb prostheses. Journal of rehabilitation research and development. 2011;48(6):619–27. pmid:21938650
- 52. Wurth SM, Hargrove LJ. A real-time comparison between direct control, sequential pattern recognition control and simultaneous pattern recognition control using a Fitts’ law style assessment procedure. Journal of NeuroEngineering and Rehabilitation. 2014;11:91. pmid:24886664
- 53. Fougner AL, Stavdahl O, Kyberd PJ. System training and assessment in simultaneous proportional myoelectric prosthesis control. Journal of NeuroEngineering and Rehabilitation. 2014;11:75. pmid:24775602
- 54. Farrell TR, ff Weir RF. The optimal controller delay for myoelectric prostheses. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2007;15(1):111–8. pmid:17436883
- 55. Johansen D, Cipriani C, Popovic DB, Struijk LNSA. Control of a Robotic Hand Using a Tongue Control System-A Prosthesis Application. IEEE Transactions on Biomedical Engineering. 2016;63(7):1368–76. pmid:26780786
- 56.
Amsuess S, Goebel P, Graimann B, Farina D. Extending mode switching to multiple degrees of freedom in hand prosthesis control is not efficient. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2014; p. 658–61.
- 57. Amsuess S, Goebel P, Graimann B, Farina D. A multi-class proportional myocontrol algorithm for upper limb prosthesis control: Validation in real-life scenarios on amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2015;23(5):827–36. pmid:25296406
- 58. Amsuess S, Vujaklija I, Goebel P, Roche AD, Graimann B, Aszmann OC, et al. Context-dependent upper limb prosthesis control for natural and robust use. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2016;24(7):744–53. pmid:26173217
- 59.
Cipriani C, Zaccone F, Stellin G, Beccai L, Cappiello G, Carrozza MC, et al. Closed-loop controller for a bio-inspired multi-fingered underactuated prosthesis. Proceedings of the IEEE International Conference on Control and Automation. 2006; p. 2111–2116.
- 60. Catalano MG, Grioli G, Farnioli E, Serio A, Piazza C, Bicchi A. Adaptive synergies for the design and control of the Pisa/IIT SoftHand. International Journal of Robotics Research. 2014;33(5):768–82.
- 61.
Matrone G, Cipriani C, Carrozza MC, Magenes G. Two-channel real-time EMG control of a dexterous hand prosthesis. Proceedings of the 5th International IEEE/EMBS Conference on Neural Engineering (NER). 2011; p. 554–7.
- 62. Došen S, Cipriani C, Kostić M, Controzzi M, Carrozza MC, Popović DB. Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation. Journal of NeuroEngineering and Rehabilitation. 2010;7:42. pmid:20731834
- 63. Marasco PD, Hebert JS, Sensinger JW, Beckler DT, Thumser ZC, Shehata AW, et al. Neurorobotic fusion of prosthetic touch, kinesthesia, and movement promotes intrinsic. Science Robotics. 2021;6:eabf3368.
- 64. Valevicius AM, Boser QA, Lavoie EB, Chapman CS, Pilarski PM, Hebert JS, et al. Characterization of normative angular joint kinematics during two functional upper limb tasks. Gait & Posture. 2019;69:176–86. pmid:30769260
- 65. Lavoie EB, Valevicius AM, Boser QA, Kovic O, Vette AH, Pilarski PM, et al. Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks. Journal of Vision. 2018;18(6):18. pmid:30029228
- 66. Williams HE, Faridi P, Shehata AW, Hebert JS, Pilarski PM. Using Transfer Learning to Reduce the Burden of Training for Position-Aware Myoelectric Prosthetic Control. Proceedings of RehabWeek Virtual ‘21. 2021; p. P–130.
- 67. Patel GK, Hahne JM, Castellini C, Farina D, Dosen S. Context-dependent adaptation improves robustness of myoelectric control for upper-limb prostheses. Journal of Neural Engineering. 2017;14(5):056016. pmid:28691694
- 68. Fu Q, Santello M. Improving fine control of grasping force during hand-object interactions for a soft synergy-inspired myoelectric prosthetic hand. Frontiers in Neurorobotics. 2018;11:71. pmid:29375360
- 69. Bouwsema H, Kyberd PJ, Hill W, van der Sluis CK, Bongers RM. Determining skill level in myoelectric prosthesis use with multiple outcome measures. Journal of Rehabilitation Research and Development. 2012;49(9):1331–48. pmid:23408215
- 70.
Zhang W, White M, Zahabi M, Winslow AT, Zhang F, Huang H, et al. Cognitive workload in conventional direct control vs. pattern recognition control of an upper-limb prosthesis. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2017; p. 2335–40.
- 71. Ingraham KA, Ferris DP, Remy CD. Evaluating physiological signal salience for estimating metabolic energy cost from wearable sensors. Journal of Applied Physiology. 2018;126(3):717–29.
- 72. Hart SG. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society. 2006; p. 904–8.
- 73. Park J, Zahabi M. Cognitive Workload Assessment of Prosthetic Devices: A Review of Literature and Meta-Analysis. IEEE Transactions on Human-Machine Systems. 2022;52(2):181–95.
- 74.
Brenneis DJA, Dawson MR, Tanikawa H, Hebert JS, Carey JP, Pilarski PM. The effect of an automatically levelling wrist control system. Proceedings of the IEEE International Conference on Rehabilitation Robotics. 2019; p. 816–823.
- 75. Hebert JS, Lewicke J. Case report of modified Box and Blocks test with motion capture to measure prosthetic function. The Journal of Rehabilitation Research and Development. 2012;49(8):1163–1174. pmid:23341309
- 76. Hebert JS, Lewicke J, Williams TR, Vette AH. Normative data for modified Box and Blocks test measuring upper-limb function via motion capture. Journal of Rehabilitation Research and Development. 2014;51(6):918–32. pmid:25356979
- 77. Ahmadizadeh C, Khoshnam M, Menon C. Human Machine Interfaces in Upper-Limb Prosthesis Control: A Survey of Techniques for Preprocessing and Processing of Biosignals. IEEE Signal Processing Magazine. 2021;38(4):12–22.
- 78.
Leone F. Pattern Recognition algorithms for upper-limb prosthetics control Co-Supervisor. Università Campus Bio-Medico; 2021.
- 79. Fougner A, Scheme E, Chan ADC, Englehart K, Stavdahl O. Resolving the limb position effect in myoelectric pattern recognition. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2011;19(6):644–51. pmid:21846608
- 80. Hallworth BW, Austin JA, Williams HE, Rehani M, Shehata AW, Hebert JS. A Modular Adjustable Transhumeral Prosthetic Socket for Evaluating Myoelectric Control. IEEE Journal of Translational Engineering in Health and Medicine. 2020;8:0700210. pmid:32670675
- 81. Wells ED, Shehata AW, Dawson MR, Carey JP, Hebert JS. Preliminary Evaluation of the Effect of Mechanotactile Feedback Location on Myoelectric Prosthesis Performance Using a Sensorized Prosthetic Hand. Sensors. 2022;22(10):3892. pmid:35632311
- 82. Côté-Allard U, Campbell E, Phinyomark A, Laviolette F, Gosselin B, Scheme E. Interpreting Deep Learning Features for Myoelectric Control: A Comparison With Handcrafted Features. Frontiers in Bioengineering and Biotechnology. 2020;8:158. pmid:32195238
- 83. Parajuli N, Sreenivasan N, Bifulco P, Cesarelli M, Savino S, Niola V, et al. Real-Time EMG Based Pattern Recognition Control Challenges and Future Implementation. Sensors (Switzerland). 2019;19(20):4596. pmid:31652616
- 84.
Dawson MR, Williams HE, Murgatroyd GS, Hebert JS, Pilarski PM. BrachIOPlexus: Myoelectric training software for clinical and research applications. Proceedings of the Myoelectric Controls Symposium (MEC20). 2020; p. 87–90.
- 85. Boser QA, Valevicius AM, Lavoie EB, Chapman CS, Pilarski PM, Hebert JS, et al. Cluster-based upper body marker models for three-dimensional kinematic analysis: Comparison with an anatomical model and reliability analysis. Journal of Biomechanics. 2018;72:228–34. pmid:29530500
- 86. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology. 1988;52:139–83.
- 87. Parr JVV, Galpin A, Uiga L, Marshall B, J WD, Franklin ZC, et al. A tool for measuring mental workload during prosthesis use: The Prosthesis Task Load Index (PROS-TLX). PLoS ONE. 2023;18(5):e0285382. pmid:37141379
- 88.
Scheme E, Biron K, Englehart K. Improving myoelectric pattern recognition positional robustness using advanced training protocols. Proceedings of the 33rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2011; p. 4828–31.
- 89.
Boschmann A, Platzner M. Reducing the limb position effect in pattern recognition based myoelectric control using a high density electrode array. Proceedings of the ISSNIP Biosignals and Biorobotics Conference (BRC). 2013.
- 90.
Radmand A, Scheme E, Englehart K. A characterization of the effect of limb position on EMG features to guide the development of effective prosthetic control schemes. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2014; p. 662–7.
- 91. Wang SL, Civillico G, Niswander W, Kontson KL. Comparison of Motion Analysis Systems in Tracking Upper Body Movement of Myoelectric Bypass Prosthesis Users. Sensors. 2022;22(8):2953. pmid:35458943