Wheelchair simulator system

The virtual wheelchair simulator that will be used in the pilot study is EWATS system as per Fig 1 [38]. This system combines two primary components: the wheelchair simulator (Fig 1A) and devices for capturing user response data (Fig 1B), which is controlled using a wheelchair joystick adapted for simulator use on a computer. It collects implicit physiological data through tools such as the smartwatch from Empatica (EmbracePlus), HD1080p Logi Camera integrated with the OpenFace library [39], Mindband and OpenVibe Interface [40]. Additionally, it gathers explicit self-report data and feedback from semi-structured interviews conducted with users interacting with the wheelchair simulator. Together, these data sources provide a comprehensive understanding of user experience and performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Wheelchair simulator system.

(A) Diagram of the simulator’s core components, including the computer, virtual environment, and control interface (joystick controller). (B) Devices for capturing user responses and physiological data.

https://doi.org/10.1371/journal.pone.0325186.g001

The simulator was developed through a collaborative process with key stakeholders, including wheelchair users, clinicians, and rehabilitation specialists. This co-design approach ensured the system accurately reflects real-world mobility challenges while assessing both motor skills and cognitive abilities. Wheelchair users provided input on usability and task realism, while clinicians ensured alignment with clinical standards like the, WST [6], PMRT [7] and MoCA [41].

Through several iterations and adaptations, the simulator has undergone significant improvements based on feedback from wheelchair users and a control group of non-wheelchair users [42–45]. For instance, previous refinements informed adjustments to the simulator’s tasks and algorithms, enhancing its ability to assess power mobility skills and cognitive abilities. In this pilot study, the focus is on evaluating the feasibility of using simulator-derived metrics to estimate scores for the reference’s standards (PWRT, WST and MoCA). The study will explore the extent to which its performance metrics align with these reference standards, informing future refinements in subsequent validation research.

The simulator has been specifically tailored to accommodate different user groups. Wheelchair users interacted with virtual tasks designed to replicate real-world mobility challenges relevant to their daily experiences, while the control group, consisting of non-wheelchair users provided a baseline for evaluating the system’s sensitivity. While non-disabled individuals may still face challenges using a powered wheelchair, their inclusion allows for a controlled baseline comparison, helping assess whether the simulator can differentiate performance variations based on prior experience rather than underlying physical or cognitive impairments. A more detailed rationale for selecting this control group, including its role in assessing cognitive influences and mobility performance differentiation, is provided in the group description section.

To further refine the simulator, post-test semi-structured interviews will capture end-user feedback to guide future enhancements. This iterative refinement process ensures that the simulator continues to evolve based on real-world needs, making it a robust and adaptable tool for assessing mobility and cognitive abilities in virtual environments.

Study design and setting

This pilot study adopts a prospective, multicentric, mixed-methods approach with a cross-sectional exploratory design. The primary aim is to develop and refine a protocol for evaluating the feasibility and reliability of a virtual wheelchair simulator as a clinical tool for assessing power mobility skills and cognitive abilities in wheelchair users with varying levels of experience in controlling a powered wheelchair. Therefore, this study will also explore the simulator’s ability to differentiate between novice and experienced wheelchair users while examining the acceptability of the tool for both assessment and training purposes.

This multicentre study will be conducted across multiple locations, including Irish Wheelchair Association (IWA) centres and Technological University of the Shannon (TUS). Conducting the study in multiple settings will provide a diverse participant pool, improving the transferability and generalizability of the findings.

Participants

The study will encompass three distinct participant groups: wheelchair users, healthcare professionals, and a control group of non-disabled individuals. Participants will be identified through local healthcare networks, TUS and IWA. Data collection will be conducted in two phases.

In Phase 1, wheelchair users will interact with the simulator to assess their power mobility skills and cognitive abilities, with classifications based on their experience level and cognitive function. In Phase 2, healthcare professionals will review and evaluate the simulator-generated metrics, ranking their clinical relevance and providing feedback on the usability and effectiveness of the tool. The control group, composed of individuals with no prior wheelchair experience, will serve as a baseline to test the simulator’s sensitivity in distinguishing variations in mobility and cognitive performance.

In addition to the quantitative and performance data, the study protocol includes semi-structured interviews with people with disabilities and healthcare professionals at the post-test session of each respective phase. These interviews will provide e qualitative insights into the participants’ experiences and perceptions of the simulator, contributing to a comprehensive evaluation of its effectiveness and utility.

Assessments

Participants will undergo a series of assessments at different stages: before, during, and after using the simulator as outlined in Table 1. Cognitive functioning will be evaluated using MoCA, while power mobility skills will be measured through the WST and PMRT. Additionally, self-reported questionnaires will assess confidence levels (sub-score of WST) and overall QoE with the simulator.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Outcomes table.

https://doi.org/10.1371/journal.pone.0325186.t001

QoE will be assessed after the simulator experience through several key aspects, including immersive levels (Immersive Experience Questionnaire, IPQ, [48]), emotional responses (Self-Assessment Manikin, SAM, [49]), cognitive workload (PAAS scale [25] after each task and NASA-TLX [50] at the end of test), and usability (System Usability Scale, SUS, [51]).

In Phase 2, healthcare professionals will review the performance records from Phase 1 (wheelchair users and control group), scoring participants’ mobility skills using the PMRT. Semi-structured interviews will also be conducted to gather expert feedback on the simulator’s impact on both mobility skills and cognitive functioning, as well as its potential for integration into clinical practice.

Outcomes

The study will evaluate a range of outcomes across primary and secondary categories to assess the effectiveness of the wheelchair simulator, as presented in Table 1. Primary outcomes will include mobility skills measured through the PMRT and WST, cognitive functioning assessed via the MoCA, and evaluations by healthcare professionals during Phase 2, who will review Phase 1 records and score participant performance using the PMRT.

Secondary outcomes will encompass various aspects of QoE, including immersive levels assessed with the IPQ, emotional responses measured with the SAM, and usability evaluated with the SUS, all conducted after the simulator session. Cognitive workload will be assessed using the PAAS after each task and the NASA-TLX following the session. Participants’ simulator performance metrics will be collected throughout the sessions. Physiological metrics, including heart rate and stress levels, will be monitored using wearable devices during the tests. This comprehensive evaluation aims to provide valuable insights into the simulator’s effectiveness in enhancing mobility skills and cognitive functioning, while also considering the overall experience of participants and the perspectives of healthcare professionals.

Study procedure

The study will follow a structured protocol divided into two phases as per Fig 2, commencing with participant recruitment, which will include wheelchair users, control and healthcare professions participants. Informed consent will be obtained from all participants prior to any assessments and demographics information as detailed in Appendix S1 and S2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Study procedure overview.

Schematic representation of the mixed-methods study. The procedure includes participant recruitment, pre-assessment, simulator-based tasks with continuous data recording, post-assessment questionnaires, and final evaluation sessions.

https://doi.org/10.1371/journal.pone.0325186.g002

Phase 1: Wheelchair users and control group.

In the pre-test phase, participants will first rest for five minutes to collect physiological baseline data. Following this, a trained healthcare professional will administer the MoCA assessment to evaluate cognitive functioning. The WST questionnaire will then be completed by participants as a self-reported measure to assess their baseline confidence levels and mobility skills. After these assessments, participants will engage in a simulator free practice session, allowing them to familiarize themselves with the control features and settings of the wheelchair simulator, including speed and acceleration adjustments.

During the test phase, participants will complete a series of predefined tasks based on the PMRT structure tasks while using the simulator. Throughout this simulation, data will be collected on various performance metrics, including task completion time, errors, and joystick control accuracy. Physiological metrics such as heart rate and electrodermal activity will be continuously monitored using wearable devices. Additionally, cognitive workload will be assessed after each task using the PAAS scale.

Following the simulation tasks, participants will undergo a series of post-test assessments. These will include the QoE assessment and open-ended questions to capture qualitative feedback on participants’ experiences. Cognitive workload will be measured using the NASA-TLX at the end of the test, alongside the PAAS Scale, which will be administered during the task. The steps of phase 1 are outlined in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Phase 1 overview.

Illustration of Phase 1 involving wheelchair users and control participants. This phase includes pre-assessment, simulator task performance, continuous data capture, post-assessment, and comparison of simulator-based metrics with clinical scores.

https://doi.org/10.1371/journal.pone.0325186.g003

Phase 2: Healthcare professionals.

In this phase, healthcare professionals will evaluate the performance records of wheelchair users and control group from Phase 1. Prior to their assessments, each professional will be briefed on the study’s objectives and the specific metrics to consider, and the methodology for evaluation. The professionals will review anonymized simulator session data, including visualization of the wheelchair path and performance records. These observation-based scores will remain confidential and will only be disclosed to the healthcare professionals after they have completed their independent evaluations.

Each healthcare professional will complete the PMRT sheet anonymously, scoring participants’ performance based on established criteria, which will evaluate task completion, control proficiency, and overall performance in the simulated environment. This method helps to ensure objectivity and minimizing potential bias. Additionally, they will review results from the Wheelchair Skills Test questionnaire to gain insights into participants’ self-reported confidence levels with various wheelchair skills as measured by the simulation.

Healthcare professionals will also complete Quality of Experience assessment and open-ended questions specifically designed to gather qualitative insights into the simulator’s potential as a clinical tool, providing valuable feedback on its effectiveness for clinical training and assessment, as well as identifying strengths, limitations, and areas for improvement.

Following their evaluations, healthcare professionals will offer additional qualitative insights based on their observations, contributing to the overall analysis of the simulator’s effectiveness in enhancing mobility skills and cognitive functioning. The findings from this phase will be synthesized with the anonymized data from Phase 1 to facilitate a robust analysis of the simulator’s impact on participants. The steps of phase 2 are outlined in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Phase 2 overview.

Illustration of Phase 2 involving healthcare professionals. Phase 1 performance records are independently reviewed and scored using the Power Mobility Road Test (PMRT) sheet, followed by qualitative interviews and evaluation of the simulator’s assessment utility.

https://doi.org/10.1371/journal.pone.0325186.g004

Ethics

This study has been approved by the Research Ethics Committee at Technological University of Shannon (TUS), Athlone, Ireland, before commencing the recruitment. All procedures adhere to the principle of the Helsinki Declaration and follow the STARD (Standards for reporting Diagnostic Accuracy) guidelines [52] for reporting, supplementary material S1 Appendix. Informed, written consent will be obtained from all participants before their involvement in the study. Recruitment has been taken place at the Irish Wheelchair Association (IWA) and TUS facilities from 15^th of August 2024–29^th May 2025.

Status and timeline of the study

Participant recruitment began on 15^th August 2024 and is currently ongoing, with an estimated end date of 29^th May 2025. Table 2 provides an overview of the study’s timeline, outlining the stages, their respective status, and detailed descriptions of activities undertaken or planned.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Study timeline.

https://doi.org/10.1371/journal.pone.0325186.t002

Test methods

Quantitative data analysis

Principal Component Analysis (PCA) will be applied to the simulator’ metrics to reduce the dimensionality of the dataset [56,57]. This approach will condense the data into a smaller set of principal components, retaining the essential variance while minimizing redundancy [56,57]. PCA will serve as a preprocessing step for the statistical models (e.g., decision trees, random forests, or support vector machines) by streamlining the input features, thus improving model efficiency and reducing overfitting [58,59]. It will also be used for exploratory data analysis to uncover patterns in the data, such as differentiating or identifying similarities between novice and expert users based on their mobility and cognitive profiles [60,61]. Clustering methods, like k-means, may be applied to these principal components to profile user groups [59,62].

Later, another statistical method will be employed to estimate clinical accuracy metrics, including sensitivity, specificity, and area under the curve (AUC), by comparing the outcomes of the virtual wheelchair simulator with the results of PMRT and the MoCA. ROC curve analysis will be utilized to evaluate the performance of the simulator and determine optimal cut-off points for identifying mobility skills and cognitive abilities. To evaluate the clinical accuracy of the simulator’s metrics, the agreement between the estimated scores from the simulator and the actual PMRT and MoCA scores will be assessed using correlation coefficients, Bland-Altman plots, and error measures such as mean absolute error (MAE) and root mean square error (RMSE). Participant age will be recorded and included as a covariate in these statistical analyses. Given that age may influence both mobility performance and cognitive function, adjusted analyses will be performed to control for potential confounding effects and ensure accurate interpretation of simulator-derived metrics. These analyses will assess whether the simulator can reliably support clinical decision-making in mobility and cognitive assessments.

Additionally, artificial intelligence (AI) models will be implemented to estimate PMRT and MoCA scores based on simulator-derived performance metrics. The AI analysis workflow will align with methodologies from peer-reviewed studies [63,64], following a structured approach: first, collecting a range of simulator-derived metrics and applying feature selection to identify the most relevant indicators of driving skills and cognitive abilities. Next, training machine learning (ML) models on datasets where both simulator-derived metrics and actual clinical scores or expert ratings are available.

The primary objective is to predict scores on a 1–4 scale, indicating proficiency in power wheelchair operation, and to estimate MoCA scores for classifying users into cognitive function categories: no impairment, mild impairment, or moderate impairment. Advanced cognitive impairment cases will be excluded as per the study criteria. A range of ML techniques, including regression models, decision trees, random forests, and support vector machines (SVM), will be explored to develop predictive models. These models will be trained using simulator performance data, such as joystick movements, task completion times and navigation accuracy, to establish relationships with standardized clinical assessments. To enhance model accuracy, feature selection techniques, such as PCA will be applied to refine input variables, ensuring the models prioritize the most relevant mobility and cognitive indicators.

Any indeterminate results from the index test will be documented and excluded from the final analyses to maintain the integrity of the clinical accuracy calculations. Furthermore, missing data will be addressed using imputation techniques, such as last observation carried forward (LOCF) or multiple imputation (MI), ensuring a robust analysis while minimizing potential bias in the results [65,66]. LOCF is a single imputation technique commonly used in longitudinal clinical trials to handle dropouts; the subject’s last observed value is carried forward to fill in all subsequent missing time points [66]. MI imputation is a modern approach that addresses missing data by creating several complete datasets and combining results [65,66]. Instead of filling in one “best guess” for each missing value, in other words, MI generates multiple plausible values based on the distribution of the observed data and uncertainty about the missing data.

Qualitative data analysis

The QoE assessments will employ several Likert-style scales tailored to each evaluation instrument. These include the NASA-TLX [50] and PAAS [25] for cognitive load, SAM [49] scale for emotional state, and 5-point scale for usability and effectiveness of simulator [51]. Each instrument’s unique scale will be analysed separately to gain quantitative insights into the specific dimensions of experience, such as workload, emotional response, and perceived usability. Descriptive statistics, such as means and standard deviations, will summarize these scales, while further statistical tests may be employed to compare QoE scores across different participant groups.

To complement these Likert-scale data, qualitative data from open-ended feedback provided by healthcare professionals will undergo thematic analysis following the framework described by Braun and Clarke [67,68]. The analysis will begin with transcription and familiarization, ensuring an in-depth understanding of participant responses through repeated reading of the data. Next, initial coding will be conducted by highlighting significant excerpts and grouping them into meaningful categories. These codes will then be examined for patterns and relationships, leading to the development of themes that consolidate related concepts. The final phase will involve reviewing, defining and naming themes, ensuring they represent the perspectives of healthcare professionals regarding the simulator’s clinical utility, usability, strength and areas for improvement. This analysis will capture more nuanced insights that are not fully captured by the structured scales, providing a qualitative perspective on the simulator’s perceived value and potential limitations in practice.

By integrating thematic findings with quantitative scale-based results will provide a comprehensive evaluation of the simulator’s effectiveness in assessing mobility skills and cognitive abilities. Therefore, the combination of quantitative and qualitative insights will inform future refinements and the potential integration of the simulator into clinical assessment and training frameworks.

Data management, availability and sharing

The de-identified dataset, which includes anonymized participant data, simulator performance metrics, and assessment scores, will be made publicly available for replication and further analysis. All personally identifiable information, such as participant records, images, and other sensitive data, will be securely anonymized, ensuring that no data traceable to individual participants will be accessible. These measures ensure the privacy and confidentiality of study participants.

To adhere to the FAIR principles—Findability, Accessibility, Interoperability, and Reusability—several practices will be followed. Metadata will be created to provide comprehensive information about the dataset, covering data collection methods, key variables, and file formats. The dataset will be stored in a secure, publicly accessible repository on the Open Science Framework (OSF), accessible at https://osf.io/s2tmz/, ensuring ease of access while maintaining data integrity.

The data will be provided in standardized formats, such as CSV and JSON, to facilitate compatibility with various analysis tools and software. Additionally, extensive documentation and metadata will accompany the dataset, enabling researchers to easily understand and reuse the data for future studies, thus contributing to a robust foundation for ongoing research in this field.

Protocol adaptation and implementation

As the study progresses, amendments may be necessary to adjust participant inclusion criteria, simulator tasks, or assessment protocols based on the feedback received. Any significant changes will be reviewed and approved by the relevant ethics boards. In cases where unexpected challenges—such as technical difficulties or recruitment issues—arise, the study may be paused or terminated. If termination is required, all collected data will be analysed and shared transparently to contribute to the understanding of wheelchair simulators in clinical practice.

The present protocol considers participants’ gender, fragility levels, psychological aspects involved in the mental representation of body image, personal endurance, and tolerance toward using wheelchair similar. However, if culture effects emerge, authors will consider implementing a cross-nation design.