Scoring Tools for the Analysis of Infant Respiratory Inductive Plethysmography Signals

Infants recovering from anesthesia are at risk of life threatening Postoperative Apnea (POA). POA events are rare, and so the study of POA requires the analysis of long cardiorespiratory records. Manual scoring is the preferred method of analysis for these data, but it is limited by low intra- and inter-scorer repeatability. Furthermore, recommended scoring rules do not provide a comprehensive description of the respiratory patterns. This work describes a set of manual scoring tools that address these limitations. These tools include: (i) a set of definitions and scoring rules for 6 mutually exclusive, unique patterns that fully characterize infant respiratory inductive plethysmography (RIP) signals; (ii) RIPScore, a graphical, manual scoring software to apply these rules to infant data; (iii) a library of data segments representing each of the 6 patterns; (iv) a fully automated, interactive formal training protocol to standardize the analysis and establish intra- and inter-scorer repeatability; and (v) a quality control method to monitor scorer ongoing performance over time. To evaluate these tools, three scorers from varied backgrounds were recruited and trained to reach a performance level similar to that of an expert. These scorers used RIPScore to analyze data from infants at risk of POA in two separate, independent instances. Scorers performed with high accuracy and consistency, analyzed data efficiently, had very good intra- and inter-scorer repeatability, and exhibited only minor confusion between patterns. These results indicate that our tools represent an excellent method for the analysis of respiratory patterns in long data records. Although the tools were developed for the study of POA, their use extends to any study of respiratory patterns using RIP (e.g., sleep apnea, extubation readiness). Moreover, by establishing and monitoring scorer repeatability, our tools enable the analysis of large data sets by multiple scorers, which is essential for longitudinal and multicenter studies.

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

II. Acknowledgement
The development of RIPScore was supported in part by the Natural Sciences and Engineering Research Council of Canada. The work of Carlos Alejandro Robles Rubio was supported in part by the Mexican National Council for Science and Technology, and in part by the Queen Elizabeth Hospital of Montreal Foundation Chair in Pediatric Anesthesia. Karen A. Brown was supported in part by the Queen Elizabeth Hospital of Montreal Foundation Chair in Pediatric Anesthesia.

A. Description
RIPScore is an open source, interactive software application for manual scoring of infant Respiratory Inductive Plethysmography (RIP) data. A detailed description of the analysis that can be performed using RIPScore is in [1]. RIPScore was developed in MATLAB TM (The MathWorks, Inc., Natick, MA, USA).

B. Required Libraries
It is necessary to obtain the following libraries to use RIPScore. 2) A library of "true-pattern" segments as defined in [1]. This library is available for download free of charge at datadryad.org (for details see [1]). Download the data and save them in the directory C:\McCRIB\Data.

C. Run RIPScore
The following steps provide a guide to start using RIPScore with default configuration parameters. 3) Start the application by typing RIPScore in the MATLAB command window. 4) RIPScore will load with the pre-defined, default configuration (Blind Scorer Mode). Click OK in the following screen: 6/17 5) RIPScore will then ask for Scorer Identification. Select Add New…: 6) Type the Scorer ID in the text box (e.g., TEST), and click OK: 7) RIPScore will display the number of files available for scorer TEST to analyze. Click OK.
8) At this point RIPScore will randomly select one of the files left and load it for scorer TEST to analyze. 9) Since this is the first time scorer TEST will analyze the current file, RIPScore will output the following message: 10) Click OK. RIPScore will load in Visualization Mode. Scorer TEST is ready to analyze the data. 11) For a detailed description of the main screen controls, and their functionalities see [1]. 12) In this default configuration, the scoring results are stored at: C:\McCRIB\McCRIBS\RIPSCORE\scored 7/17 13) Also, the sample data records are loaded from:

C:\McCRIB\McCRIBS\RIPSCORE\data
These records constitute brief segments of infant cardiorespiratory data, pre-processed by inserting segments with known "true-patterns" using the script preprocess_data_records.m. The original data records were part of the infant data described in [1].
14) RIPScore will save a backup of the analysis every 5 min. 8/17

IV. Configuration
By default, RIPScore is configured to start in Blind Scorer Mode, a mode that randomly selects the next record to analyze, from a directory containing sample data records. For project specific needs it may be necessary to use alternative configurations. There are 2 ways to trigger RIPScore's configuration process, which are: 1) Run the function RIPScore_setParameters with parameter 'C:\McCRIB\McCRIBS\RIPSCORE\RIPScore', or 2) Delete the file C:\McCRIB\McCRIBS\RIPSCORE\RIPScore\cnf.mat and run RIPScore.
This will initialize an interactive configuration process where RIPScore will ask the questions described next.
1) Select RIPScore Mode: Set the Mode in which the application will run.
 Administrator (open): RIPScore runs in Scorer mode; it lets the user load any RIPScore data file. This mode can be used for non-randomized scoring, and to review the scores from different scorers.  Blind scorer (default): RIPScore runs in Scorer mode; it keeps the file selection blinded from the user, by automatically, and randomly selecting the next file to analyze.  Training: RIPScore runs in Training mode; it generates simulated data, provides practice with interactive feedback for the user, and evaluates the user performance. 2) Scoring results path: Choose/create a directory where the scoring analysis results will be saved. In the default configuration, this directory is C:\McCRIB\McCRIBS\RIPSCORE\scored. After selecting the directory, RIPScore will output the full path to it: 9/17 3) Data records path: Choose/create a directory where the raw data to be analyzed is saved. In the default configuration, this directory is C:\McCRIB\McCRIBS\RIPSCORE\data. After selecting the directory, RIPScore will output the full path to it:

4) Set Environment Variables:
Enter the length of the data epoch displayed on the screen (default is 30 s), and the sampling frequency of the data (default is 50 Hz). Click OK.

5) Set Training Mode Variables:
10/17 This is only required if RIPScore is set to run in Training Mode (just click OK for any other mode). RIPScore will suggest the default values. The variables are:  Bootstrap resamples (#): This is an integer used to estimate standard deviations of kappa estimates.  ALPHA: This is a real value in the open interval (0, 1), used for computation of (1-ALPHA) confidence intervals.  wL (samples): This is an integer with half the length of the segment concatenation window; see function combineSignals for additional information.  winType: This is an integer with the type of the concatenation window; see function combineSignals for additional information.
 Consecutive segments (#): This is an integer with the number of consecutive segments, of each pattern type, that are required to finish practice stage.  Segment proportion: A real value in the interval (0,1] that indicates the proportion of a scored segment that has to match the Actual Pattern to be considered correct (in practice stage). This is only required if RIPScore is set to run in Training Mode (click OK and then Esc for any other mode). Choose the file containing the library of "true-pattern" segments. In the default configuration, this file is: C:\McCRIB\Data\POA\TruePattern_Segment_Library\TruePattern_Library.mat.
After selecting the file, RIPScore will output the full path to it: 7) Click OK. RIPScore is now configured, and will save a new cnf.mat file in the application's main directory. 12/17

V. Data Files
This section describes the format of files used by RIPScore; it is divided in input and output files.

B. Output
RIPScore generates several files related to the training of scorers and their analyses. The files are in MATLAB TM *.mat format. The following list describes the contents of these files.
1) Scorer data (e.g., scorer_TEST.mat). This file has a summary of the files analyzed by scorer TEST. The file contains the variables described next.  SCORER: A struct with the following fields: o finished, a struct with a list of all files that have been completely scored. It has the following fields:  file, a 1-by-F cell array of strings, where each cell has the name of a file that has been fully scored.  alias, a 1-by-F cell array of strings that lists the aliases for the files listed in SCORER.finished.file. o current, a struct with the following fields:  file, a string with the name of the file that is currently being scored.  alias, the alias assigned to the file that is currently being scored.
o RIPScoreVersion, a string with the version of RIPScore that created this file.
2) Scoring results (e.g., scored_TEST_ALIAS.mat). This file has the scoring details from an analysis performed by scorer TEST. The term ALIAS in the file name corresponds to the alias of the scored file with name listed in SCORER.finished.file or SCORER.current.file. This file contains the variables described next.  SCORING: A struct with the detailed results from the manual analysis. It has the following fields: o NextScore, an index indicating the number of segments scored (S) plus 1. o Events, an S-by-4 matrix with the RIP patterns assigned to each data segment. The columns are: (1) segment start time, (2) segment end time, (3) code of pattern assigned to the segment (pause=1, asynchronous-breathing=2, movement artifact=3, synchronous-breathing=4, sigh=5, unknown=99), (4) timestamp. o Scorer, a string with the scorer ID. o Comments, a struct with comments that the scorer assigned to scored segments. It has the following fields:  start, a C-by-1 vector with the start time of the segment linked to the comment. 3) Trainee data (e.g., trainee_TEST.mat).
This is a file with a summary of the training results from scorer TEST. The file contains the variables described next.  TRAINEE: A struct with the following fields: o Scorer, a string with the scorer's ID. o level, an integer indicating the latest scorer's training level. o iteration, a 1-by-2 vector with the last session completed in level 1 (col1) and level 2 (col2). o InterAgreement, a 1-by-2 cell array with estimates of the accuracy performance (estimated using the Fleiss' kappa statistic [2,3]) in level 1 (col1) and level 2 (col2). Each cell is a struct array with the following fields:  k, a 1-by-M vector with the kappa estimate from each of the M training sessions.  kj, an M-by-6 vector with the pattern-specific kappa estimate from each of the M training sessions. Each column corresponds to one the 6 unique pattern types: col1=Pause, col2=Asynchronous-breathing, col3=Movement artifact, col4=Synchronous-breathing, col5=Sigh, col6=Unknown.  kstd, a 1-by-M vector with the estimated standard deviation of kappa for the M training sessions. RIPScore estimates standard deviations using the bootstrap method [4].  kjstd, an M-by-6 vector with the estimates of pattern-specific standard deviation of kappa for each of the M training sessions. RIPScore estimates standard deviations using the bootstrap method. Each column corresponds to one the 6 unique pattern types: col1=Pause, col2=Asynchronous-breathing, col3=Movement artifact, col4=Synchronous-breathing, col5=Sigh, col6=Unknown.
o IntraAgreement, a 1-by-2 cell array with estimates of the consistency performance (estimated using the Fleiss' kappa statistic [2,3]) in level 1 (col1) and level 2 (col2). Each cell is a struct array with the following fields:  k, a 1-by-M vector with the kappa estimate from each of the M training sessions.  kj, an M-by-6 vector with the pattern-specific kappa estimate from each of the M training sessions. Each column corresponds to one the 6 unique pattern types: col1=Pause, col2=Asynchronous-breathing, col3=Movement artifact, col4=Synchronous-breathing, col5=Sigh, col6=Unknown.
 kstd, a 1-by-M vector with the estimated standard deviation of kappa for the M training sessions. RIPScore estimates standard deviations using the bootstrap method [4].  kjstd, an M-by-6 vector with the estimates of pattern-specific standard deviation of kappa for each of the M training sessions. RIPScore estimates standard deviations using the bootstrap method. Each column corresponds to one the 6 unique pattern types: col1=Pause, col2=Asynchronous-breathing, col3=Movement artifact, col4=Synchronous-breathing, col5=Sigh, col6=Unknown. from each training session. The first row corresponds to level 1 and the second to level 2. Each column corresponds to each session. Each cell contains a 6-by-6 matrix where the rows correspond to the patterns assigned by the scorer and the columns to the actual pattern. Col1/row1=Pause, col2/row2=Asynchronous-breathing, col3/row3=Movement artifact, col4/row4=Synchronous-breathing, col5/row5=Sigh, col6/row6=Unknown. o EffectiveTrainingTime, a 1-by-2 cell array with the effective time (in seconds) required to complete the training session (see [1]

15/17
This is a file with the scoring details from a training session of scorer TEST. The name indicates that it corresponds to the Y th training session on level X. The file contains the variables described next.  SCORING: A struct with the detailed results from the manual analysis. It has the following fields: o NextScore, an index indicating the number of segments scored (S) plus 1. o Events, an S-by-4 matrix with the patterns assigned to each data segment. The columns are: (1) segment start time, (2) segment end time, (3) code of pattern assigned to the segment (pause=1, asynchronous-breathing=2, movement artifact=3, synchronous-breathing=4, sigh=5, unknown=99), (4) timestamp. o Scorer, a string with the scorer ID. o Comments, a struct with comments that the scorer assigned to scored segments. It has the following fields:  start, a C-by-1 vector with the start time of the segment linked to the comment.

McCRIBS
McGill CardioRespiratory Infant Behavior Software RIP Respiratory Inductive Plethysmography