2.1 OpenHSV system

We are using the OpenHSV platform introduced in [25]. Subjects are typically examined using a rigid endoscope equipped with a high-speed camera (IDT CCmini-1540) running at 4,000 fps. Illumination is granted by a high-power LED light source (Storz 300 W LED). Each recording is at least 1,000 frames long and contains synchronously acquired video and audio data. We further record patient metadata consisting of the patient’s age, gender, and condition. For each video, we save two files encoded with the h.264 codec: (1) lossy compression using common settings for video and (2) lossless compressed data to recover the original recorded raw data. For lossy compression, we use the libx264 codec, the yuv420p pixel format and set the quality to 5 resulting in a varying bitrate. For lossless compression, we used the libx264rgb codec, the rgb24 pixel format, set the ‘-crf’ flag to 0 and used the ultrafast preset.

2.2 Subjects

We assessed a total of 583 recordings acquired between November 2019 and November 2021. All recordings were acquired in line with local regulations (Ethikkommission Medizinische Fakultät, Friedrich-Alexander-University Erlangen-Nürnberg, #290_15). All data was analyzed retrospectively. We first selected only those recordings that featured a complete set of data, such that 295 recordings remained. Next, we manually investigated the data quality. We ranked each video on an ordinary scale: 0 (insufficient), 1 (okay), 2 (excellent). Videos ranked 0 were showing insufficient data quality, such as non-visible glottis or foggy videos, and were discarded. Finally, 267 recordings from 202 unique patients remained and were subjected to further analyses (Fig 2A). We report the frequency of recordings across the last two years in Fig 2B. Additionally, COVID-19 cases for Germany were provided as a reference for how data generation was affected by lock-downs. The number of recordings were higher before the first lock-down, but their fluctuation remained constant when the general clinical activity was restored. The age distribution of the patients was largely spread (Fig 2C), where the mean age is 47.2 ± 20.2 (std) years. The reported gender for the analyzed subjects is 31.2% male, 66.8% female and 2.0% had no further specified gender. The average file size for lossless and lossy recordings was 2.06 ± 1.27 GB and 8.97 ± 5.16 MB, respectively (see Fig 2D). As the lossy files have an around 235-times lower file size than the lossless compressed counterparts, we investigated in this study if the lossy compression has an impact on the segmentation performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Acquired OpenHSV data overview.

A: From initial 583 recordings, we excluded datasets that are missing any kind of data. From these, we excluded ones with insufficient data quality, such as occluded glottis. B: Frequency of recordings across the two year time frame is shown in black. Each bin corresponds to the recordings acquired in this week, denoted as year/calender week. The Germany COVID-19 cases are shown as a 7-day-average in gray. C: Age distribution across selected recordings from A. D: Comparison of file size between lossy and lossless compression.

https://doi.org/10.1371/journal.pone.0266989.g002

2.3 Image quality assessment

As we saved the data in two different compression modes, namely lossy and lossless, we evaluated if there are any compression artifacts. First, we investigated the dynamic range of the image, where 0 is no dynamic range and 1 is full dynamic range by computing the normalized absolute difference of the minimum and maximum pixel value. We determined the no-reference, completely blind image quality score “Natural Image Quality Evaluator” (NIQE) [26] to provide an unbiased score for objective image quality assessment. We further assessed full reference metrics, such as peak signal to noise ratio (PSNR) and structural similarity index metric (SSIM) [27] to compare lossy compression against the lossless stored data. PSNR can be described as follows: (1) where max(I) is the maximum value corresponding to the highest pixel value with respect to images, and MSE is the mean squared error between two images I and K, defined as follows: (2)

SSIM is defined as follows: (3) where l(I, K) is a luminance comparison function, c(I, K) is a contrast comparison function and s(I, K) is a structure comparison function between image I and image K. α > 0, β > 0 and γ > 0 are weighting factors of each of the metrics. The respective details are described elsewhere [27].

We use respective implementations of the metrics in scikit-video (NIQE) and scikit-image (PSNR, SSIM).

2.4 Deep neural network

The OpenHSV system is shipped with a deep neural network (DNN) based on the U-Net architecture [20], that was optimized for clinical use as described previously [9]. The used DNN is openly available on the OpenHSV Github account at https://github.com/anki-xyz/openhsv. The training process is described in [25]. Briefly, the DNN is setup in TensorFlow 2.2 and the high-level Keras package. The DNN was pretrained in a supervised fashion on the full training dataset (55,750 images) of the open benchmark for automatic glottis segmentation (BAGLS, [10]). The pretrained network has never been exposed to OpenHSV data during training.

3.1 Recording quality is consistent across time

To evaluate the performance of the segmentation DNN, we first assessed the overall image quality for both, lossless and lossy recordings, as this is a major confounding source for segmentation success. First, we determined the maximum dynamic range of each given image. We found that the maximum dynamic range is constantly high for both, lossless and lossy recordings (Fig 3A, left panel), however, the dynamic range is significantly lower for lossy recordings as for lossless recordings (paired Student’s t-test, p <0.01, Fig 3A, right panel). Using a complete blind, non-reference metric, the NIQE score [26], we could show that lossless recordings have lower, therefore better NIQE scores until beginning of 2021 (Fig 3B), afterwards the NIQE scores were highly overlapping. In [25], a mean NIQE for the OpenHSV system of 13.19 was reported, showing that image quality has been consistent since the introduction of the system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Image quality is relatively stable across time.

A: Dynamic range per recording. Lossless compressed data in blue, lossy in orange. Boxplots show the 95-percentile of the data. B: NIQE score for lossy (orange) and lossless (blue) compressed videos across time. C: Peak-Signal-To-Noise-Ratio (PSNR) across time. D: Structural Similarity Index Measure (SSIM) across time.

https://doi.org/10.1371/journal.pone.0266989.g003

To compare images retrieved from lossy recordings to their lossless counterparts, we relied on two reference metrics, peak-signal-to-noise-ratio (PSNR) and structural similarity index measure (SSIM). PSNR has a constant high value above 90 dB, suggesting a high-quality compression (Fig 3C). In contrast to PSNR, SSIM also takes the perceptual change in structural information into account. In Fig 3D we show that the lossy compressed videos nevertheless are in almost perfect agreement with the lossless reference.

In summary, we found that the image quality has single outliers, but overall is highly consistent across time. Further, high PSNR and SSIM values indicate an overall accurate conversion from lossless to lossy image content and high quality in lossy compressed videos.

3.2 Segmentation performance is not affected by lossy data compression

To further evaluate the performance of the segmentation DNN, we manually annotated the glottal area in the first one hundred frames of ten randomly selected videos, which were manually rated as quality 2 (excellent). Using this ground truth and the predicted segmentation masks by the DNN, we computed the Intersection over Union (IoU) for each frame across all videos. We found that lossy and lossless saved videos achieve a similar performance with a median IoU of 0.756/0.742 and 0.770/0.768 for segmentation masks computed with and without ROI, respectively (Fig 4B). These IoU values are comparable to other works [9], where IoU scores between 0.741 and 0.769 were achieved, and are sufficient for clinical reliability. The use of an ROI enhances the segmentation speed as smaller images are used, however, this leads to worse results. We hypothesize that this is caused due to the loss of global spatial information. We further mined the computed IoU scores to determine why very low IoU scores are obtained. Fig 4C shows that the low IoU scores emerge with a small segmented area, i.e. when the glottis is closed. This is in line with previous reports [10], and has a negligible effect on the data analysis. We next investigated if any configuration has an impact on the clinically relevant glottal area waveform (GAW) signal. We were able to confirm that all combinations despite their deviation in the IoU score have little to no effect on the GAW, as they do not deviate (S1A Fig in S1 File) and almost perfectly correlate with each other (S1B Fig in S1 File), which is important for downstream computation of quantitative parameters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Lossy compression does not affect segmentation performance.

A: Input data for DNN inference. Either full-frame (magenta) or ROI-constrained (green) image data were used for inference. Image data were stored either lossless or lossy compressed as .mp4-files (see Methods). B: Intersection over Union (IoU) score per configuration shown in A. C: Dependency of IoU scores on the segmented glottal area. D: Exemplary glottal area waveform for full frame (w/o ROI) and ROI-based (w/ ROI) segmentations. Lossy and lossless compressed data are plotted in orange and blue, respectively.

https://doi.org/10.1371/journal.pone.0266989.g004

To further emphasize the practical relevance of our model, we measured the inference time on a CPU (Intel Xeon Silver 4110) and a GPU (NVIDIA GeForce RTX 2080 Ti) with and without ROI. For this, we calculated the average ROI size of all annotated ROIs, resulting in an input image size of 224x160 px. Predicting 100 images with an image size of 1024x1024 px on CPU and GPU takes 38.55 ± 0.35 seconds and 1.73 ± 0.08 seconds, respectively. Using the average ROI frame size on CPU and GPU results in a reduction of the inference time to 1.79 ± 0.23 seconds and 0.12 ± 0.009 seconds, respectively. Even on a CPU the inference is quite fast when using ROIs instead of full frames. This underpins the possibility of using our model in a clinical environment where computing resources are limited.

3.3 Continual training for DNN fine-tuning improves performance

Despite the fact that we gained mostly successful and accurate segmentations, we manually reviewed all segmentation masks and found two common issues in a minority of videos (9%): (1) artifacts in the segmentation masks and (2) empty segmentation masks (S2A Fig in S1 File). We hypothesized that continuous integration of new, system-specific data using continual training [30, 31] is increasing the DNN performance. Here, we evaluated two strategies for integrating new data (Fig 5A). We either used a fixed time period or fixed quantities of videos. We used artifact-free, full-frame, lossy compressed videos for continual training, as full frames resulted in higher IoU scores (Fig 4A). We found that all strategies were able to reduce the number of artifacts after only the first iteration of continual training and removed artifacts on average by 38–48% (Fig 5B). Additionally, the more data is used for each training step, the higher the impact on artifact removal, and that a fixed quantity is preferable to a fixed time period (Fig 5B). In particular, when determining the median, i.e. what is the reduction in 50% of all cases, we found that the fixed amount of 40 videos has the best overall performance of reducing the artifacts of by 48%. In general, we can confirm that a fixed data amount seems to be preferable to a fixed time frame, as the latter also shows an unstable performance behavior across continual learning epochs (S1 File). To further quantify this effect more objectively, we annotated the first 30 frames of each artifact video to compute IoU scores after each continual learning step (Fig 5C and 5D). Each strategy was able to increase the IoU score already after the first continual learning iteration, whereas a fixed data amount has a more stable and constant performance (Fig 5C) compared to a fixed time interval (Fig 5D), similar to our observations with manually scored artifacts (S1 File). In addition, we determined the performance of several other image segmentation metrics after each continual learning iteration. We found consistent results across assessed metrics compared to the IoU score reported in Fig 5C, strengthening our continual learning scheme (S1 File). In contrast, the fixed-time-period strategy shows no apparent during the first COVID-19 wave, which again suggests that the use of fixed quantities is preferable to fixed time periods. Taken together, our data shows that the segmentation DNN is not only able to quickly adapt to new data, but also that continual learning is an important feature in using DNNs in a clinical context.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Continual training alleviates segmentation artifacts.

A: Continual training strategies. B: Reduction of artifacts depending on continual learning strategy. C: IoU score of the first ten frames of the evaluated data after continual learning with a fixed amount of videos. D: Same as in C, but with a fixed time interval.

https://doi.org/10.1371/journal.pone.0266989.g005

4.1 Acquisition circumstances as potential confounders

We could show that the analyzed data has a relatively high image quality throughout the analyzed time frame (Fig 3). However, other systems have been shown to suffer from low lighting, which can be rescued with Deep Learning methods [32]. As the data acquisition is a manual procedure, the motion of the patient or the examiner can be a confounder. Motion correction techniques were proposed [33, 34] that can be used as a pre-processing step that have not been employed in this analysis.

4.2 Comparison to other glottis segmentation platforms

In this study, we investigated the performance of a single DNN. Offline image analysis platforms, such as the Glottis Analysis Tools [34] (GAT), can further serve as a reference for segmentation performance and may allow segmentations of higher quality. In comparison, the analyzed OpenHSV DNN has similar performance as the smallest GAT neural network, however, larger and more elaborate networks have superior performance (S1 File). Notably, this effect is largely compensated by our continual training scheme (Fig 5). Nevertheless, the average IoUs obtained in this study are in an acceptable range (0.742–0.770) that do not impact the clinical soundness of downstream quantitative parameter computation as shown previously [9, 25].

4.3 Continual learning strategies

Our proposed continuous integration of more data is straightforward and effortless. We have seen that using a fixed data mount is beneficial to a fixed time interval, which is maybe due to the irregular patient stream (Fig 2). We further observe a decline in IoU scores across data and time (Fig 5C and 5D) and a tendency to more artifacts after integrating a large amount of data (Fig 5B, S1 File). To counteract catastrophic forgetting [30], a common problem in continual learning schemes, additional precautions can be introduced, such as integrating the BAGLS dataset [10] in the continual learning training dataset or testing on previous artifact-free images if the current model performs better than before. Combining both aforementioned examples would incorporate external and internal regularization factors that are maybe beneficial in a more elaborate continual learning paradigm. In addition, we have not investigated how more sophisticated human-in-the-loop strategies [35], such as manual segmentation and retraining, perform. With wider adoption of OpenHSV, we also believe that federated learning techniques [36] will further boost the DNN segmentation performance.