Dataset

The TEE images were obtained from dataset of Intra-Cardiac Arrest Research Using Sonography (ICARUS) registered at Wonju Severance Christian Hospital, Wonju, Republic of Korea. When patients with cardiac arrest arrived at the emergency department, CPR team performed advanced cardiac life support (ACLS) according to the CPR guideline. 20 Intra-arrest TEE was attempted if an emergency physician who was capable of performing TEE was available. An emergency physician, who was not participated in resuscitation performed TEE if return of spontaneous circulation was not obtained after approximately 10 min of ACLS. A multiplane TEE machine (EPIQ7; Philips Medical Systems, Andover, MA) was used to conduct TEE. Once TEE transducer (X8-2t 2–8 MHz transducer, Philips Medical System) was introduced into the esophagus, images were obtained according with the intra-arrest protocol including 4-chamber view, long axis view, 2-chamber view, aortic valve short axis view, ascending aorta short axis view, ascending aorta long axis view at the mid-esophageal level and descending aorta long axis view and aortic arch long axis view at the upper esophageal level. After quick observation of the heart and great vessels, the transducer was positioned at the mid-esophageal level to monitor cardiac movement and assist resuscitation measures. Since the TEE images were obtained during resuscitation, the heart was swung due to chest compression, which was different from normal beating heart. A part of the heart could not be evaluated during compression systole or diastole in 4-chamber view or 2-chamber view. Therefore, mid-esophageal long axis views were selected to precede image segmentation. The study design was approved by the Institutional Review Board of Wonju Severance Christian Hospital, Yonsei University (Approval number: CR317048). The requirement for obtaining informed consent was waived by the Institutional Review Board.

Nine patients’ data was collected; the patients were categorized as follows: six patients in a training dataset, two in a validation dataset, and one in a test dataset. To proceed with image segmentation using deep learning, there must be a mask, which is the result of image segmentation. Since masks for deep learning are usually made by humans, manufacturing masks take a lot of time. Moreover, making accurate masks for LV in TEE images in contact with severe noise is difficult without a clinician’s opinion. For this reason, it is difficult to create masks for all frames; therefore, we organized the data into frames corresponding to end-systole and end-diastole for every patient. Two experienced experts manually contoured to make ground truth for deep learning. When configuring data for deep learning, five-fold cross-validation was performed to use all datasets for evaluation and to prevent overfitting to a specific dataset. The distribution of the training data and validation data is shown in S1 and S2 Tables.

Data augmentation

Data augmentation is a technique used to increase the amount of data required. There are many methods to perform data augmentation, but the essence is to artificially change the image. the number of training data per fold was small for deep learning. Since there was a high possibility of overfitting, we applied augmentation methods to our data. Excessive deformation by data augmentation can fade the meaning of the data; therefore, methods suitable for ultrasound images were applied. The applied data augmentation methods include random contrast control, random gamma control, horizontal flip, grid distortion, three-dimensional rotation, etc.

Deep neural network architecture

We designed a deep neural network suitable for our data by appropriately placing residual blocks, residual feature aggregation modules, squeeze and excitation blocks, and attention blocks [20–23]. The proposed model is an encoder-decoder structure based on U-Net. The encoder step consists of four stages, each of which is composed of a squeeze and excitation block and two residual blocks. Residual blocks placed throughout the model prevent the vanishing gradient problem that occurs as the model goes deeper with residual mapping and the extraction of strong local features. The second residual block was used to adjust the stride to reduce the size of the feature map by half, which is called down-sampling. This not only reduces the number of parameters and computational resources, but also extracts spatial features. The extracted feature maps propagate to the next step while recalibrating the features as they pass the squeeze and excitation blocks. The attention block of the decoding stage gathers the features of the encoder and decoder steps to focus on the important features of the feature maps. Up-sampling was performed using a trainable convolution transpose filter, and spatial information was reflected as much as possible by collecting the features of the residual block using the residual feature aggregation module. In general, the process of down-sampling and up-sampling through the encoder and decoder can cause loss of information in the data. However, the skip connection that transfers the important information of the encoder step directly to the decoder minimizes the loss of information. Finally, the segmentation map of the TEE image was generated by propagating through the last convolution layer. The model that proposed in this study is depicted in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed model architecture.

The gray arrow indicates the skip connection that connects the encoder and decoder. The blue arrow indicates the encoding path and the red arrow indicates the decoding path. LV, left ventricle; TEE, transesophageal echocardiography.

https://doi.org/10.1371/journal.pone.0280485.g002

Residual feature aggregation module

The residual feature aggregation (RFA) module consists of three residual blocks and three enhanced spatial attention (ESA) blocks. These residual blocks extract powerful features while retaining the original feature map using residual mapping. As shown in S1 Fig, the RFA module aggregates the local residual features of each residual block and reuses features to minimize the loss of information. The feature map passed through the RFA module is propagated to the last residual block and concatenated before the convolution layer. In addition, it maintains information about the original information by adding itself regardless of the residual mapping in each residual block. The ESA block behind the residual block utilizes a wide receptive field to focus on more important spatial features, and the features are multiplied by the feature map. This allows the model to learn more particular features, further enhancing model performance. The detailed structures of the residual block and the ESA block are shown in S2 Fig.

Squeeze and excitation block

The squeeze and excitation block is one of the attention methods designed to enhance the representational power of convolutional features. This block is divided into two stages: squeezing and excitation. In the squeeze step, global spatial information is compressed into one channel through an average operation to extract important information for each channel. Then, in the excitation step, the compressed information from the previous step is recalibrated to highlight the important information with the channel-wise operation. The proposed model uses this block to focus on more important information when extracting features from the encoder step.

Attention block

The attention method is primarily used to process natural language. Recently, this method has also been applied to image processing, such as image segmentation. The mechanism of the attention method in this model involves a second step. The first step compresses the important features of both the encoder and decoder. The second step is to resample the compressed information and multiply it to the original data to make the data more meaningful.

Experiment detail

The proposed model was developed based on a deep learning framework called Pytorch. The original data size was resized from 600 × 800 to 256 × 256 to proceed with the learning. The batch size was set to 8, the learning rate was set to 0.00005, and the Adam optimizer was used to optimize the model. We applied a loss function, which is a mixture of binary cross-entropy loss, focal loss, and dice coefficient loss, unlike the original U-Net.

Statistical analysis

In this study, the experiment was conducted with a different distribution of data through five-fold cross-validation. The metrics used for performance evaluation are the dice coefficient [24], intersection of union (IoU), recall, and precision. The dice coefficient and IoU are metrics that compare ground truth and predict results to indicate similarity; therefore, they are commonly used in image segmentation [25–27]. These metrics were statistically analyzed using 95% confidence intervals (CIs). Since the resulting metrics do not satisfy normality and homoscedasticity (S3 and S4 Figs), we implemented a Mann-Whitney U test between the proposed model and U-Net, under a significance level of 0.05. The details of metrics are described in S3 Table.