2.1 Patient image data and ground truth generation

We used 51 clinical CT scans from the LUNA16 challenge [61], which is part of The Cancer Imaging Archive (TCIA) [62]. The data was accessed on the 2nd of March 2020 and is fully anonymized. The scans were acquired for lung cancer nodule detection which does not depend on inhalation state, therefore no information on breath-hold procedures was reported [61]. The scans used were chosen randomly by Tang et al. [14], and they provided radiologist-verified lung lobe segmentations online. We created airway segmentations for the same scans using a semi-automatic region-growing approach, based on work by Nardelli et al. [63]. These segmentations have been independently verified by a radiologist with 9 years experience. We excluded 13 samples from the LUNA16 dataset, as the segmentations only included the trachea and first bifurcation. The remaining 38 samples were used for training the models discussed in the following sections.

To validate our modelling approach, we compared to anonymized experimental data of radiopharmaceutical aerosol deposition from Conway et al. [59, 60], which we refer to throughout as the ‘Southampton/Air Liquide dataset’ (accessed on 21st of May 2021). They performed experiments with inhaled aerosol in six healthy and six asthmatic subjects. Patients sat in an erect (sitting) position while aerosol was delivered through an AKITA nebuliser. The AKITA nebuliser is breath-actuated to release drug only during inhalation, and a controlled (constant) inhalation profile to optimise deposition [64]. Deposition was imaged with single photon emission computed tomography (SPECT) and combined with low resolution X-ray CT captured in the same machine (SPECT-CT), which relates deposition images to anatomical structure of the lungs [65]. To relate deposition measurements to airway morphology, high resolution CT was acquired for each patient on a different machine on a different day [59, 66]. Full information on scanning equipment and protocol are provided elsewhere [59, 66], and only key points related to our study are discussed here. High resolution CT images were acquired in supine position, which is known to decrease lung volume compared to erect position [67]. Additionally, Conway et al. discussed imaging issues in the upper airway [59]. In half of the images acquired, the epiglottis was completely closed, meaning there was no path for air to pass from the mouth to the lung airways. This may have occurred since the images were taken during exhalation, where the glottis area is at its narrowest [68, 69]. Furthermore, motion artefacts may have blurred the already narrow glottis to appear fully closed. To allow for air and drug particles to pass from the mouth to the lung, we manually cleaned up segmentations in these problematic cases (Supplementary Material Section 1 in S1 File).

2.2 Shape reconstruction from a chest X-ray image

To reconstruct 3D lung and airway geometries from an X-ray image, we created a statistical shape and appearance model (SSAM) based on ground truth airway and lung lobe segmentations. The appearance is based on digitally reconstructed radiographs (DRRs) derived from the patient CT data (described below). The DRRs were also used for testing how well the SSAM fits to an X-ray. As there was no available dataset of paired chest CT scans and chest X-ray images for validation of our 2D-3D reconstruction, it is common to use DRRs for testing reconstruction methods [31, 32]. Below we describe the process to create a SSAM from patient CT data and segmentations described above (Section 2.1).

2.2.2 Statistical shape and appearance modelling.

Once a set of landmarks and DRRs were extracted from the LUNA16 dataset (Section 2.2.1), all sets of landmarks were aligned to zero mean and isotropic scaling was applied such that the coordinates had zero mean and unit standard deviation. Landmark gray values were also normalised to zero mean and unit standard deviation per sample. We then create a SSAM to describe and parameterise the covariance of the lung and airway shape-appearance data in the LUNA16 dataset. The SSAM parameters are then iteratively adjusted based on how well the lung and airway shape and appearance agrees with the X-ray image (Fig 1a and 1b).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic overview of image-processing framework.

Chest X-ray image(s) are provided as an input (a), where the shape is identified by our SSAM by iteratively adapting lung outline and gray-value based on a database of known lung shapes (b-c). The SSAM reconstruction can then be used to generate a surface mesh of lungs with a full airway tree (c). Alternatively, a 3D CT scan may be provided as an input (d) to our trained CNN architecture (e). The CNN returns a segmented labelmap for lungs and airways (f), which can also be used to generate the full respiratory domain (c).

https://doi.org/10.1371/journal.pone.0297437.g001

The SSAM uses principal component analysis (PCA) to create a model of the variation in the shape and the gray-values of the corresponding chest X-ray image(s). The state vector containing shape and gray-value at each landmark, i, is , where x_i is the landmark coordinate, is the landmark gray-value for the first projection, N_XR is the number of X-ray images per patient (each with its own g). In this study we use N_XR = 1 and N_XR = 2. In both cases, a frontal X-ray is used (Fig 1a). In tests with N_XR = 2, we also provided a lateral view X-ray image. The state vector containing all landmarks is a column vector, , where N_L is the number of landmarks. Therefore, z has a size of (3 + N_XR)N_L. A new state vector can be approximated by the linear function (1) where is the mean of the state vector (averaged over all samples in the training dataset). Φ is a (3 + N_XR)N_L × N_m matrix, giving the main modes of variation of the training set, where N_m is the number of modes. is the variance described by mode m. b_m is the shape parameter for mode m, where Fig 1(b) shows the effect of varying the weighting of b for the first two modes. By using PCA, the complexity of the model can be reduced by including only the number of modes, N_m, that are required to capture a specified amount of variance in the modelled shape. In this study, 90% of the population variance is included in the model (N_m = 25, Fig 2a). The SSAM was created using our Python library ‘pyssam’ for statistical shape and appearance modelling [73].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Assessment of sensitivity of SSAM metrics to number of modes included in the model.

Panels show (a) explained variance based on number of modes, (b) reconstrucion error for shapes in training set, (c) generalisation error for unseen shapes. When the model describes 90% of the population variance (indicated by blue dashed line), the generalisation error in adapting the model to unseen shapes is saturated. Tests were performed using the final 38 samples from the LUNA16 dataset.

https://doi.org/10.1371/journal.pone.0297437.g002

The accuracy obtained by the SSAM with N_m modes can be assessed in terms of its ability to reconstruct a shape in the training dataset (reconstruction error). More importantly, the SSAM can be assessed by its ability to generalise and reconstruct an unseen shape (not in the training set), which is known as generalisation error. When the shape model contains 100% of the training population variance (Fig 2a), the reconstruction error should tend to zero as all information required to reconstruct the shape is contained within the model (Fig 2b). We assessed reconstruction and generalisation error with leave-one-out testing, which was averaged over the 30 loops of the entire training set. The mean absolute error is presented, which is expressed as a percentage of the lung bounding-box size. We observed that N_m = 25 is a suitable choice to reduce the SSAM dimensionality, as the generalisation error did not increase further by including additional modes (Fig 2c).

The shape parameters, b, can then be optimised with respect to the fit of the SSAM outline () and the similarity to the training set (). The appearance model optimises the shape by minimising the gray-value loss (). Gray-value loss is calculated as the difference between the image gray-value at the landmark location and the expected gray-value from Eq 1 [21, 74, 75]. We also used the anatomical shadow loss () for airways as proposed by [29]. Anatomical shadow compares local gray-values inside a small control area inside and outside of the projected airway surface, which can enhance airway contrast compared to dense overlapping structures such as the spinal column. Therefore, the optimal isotropic scaling, translation and shape parameters are found by minimising the equation (2) where C is a coefficient that controls the weight of each fitting term.

The fit of the projected landmarks outline (‘silhouette’ landmarks) to the lung outline on the X-ray image is found by (3) where N_SL is the number of silhouette landmarks, D_i is a function based on the distance between each silhouette landmark and the nearest X-ray image outline point. Silhouette landmarks were defined as landmarks where the nearest surface mesh point is shared by two faces, with one face normal pointing towards and another away from the image source. For example, for an anterior-posterior (frontal) chest X-ray image, the projection is normal to the y-direction and therefore the two faces nearest to the silhouette landmark will have positive and negative y components in the normal vector. The silhouette landmarks represent the outline of the shape normal to the projection [20].

Baka et al. [20] normalised the outline distance to bounds of [0, 1] by (4) where C_dist = 5 was used [20]. is not evaluated for the airways as the airway outline extracted from a Canny edge map is weak and unreliable due to overlapping denser regions on the X-ray image.

Improbable shapes (relative to our training set) are penalised by the Mahalanobis distance between the created shape and the mean shape [20, 70] calculated by (5) where N_L is the total number of landmarks, is the inverse covariance matrix between x_i and .

We use the appearance model to minimise the difference between the modelled gray-value (g_model) and the gray-value at the landmark location (g_target). This is found by (6) Finally, the anatomical shadow loss is found for each silhouette landmark [29]. This is done by creating two regions-of-interest (ROIs) at some distance ±s_AS normal to the border of the silhouette landmarks. Each ROI is a circle with radius r_AS, and the gray-value ‘inside’ and ‘outside’ of the airway projection are compared as (7) We found the optimal parameters to be s_AS = 20 pixels and r_AS = 14 pixels.

Shape parameters were optimised using the NGO (Nevergrad optimiser) algorithm [76] in the Nevergrad Python library [77]. This is a competence map method, which adjusts the optimisation algorithm based on the number of optimisation iterations and number of parameters optimised. We chose this method as it can therefore adjust to more modes of variation as our dataset grows in future studies. We found this gradient-free optimisation to show good robustness against local minima. The shape parameters for each mode were initialised to b_m = 0, and were bound to b_m ± 3.

SSAM hyperparameters in the loss function (Eq 2) which produced the lowest maximum error in reconstructed lung space volume were found through Gaussian process regression [78, 79], also known as Kriging [80]. We performed 100 optimisation iterations, using a subset of 11 randomly selected samples from the LUNA16 dataset. This gave the optimal model parameters to be C_fit = 0.795, C_g = 0.687 and C_prior = 4.4 × 10⁻⁴. The anatomical shadow loss coefficient was later found through a simple grid search as C_AS = 0.2.

Landmarks obtained from fitting the SSAM to an unseen shape (called ‘new landmarks’) were converted to a surface by morphing a template mesh and corresponding template landmarks to the ‘new’ landmarks [81]. New mesh points (p_i,new) are created from the new landmarks (x_i,new), template mesh points (p_i,template) and template landmarks (x_j,template) by (8) where k is a kernel used for morphing the mesh and w_j represents the weights controlling how much each landmark point is morphed. The weights are computed by solving (8) for w_j and setting p_i,new = x_j,new. We used a surface morphing algorithm [81] with a Gaussian kernel, as we found it to produce a smoother deformation when morphing. The kernel is k(a, b) = exp(‖a − b‖/2σ²), where σ = 0.3 [82].

2.3 Segmentation from volumetric CT

To segment lung lobes and airways from volumetric CT scans (Fig 1d), we used convolutional neural networks [15] such as the U-Net shown in Fig 1e [83, 84]. We implemented the U-Net in PyTorch [85]. The U-Net architecture was detailed by Ronneberger et al. [83]. For lung segmentation we used a pre-trained U-Net [12]. For segmenting the airways we trained a U-Net using segmentations from the LUNA16 challenge (Fig 1e). Prior to training, all images and labelmaps were cropped to the extent of the airway labelmap to conserve memory and allow us to segment the entire 3D image, instead of slice-by-slice [12], or sliding-box approaches [86]. With the aim of lowering memory consumption, we also trained an ENet (‘efficient neural network’) [87]. The ENet has approximately 15 times fewer parameters than the U-Net architecture [88], which creates lower memory consumption during training and faster processing time for 3D images [87, 89].

The loss function used for training all CNNs was [14, 90], given as (9) where is the DICE loss and is the focal loss. The DICE loss maximises the total intersection between a modelled labelmap and the ground-truth labelmap during training, and is found by (10) where p_ic is the probability of a voxel i belonging to class c as predicted by the CNN. g_ic is the ground truth value for the same voxel (which will be 1 or 0). This is averaged over all voxels in the image N and the total number of classes C.

The focal loss weights the loss more towards ‘harder to learn’ voxels with a focusing parameter, γ. Focal loss was developed for images with high class imbalances, such as airway segmentations, where we found the number of voxels representing the airways to be on average 63 times fewer than the number of background voxels. The focal loss was found by (11) where we set the focusing parameter γ = 5 based on a parametric study of training with γ set to 1, 3 and 5 (S4 Fig in S1 File). We used α = 1 as recommended by [14].

To train the U-Net and ENet, we used the Adam optimiser with a batch size equal to one to minimise memory consumption. Data was augmented with rotations of ±15°. We created 50 new augmented datasets per epoch. Training was stopped once the validation loss had not decreased further in the most recent 10 epochs. To train the U-Net and ENet models, we used a compute node with 756 GB of RAM on Heriot-Watt University’s high performance computing cluster ROCKS. Training took 3 days for the U-Net (42 epochs) and 5 hours for the ENet (45 epochs). The ENet DICE coefficient was lower than the U-Net (0.89 compared to 0.93, S5a Fig in S1 File). The ENet had a significantly improved inference time compared to the U-Net (ENet gave 5.7 times speedup, S5b Fig in S1 File).

2.4 Distal airway generation

To create the rest of the conducting airways not visible in CT or from our airway SSAM, we implemented a volume-filling airway generation algorithm [38, 39]. The algorithm generates a full conducting airway tree based on information from the upper airways such as branch length and diameter. The lung space volume was discretised into seed points with a uniform spacing found by Δ = (V_lung/N_T)^1/3 where N_T = 30, 000 is the number of terminal nodes in the conducting airway tree.

The algorithm for generating conducting airways can be found in detail elsewhere [38]. We will briefly cover it here. Seed points were first clustered based on their nearest terminal branch in the airway tree. The centroid of each cluster was found and a splitting-plane was defined based on the centroid and the node of the parent branch. This splitting-plane was used to divide each seed point cluster into two more clusters. A new branch was generated extending towards each seed point set by 40% of the distance to the centroid. When the seed point set had only one node, or the branch length was less than 2 mm, the branch is classed as a terminal bifurcation and the nearest seed point was deleted. This was repeated until no seed points remained in the tree. Once the tree was grown, a diameter was assigned to each branch segment as described in Supplementary Material Section 5 in S1 File. In our CPFD simulations, the image-based airways are used until generation 3 and airway tree generated from the volume-filling algorithm [38] is used to generate a surface from generations 3 to 6, similar to [91].

2.5 Deposition simulation configuration

To prepare the airway segmentations of the LUNA16 dataset for deposition simulations, we added a generic mouth-throat geometry obtained from a healthy adult patient [92] as done in [7] in cases where the upper airways were not imaged (chest CT, as is most common). This was done by scaling the surface mesh of the healthy patient’s upper airways such that the diameter at the interface between healthy patient upper airways and the patient’s trachea reconstructed was the same [7]. As jet-like flow structure produced in the upper airways has a key influence on the swirling flow observed in the trachea [93], it is important to include at least a generic representation of the upper airways in respiratory CPFD models [7, 93]. Images in the Southampton/Air Liquide dataset contained the lungs and the mouth-throat region (unlike the LUNA16 dataset). Therefore, we used these images directly and obtained segmentations of the mouth and throat by region-growing. We did not use the CNN for this as the mouth-throat region was not present in the training data. Also, this region can easily be segmented with region-growing [11]. The mouth-throat segmentation was then joined to the trachea directly.

The deposition simulations in this study were performed using OpenFOAM v6 [94] using our custom solver deepLungMPPICFoam [95]. Briefly summarised, we solved mass and momentum conservation equations for transport of an incompressible fluid (air). As described in [7], our computational fluid dynamics mesh had a spacing of Δ = 500 μm, which gave excellent agreement with flow through 3D printed airways [92]. The fluid turbulence modelling is discussed in detail in [96]. Particles were tracked in a Lagrangian approach by solving Newton’s equations of motion accounting for drag and gravity, as we showed in our previous study that particle-particle interactions are not influential [7]. Particles were deleted when they reached an outlet, and were given a ‘sticking’ condition when they hit the wall boundary. The number of particles tracked was 200,000 with diameter, d_p = 4 μm as used in our previous study [7]. In simulations with airways from the LUNA16 dataset, particles were released from the inlet over a period of 0.1 s with initial velocity as 10m/s to represent a metered-dose inhaler [97]. More information on simulation configuration for comparing to the Southampton/Air Liquide dataset is given later in this section.

To drive flow in the simulation, we model pressure at the outlet of the 3D domain using a lumped-parameter model based on a resistance-compliance circuit [51]. In this model, the pleural pressure required to drive flow during inhalation, p_d, is found by (12) where R_global is the global resistance of the circuit, C_global is the global capacitance, Q(t) is the flowrate as a function of time, V(t) is the volume of air inhaled as a function of time, and p_atm is the atmospheric pressure. As the flow is incompressible, there is no effect of atmospheric pressure and therefore it is dropped (p_atm = 0). In simulations of inhalation in airways from the LUNA16 dataset, we model tidal inhalation where the volume of air inhaled to the lung is [51] (13) where V_T is the tidal volume and T_B is the time for one breath. Oakes et al. estimated respiratory parameters such as R_global, C_global, V_T using empirical relationships which are often based on parameters such as age, weight, height, gender [51]. We used the adult parameters from [51], R_global = 7 × 10⁻³ cm H2Os/mL and C_global = 59mL/cm H2O. Pressure at the outlet is then calculated by solving the following equation at each outlet (subscript _(i)) at each time-step of our simulation: (14) where ϕ_(i) is the flowrate (also called ‘flux’) at the outlet computed from the fluid solver. Volume at the outlet is then found by integrating the flux in time. Local compliance and resistance at each outlet are estimated based on (15) where the A_(L) is the sum of areas of all outlets in each lung, L. α_(L) is the fraction of (static) volume of a lung to the combined volume of both lungs, which assumes that the volume change during inhalation is a factor of the lung’s volume only (not accounting for localised disease).

By defining Neumann boundary conditions for velocity at the inlets and outlets, the simulation can be unstable. Particularly, when the flow at the outlet is reversed (due to turbulence or local changes in geometry curvature) the flux at the face has a negative sign (creating positive pressure) and can cause numerical divergence [98, 99]. To prevent backflow instabilities, we (i) set maximum pressure and minimum flux at each outlet to zero, (ii) use limiters when calculating gradients at outlet faces to prevent small negative fluxes caused by interpolation errors. We also (iii) set the maximum Courant number to be 0.1 when the flow is developing, and Courant number is set to 0.2 when the flow is developed (quantified by 0.2 < |sin(2πt/T_B)|).

Simulations of the Southampton/Air Liquide dataset modelled inhalation from a breath-actuated AKITA nebuliser [59]. This provided a constant inhalation flowrate of 18L/min. Therefore, we assigned V(t) = V_Tt/T_inhale, where T_inhale is the time for one inhalation, T_inhale = T_B/2. In the experimental study, inhalation duration was varied between 2 and 3.3 s (shallow and deep breathing). However, Fleming et al. [60] observed no significant difference in regional deposition when changing from shallow to deep inhalation. As our model cannot account for particle motion in the deep lung, we cannot model particles re-entering the central airways from the deep lung during exhalation. Therefore, we used an inhalation duration of 2 s for our simulations to reduce simulation clock-time, since the longer inhalation and breath-hold would only influence particle motion in the deep lung (increasing time for deposition due to diffusion or settling). As the inhalation flowrate is steady and we could not model exhalation, we only modelled one inhalation. This was also done by de Backer et al. [100] to compare with SPECT-CT data, although they used a steady solver which cannot capture transient flow features.

In the Southampton/Air Liquide experimental study [59, 60], the particle diameters were d_p = 3.1 and 6.05 μm. To simplify our analysis, we used monodisperse particles as the experimental size distribution was reported as being narrow [59]. Simulated particles were released from a 5 mm disk in the AKITA mouthpiece, which represented the diameter of the tubing that delivers the particles from the nebuliser. Simulated particles were released with initial velocity of 0 m/s, since the particle timescale was small it would quickly adapt to the local fluid velocity in the mouthpiece (τ_p = 28 μs for d_p = 3.1 μm and τ_p = 108 μs for d_p = 6.05 μm).

Simulations of flow in airways from the LUNA16 dataset were performed using ROCKS with 28 CPUs each (Intel Xeon Silver), taking between one and three days. Simulations of flow in patient airways from the Southampton/Air Liquide dataset were performed on Oracle cloud with two bare metal compute nodes each with 36 CPUs (BM.optimised3.36, Intel Xeon Gold), taking between one and three days depending on the airway geometry.

2.6 Model evaluation

To evaluate our model, we first aimed to verify our 2D-3D reconstruction algorithm and CNN segmentation by comparing reconstructed lung and airway morphological properties using the LUNA16 and Southampton/Air Liquide datasets (Fig 3). The morphological analysis was done using the U-Net for the airways in the LUNA16 dataset, and the ENet for airways in the Southampton/Air Liquide dataset (due to larger memory requirements). For both datasets, the CNN segmentation of the lobes was done using a pretrained 2D U-Net [12].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Flowchart overview of the model evaluation procedure.

Model inputs are given in blue ellipses, image-processing methodology are in green boxes, outputs are in orange boxes. Blue arrows show model quantities validated against the ground truth. For the Southampton/Air Liquide dataset, ground truth deposition is obtained from SPECT and SSAM deposition is not computed (boxes with *).

https://doi.org/10.1371/journal.pone.0297437.g003

We then compared CFD deposition results from the 2D-3D reconstruction and the CNN segmentation against CFD simulations based on ground truth segmentations from the LUNA16 dataset (Section 2.1). Finally, we aimed to validate our drug deposition modelling by comparing deposition predictions in airways segmented from CT images with our CNN to in vivo SPECT deposition data described in Section 2.1 [59, 60]. As our SSAM did not contain training data from the mouth-throat region, we did not test the SSAM against the Southampton/Air Liquide deposition measurements.

The Southampton/Air Liquide experimental data contained measurements from deposition in six healthy and six asthmatic patients. Two experiments were performed for each patient, testing varying combinations of aerosol size, breathing pattern (shallow/deep) and inhaled gas (air compared to He/O2) giving 24 total measurements. We omitted cases with He/O2 inhaled gas (4/24 experiments). Images in three asthmatic patients had blockages in the upper airways due to tongue positioning, which meant we could not segment the entire upper airways and these patients were omitted (4 of remaining 20 experiments). Additionally, as our model does not account for particle transport in the deep lung or exhalation we omitted the ‘shallow’ inhalation in cases where shallow and deep inhalations were the independent variable compared (4 of remaining 16 cases). Therefore, we performed 12 simulations to evaluate our model against the available experimental data [59, 60].

For all CFD analyses, we evaluated deposition fraction computed as the number of particles depositing in a region (relative to the total number of inhaler particles). Deposition predictions were assessed based on regional drug deposition fraction (right and left lung) for both the LUNA16 and Southampton/Air Liquide datasets.

For CFD analyses of the LUNA16 dataset we also computed a local drug deposition concentration metric known as deposition enhancement factor (DEF) [101, 102]. We calculate this using the number of particles deposited within a fixed distance (1 mm, area A_conc = π(1 mm)²) of the central point of each wall face [103]. This is made relative to the global deposition by (16) where N_p,deposit(A) is the number of deposited particles in a surface area A. A_tot is the total airway surface area. We use this instead of the sum of the deposition efficiencies as the defined areas may overlap, and this therefore prevents particles being counted multiple times towards the global average. To minimise differences in total number of deposited particles or total surface area interfering with quantitative comparison between DEF in the ground truth and SSAM or CNN, we used the ground truth denominator from Eq 16 in DEF calculations in the SSAM and CNN [104].

3.1 Morphological assessment on LUNA16 dataset

We found our SSAM to yield a relative lung space volume error of 9.9% median and a 95th percentile of 34.3% on the LUNA16 dataset (Fig 4a and Table 1). The maximum error for the right lung was 33.64% and 41.66% for the left lung. The concordance correlation coefficient between SSAM lung volume and ground truth lung volume was 0.893 with two DRRs used for SSAM fitting (Fig 4b), and 0.884 for one DRR in the SSAM (S6a Fig in S1 File). The total volume error (averaged over left and right lungs) had a median value of 9.8% and a 95th percentile of 26.1%. Larger error in the left lung may be due to the presence of the heart interfering with the edge-map, which would cause a poor fit at the inner side of the lung. In contrast, the absolute relative error for the U-Net lung segmentation gave a median value of 1.35% and a 95th percentile of 3% (Fig 4). The U-Net is well-suited to segmenting lungs from CT, due to the high-contrast and well-resolved boundary between dark (air-filled) lung parenchyma and surrounding soft and hard-tissue. The SSAM lung reconstruction yielded a larger maximum error due to the poor contrast between lung and surrounding tissues on some radiographs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Evaluating SSAM error in lung space volume compared to ground truth lungs from the LUNA16 dataset.

(a) Relative error from SSAM and U-Net reconstructions of patients from LUNA16 dataset. Inset shows U-Net lung space volume error, where the U-Net results were produced using a pretrained U-Net [12]. (b) Identity plot comparing absolute lung space volumes from SSAM and ground truth. CCC is the concordance correlation coefficient.

https://doi.org/10.1371/journal.pone.0297437.g004

The median lung volume error for the U-Net and SSAM were increased on the Southampton/Air Liquide dataset (Table 1). The correlation coefficient obtained by the SSAM was 0.73, which is a reduction from the 0.88 on the LUNA16 dataset. This may be improved by including a more diverse dataset in SSAM training, as was done for the lung U-Net by Hofmanninger et al. [12]. The inclusion of more diverse training data may explain how the U-Net was able to achieve similar correlation coefficients on both datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of reported errors for various tests.

“UoS/AL” refers to the Southampton/Air Liquide dataset [59, 60]. Lung U-Net results were produced from a pretrained U-Net [12]. SSAM results shown are with two projections (N_XR = 2).

https://doi.org/10.1371/journal.pone.0297437.t001

In Fig 5 we compared the output SSAM landmarks against the input X-ray image and lung outline map in two outlier cases and the best case (in terms of lung space volume error, Fig 4). In both cases, we observed that the outer edge of the left lung has incorrectly fitted to a spurious edge introduced to the lung edge map. In the largest error case (Fig 5a), the lower corner of the SSAM landmark point cloud has fitted to the outline of the patient’s torso. In the case with the second largest error (Fig 5b), the left lung outer edge and right lung lower edge have mistakenly fitted to an edge in the soft tissue. In the lowest error case (Fig 5c), there is a spurious edge for the outline of the patient’s torso that slightly interferes with the outer edge of the patient’s right lung. However, this is located very near the lung itself as the patient has a narrow torso, which therefore appears to cause a minor influence on the SSAM fitting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Example of fitted SSAM compared to projection and lung outline for two largest lung space volume errors in the LUNA16 dataset.

Yellow markers are output landmarks from the SSAM, and black markers represent the X-ray edges detected by the Canny edge map described in Section 2.2.1. Panels show (a) the largest error, (b) second largest error and (c) the lowest error. In (a) the right lung error was 31% and left lung error was 42.6%. In (b) the right lung error was 15.5% and left lung error was 38.3%. In (c) the right lung error was 1.8% and left lung error was 0.06%.

https://doi.org/10.1371/journal.pone.0297437.g005

Diameter predictions did appear to be sensitive to outliers, as the maximum diameter error was above 30% in the main bronchi (Fig 6). We compare airway diameter error with a single DRR (anterior-posterior plane projection) and two DRRs (anterior-posterior and sagittal plane projections) on the LUNA16 dataset. The maximum trachea diameter error was 20.4% for one projection and 21.9% for two projections, which shows the trachea is less sensitive to outliers than the main bronchi due to higher visibility on an X-ray image. The absolute trachea diameter error had a median value of 8.5% and 7.8% for one and two projections, respectively. The main bronchi maximum error was 26.9% and 33.2% for one and two projections, and the upper quartile error was 13.6% and 12.4%. The concordance correlation coefficient was slightly higher for two projections than one (0.71 compared to 0.657, S7 Fig in S1 File). There was no statistically significant difference between the absolute error when comparing SSAM results with one and two projections (Wilcoxon signed-rank test p = 0.6 and p = 0.69 for the trachea and main bronchi, respectively). Similar to the lung volume error discussed above, the correlation coefficient of SSAM airway diameter predictions in the Southampton/Air Liquide dataset was lower than the LUNA16 dataset (0.71 compared to 0.53, Table 1). The CNN-based segmentations yielded similar performance on both datasets, as the median error was 5.5% and 6.5% on the LUNA16 (U-Net) and Southampton/Air Liquide (ENet) datasets, respectively. Correlation coefficients of airway diameter on both datasets were above 0.9 (Table 1), despite using an ENet on the Southampton/Air Liquide dataset which has fewer trainable parameters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of airway diameter errors predicted by our SSAM compared to the ground truth segmentations from the LUNA16 dataset.

We show the influence of including an additional projection on diameter error in the trachea and main bronchi.

https://doi.org/10.1371/journal.pone.0297437.g006

To evaluate the effect of imaging error propagation into distal airway diameters, we evaluated the diameter generated from the distal airway generation algorithm from the ground truth compared to the SSAM and CNN (Fig 7). Here, the ground truth refers to results from the airway generation algorithm using the central airway tree from the ground truth segmentations. CNN performance varied due to the dependence on the diameter in the central airways. The median absolute error in mean airway diameter per generation was 24.2% (similar to the SSAM with parent model, S8 Fig in S1 File). Additionally, the maximum absolute error was 77.6%, compared to 76% for the SSAM. The CNN airways show improved predictions of the diameter standard deviation (Fig 7), as the maximum error in standard deviation was 125.5% for the SSAM and 69.6% for the CNN.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Generated airway diameter for our image-processing modalities (SSAM and U-Net CNN) on two cases from the LUNA16 dataset.

Markers show the normalised mean diameter per generation. Error bars show ±1S.D. where S.D. is the standard deviation.

https://doi.org/10.1371/journal.pone.0297437.g007

3.2 Deposition analysis

In this section, we first compare deposition predictions from CPFD simulations in the LUNA16 dataset. We compare predicted deposition in ground truth airways with deposition predictions in airways obtained from SSAM and CNN segmentations. Following this, we then compare our simulated deposition from airways segmented by the CNN with experimental results from the Southampton/Air Liquide dataset.

Errors in deposition fraction from simulations of flow in airways from the LUNA16 dataset are shown in Fig 8. Comparing simulations in the CNN with the ground truth, the median absolute error was 5.3%, with a 95% confidence interval of −3.3 ± 12.1% (Fig 8a). The Pearson correlation coefficient between ground truth and CNN deposition results was 0.88. Deposition in the SSAM airway had a similar median absolute error as the CNN deposition (8%, Fig 8b). SSAM deposition error was more widely distributed, as the 95% confidence interval was −3.5 ± 20.5%. The Pearson correlation coefficient between deposition in the SSAM with the ground truth was 0.6. The error range for deposition in the SSAM was similar to diameter error (Fig 6) which had a confidence interval of −2.1 ± 21.7%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Bland-Altman plot comparing drug deposition in each lung predicted in ground truth airways, compared to airways reconstructed by (a) the U-Net, and (b) the SSAM.

Number of patients tested was 10 samples from the LUNA16 dataset. Each marker represents one lung, and each colour represents a different patient.

https://doi.org/10.1371/journal.pone.0297437.g008

To evaluate the effect of SSAM reconstruction on local deposition hotspots, we computed the deposition enhancement factor (DEF). The deposition hotspots are highly sensitive to local flow and geometry [7, 105], which itself is susceptible to geometric differences by image acquisition and image processing [106]. We can see that even for the case with the lowest deposition error (Fig 9), the deposition hotspots in the SSAM do not match the ground truth. Particularly, in the ground truth and CNN segmented airways (Fig 9a and 9c), the drug is noticably more dispersed than in the SSAM (Fig 9b). Less dispersed hotspots in the SSAM is likely due to additional turbulence generated in the CT-based airways which is not present in the SSAM, since it contains a reduced number of principal components. A lower number of principal components essentially simplifies the geometry and removes intricate details that may onset turbulence such as cartiligenous rings in the trachea [107, 108].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Deposition enhancement factor for simulations in airways constructed using the developed SSAM and U-Net, compared to simulations in a ground truth segmentation.

This represents one sample from the LUNA16 dataset. Columns show (left) ground truth, (middle) SSAM, and (right) U-Net segmented airways. Rows show (top) front and (bottom) rear view of the airways.

https://doi.org/10.1371/journal.pone.0297437.g009

Deposition hotspots in the CNN-based airways appear qualitatively similar to the ground truth (Fig 9a and 9c). The surface area with DEF > 1 at the first bifurcation in the ground truth is 17% larger than the CNN. However, the intensity of the hotspots is significantly different since the average DEF at the first bifurcation was 2.49 for the ground truth (maximum value 400), compared to 0.93 for the CNN (maximum value 225). At the first bifurcation, the mean DEF in our SSAM was 1.06 (maximum value 225). The surface area with DEF > 1 at the first bifurcation was 2.16x larger in the ground truth than the SSAM. Interestingly, the mean DEF at the first bifurcation in the SSAM and U-Net are similar (1.06 compared to 0.93 for SSAM and U-Net, respectively). However, there is a significant difference in the surface area covered by the deposition hotspots here (Fig 9).

Finally, we performed simulations of nebulised aerosol inhalation in healthy and asthmatic patient cases of the Southampton/Air Liquide dataset [59, 60]. The maximum deposition absolute error averaged over both lungs was 12.1%, which belonged to the case with most obstructed upper airway images shown in S1a Fig in S1 File (case H02 with small particles). The median, 75th percentile and maximum absolute error was 4.9%, 7.9% 13.1%, respectively. The concordance correlation coefficient between simulated and experimental lung deposition measurements was 0.43. The mouth-throat deposition had a concordance correlation coefficient of 0.525 (Fig 10b). The maximum mouth-throat deposition error was also in case H02 with small particles (27.2%). The linear best-fit line shows that the cases with high error in the mouth-throat region causes the distribution to depart from the 45° line (Fig 10a and 10b). We observed that mouth-throat deposition was best in the two cases where the glottis did not require manual cleanup (grey and yellow markers in Fig 10b). The error in these cases was 1.3% and 3.1%. The median and 75th percentile mouth-throat deposition errors were 5.7% and 11%, respectively. These results show that our models agree well with in vivo deposition data, particularly in the absence of imaging artefacts (Fig 10).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Comparison of simulated deposition against experimental measurements from the Southampton/Air Liquide dataset [59, 60].

Airways for simulations were obtained using the ENet CNN (due to larger memory requirements). Panels show (a) right and left lung deposition, (b) mouth-throat deposition, and (c) the partially corrected right and left lung deposition. Cases which are adjusted by correcting over-predicted mouth-throat deposition are shown with a blue border on the marker. CCC is the concordance correlation coefficient. The linear regression lines shown have (a) r² = 0.213, (b) r² = 0.36, (c) r² = 0.699. The cases requiring no manual cleanup of the glottis have grey and yellow markers.

https://doi.org/10.1371/journal.pone.0297437.g010

In an attempt to understand how our simulations would perform in a perfect case with no imaging difficulties in the upper airways, we partially corrected the simulation and experimental data by normalising the deposition fraction by 1 − DF_mouth, where DF_mouth is the deposition fraction in the mouth (Fig 10c). This makes deposition relative to number of particles entering the trachea, rather than the total number of particles entering the mouth. We apply this only to cases where the experimental lung deposition fraction was 5% larger than the simulated lung deposition (only outlier cases above the 45° line), as these were cases where the mouth-throat deposition was severly over-estimated (for example, green markers in Fig 10). Once these problematic cases were corrected, we see an improvement in the correlation between experimental and simulated data, as the concordance correlation coefficient increases from 0.432 to 0.810 (Fig 10a and 10c). This can be seen quantitatively by the fitted linear regression line nearly matching the 45° line. The resultant r² value is 3.3 times larger, at a value of r² = 0.699. The median, 75th percentile and maximum errors all decreased to 2.12%, 4.14% and 8.6%, respectively. This idealised test shows that the physical model is working well, but manually cleaning the segmentations around the glottis can introduce uncertainty that lowers correlation with experimental data (Fig 10a and 10c). Additionally, this idealised test points to an avenue for future development of image-processing tools for this region.

4.1 Limitations

A limitation of this study was the difficulty in capturing the upper airway anatomy (Supplementary material Section 1 in S1 File). The glottis can dilate and narrow during inhalation and exhalation, respectively [68, 69], which created difficulties in capturing CT images of the glottis [59]. Changes in the shape and area of the airways here can cause increased deposition in the glottis [125], but also downstream due to the flow separation at the expansion which creates an unsteady wake whose structure will likely vary based on patient anatomy [126–128]. Therefore, our attempt at cleaning up the glottis cannot perfectly represent instantaneous patient-specific flow patterns observed in vivo. Despite this, we observed good agreement with experimental lung and mouth-throat deposition measurements (Fig 10). An alternative approach to ours is a recent study on the same dataset by Grill et al. [116] who replaced the 3D mouth-throat geometry with an analytical model to predict deposition in the mouth-throat and resultant dosage entering the trachea. This also showed very good agreement with the experimental data. Future efforts should be focussed towards combining our developments with dynamic upper airway models [125, 128], to enable modelling the complex biomechanical fluid-structure interaction of respiration. However, to perform fluid-structure interaction simulations of these cases still requires a reliable baseline airway model, which requires manual cleanup of outlier cases shown in Fig 10. An improved approach to manually cleaning the glottis segmentations may be to determine the missing airway structure based on statistical shape models, as done for segmenting obstructed bronchi by Irving et al. [28]. Therefore, a template mesh could be morphed to approximate the glottis shape based on the shape of the nearby upstream and downstream airways (the throat and trachea). Combining this improved segmentation cleanup approach with dynamic mesh simulations in the upper airways is likely to produce a robust approach for modelling upper airway dynamics.

An additional limitation is the use of a healthy patient mouth-throat geometry [92] as a representative generic geometry for the upper airways in the LUNA16 dataset. Due to unavailable imaging data above the trachea in most clinical CT datasets, it was not possible to segment this from CT with our CNN, or include it as part of our SSAM to reconstruct from X-ray images. This of course limits the understanding of how well our imaging approach can capture the upper airways and flow patterns specific to a patient. The effect of this would be negligible on the CNN segmentation, as the upper airways are easily segmented with a region-growing approach [11]. It is unclear how well the SSAM could reconstruct the mouth-throat from a patient X-ray image, as there is no existing SSM that shows the main modes of variance in this part of the airways. However, literature agrees that the glottis cross-sectional area is the most significant upper airway morphological parameter influencing deposition [93, 129]. Moreover, realistic mouth-throat geometries were shown to give similar regional deposition predictions as a simplified mouth-throat geometry with an elliptical cross-section when d_p ≤ 10 μm [129]. This suggests that it would not be essential for a SSAM to capture the exact shape of the mouth-throat, but an accurate prediction of the glottal cross sectional area is required. With a good estimation of glottal area, the SSAM could produce deposition that agrees with the deposition as shown for our generic healthy patient mouth-throat (Fig 8).

Our deep lung model was a 0D approximation of deep lung mechanics [50, 51]. Using this 0D model as an outlet boundary condition for the 3D domain (3D-0D model) has been shown to accurately predict lobar deposition in healthy and emphysematous rats [50]. In this 0D model, particles reaching the outlets are deleted and classified as depositing in the deep lung. Oakes et al. [130] developed a 1D model for fluid and particle transport in distal airways (3D-1D model), which tracks particle concentration in the distal airways. This 1D formulation showed minor improvements in agreement with experimental lobar deposition measurements over the 0D formulation [130]. A key benefit of the 1D formulation is that particles can re-enter the 3D domain during exhalation as it is known how many particles are still floating in the distal lung [130]. An inability to model particles re-entering the domain during exhalation is a limitation of our model, as we cannot compare the fraction of drug exhaled with experimental measurements [59, 60]. In the majority of experiments, the exhaled fraction was below 10% [59, 60], meaning this has only a minor influence on the validity of our model for inhalation studies. Incorporating a 1D model for particle and flow transport in distal airways would enable further comparison to SPECT-CT data, as we could assess if our model captures how deep particles penetrate into the lung, rather than only regional deposition given here (Fig 10).

Our 0D deep lung model used global resistance and compliance metrics which were based on empirical correlations from experimental measurements [51, 131]. The resistance and compliance per outlet was then based on the outlet area and the lung’s volume fraction (relative to combined volume). Oakes et al. [51] used this procedure as a initial condition, and then determined the true resistance (accounting for flow resistance in the 3D domain) by iteratively computing the pressure drop between the trachea and each outlet and adjusting the resistance until convergence. This would require many simulations for each patient to determine the exact distribution of flow and particles delivered to each individual outlet in the domain. As we are only interested in regional deposition (right and left lung) at this stage to validate our imaging and model predictions, this was not necessary. Therefore, we have not accounted for 3D domain flow resistance as it has no effect on the distribution of particles between right and left lung, which is dominated by the lobe volume fraction, α_(L), and airway morphology. In future studies where exact local deposition is required, the iterative simulation approach of Oakes et al. [51] may be improved by combining full 3D-0D simulations with a low-fidelity surrogate model to decrease the number of time-consuming simulations [78].

A final limitation is the use of DRRs to train and validate the SSAM. As the clinical end-user would use real chest X-rays as an input, here we discuss the potential issues of this and their mitigations. DRRs are commonly used in the absence of paired X-ray and CT images in 2D-3D reconstruction studies [31–33]. We observed the primary difference between DRRs and real chest X-rays is the reduced detail in DRRs due to the coarser resolution of the CT used to derive the DRR (S2 Fig in S1 File Section 2). This additional detail adds noise to the pixel intensity profiles (S3 Fig in S1 File), which could easily be removed with a Gaussian filter. Alternatively, one could use robust PCA [132] to limit the influence of noisy data on SSAM fitting. Another complication of real chest X-ray data may connected devices that could occlude the underlying lung structure [133], which could also be removed with robust PCA as shown in face recognition literature [134].