Fig 1.
Schematic representation of sample acquisition (A), label-free imaging using a THG/MPEF microscope (B) and standard cytology processing using Diff-Quick (DQ) staining (C). Images partly created with BioRender.com.
Table 1.
Overview of the imaged samples, acquired number of mosaics and total imaged areas.
Table 2.
Reference cytology differential cell percentages and standard deviation per sample.
Fig 2.
Deep learning cell counting pipeline.
Pre-processed THG/MPEF mosaics are forward fed into the pre-trained ResNet50 model, together with reference cytology leukocyte ratios as labels. Class weights keep the model from overfitting on the ratio distribution. Mean absolute error is calculated between the fractions input and softmax output. After optimization, Grad-CAM activation maps can be extracted from the last convolutional layer for model validation. Scale bar 150 μm.
Fig 3.
Label-free imaging of a BALF sample showing cellular structures. The BALF images contain cells of different sizes and shapes, and varying THG and 2PEF signal intensity. Magnified images show cell nuclei (CN), nuclear lobes (NL), and cell cytoplasm with increased 2PEF signal (F), THG signal (T) or both (FT). Non-cellular structures were also present, indicated as dirt (D). Acquisition time of this quarter of the acquired 2D mosaic was 38 seconds.
Fig 4.
Label-free images of different leukocytes showing the discriminating characteristics in cellular and nuclear morphology and signal intensity.
A-C: THG, 2PEF and 3PEF images of neutrophils, eosinophils, and lymphocytes from various blood fractions (A) and one bronchoalveolar lavage sample (B) show similar cell morphology as in standard cytology Diff-Quick (DQ) stained images. Neutrophils and eosinophils were imaged from the same granulocyte fraction and the images were saved with the same contrast settings. Two examples of macrophages are presented from two different BALF samples (C). D: 8 μm depth scans (with steps of 1 μm) of two neutrophils show that the visible number of nuclear lobes depends on the imaging plane. A-D: For all images the same scale was used, for direct comparison of cell sizes.
Table 3.
Leukocyte characteristics to distinguish the different cell types, based on THG and MPEF (2PEF and 3PEF) images.
Fig 5.
Mean absolute error values of top-performing model during training and validation.
Graphs show mean absolute error (MAE) reported during training and validation. Solid blue line indicates pre-weighted training MAE values, solid orange line weighted training MAE values, and gray line validation MAE values. Weighting of the MAE takes place only during training by means of the class weights to avoid the model from overfitting on the leukocyte ratio distribution. As can be seen from the graphs, the class weights penalize raw MAE. The validation MAE is consistently lower than the training MAE, which could be explained by the lack of real-time data augmentation during validation.
Fig 6.
Deep learning regression outputs on the validation and testing set compared to reference cytology counts.
See Table 2 for included cases in validation and test sets. Note that there is no standard deviation bar for the reference cytology of the PBMC in the testing set, because this sample was counted once.
Fig 7.
Grad-CAM activation results for test case BALF 4.
See S1 Table for the regression performance of the ResNet50 model in this case. The activation maps are focused on the yellow square inlet from Fig 3 in the original mosaic image to illustrate individual cell activation. The maps are displayed using a ramping lookup table, which expresses the pixel value in four colors ranging from minimum to maximum activation: blue, green, yellow and red. Noticeable is the low activation of background pixels (green) and medium-to-high activation of cellular objects (yellow to red). Between class activation maps, the same part of a cell might be less or more activated. Scale bar 50 μm.