Fig 1.
SimpleMind multi-agent thinking architecture.
Schematic showing agent types and the Blackboard for sharing of results during image understanding.
Fig 2.
SimpleMind semantic network (SN) for kidney segmentation.
The human readable text files of the SimpleMind semantic network (SN) for kidney segmentation. (A) The “Node List” text file of the SN where each line corresponds to a node in the SN. (B) The text file for the thorax node, containing the attributes for recognizing the thorax object. (C) The text file for the kidney_left_init node. (D) The text file for the kidney_right_init. (E) Graph visualization of the kidney SN.
Fig 3.
Kidney segmentation on CT: (A) original image; (B) reference kidney segmentation (green overlay); (C) segmentation result for the kidney_cnn node using a DNN alone (the baseline results); (D) SimpleMind segmentation of the right_kidney node (red overlay); (E) SimpleMind segmentation of the left_kidney node (red overlay).
Fig 4.
The SimpleMind semantic network (SN) for prostate segmentation.
The SN of a Node List file listing the nodes in the SN and the corresponding set of human readable node files. (A) The prostate SN Node List file; (B) The prostate_apex_box node file; The node file is derived with the attributes for recognizing the apex area slices from the initial prediction from the prostate_whole_cnn node. (C) The prostate_apex_focused_cnn node file. (D) The prostate_apex_attention node file. (E) Graph visualization of the prostate SN.
Fig 5.
Preprocessing of DNN inputs for the prostate segmentation.
(A) original image; (B) The segmented prostate_apex_box node; (C) preprocessed image for input to the DNN in the prostate_apex_focused_cnn node (red box indicates the guided area for the local histogram generated from the prostate_apex_box node); (D) preprocessed image used for the DNN in the prostate_whole_cnn node.
Fig 6.
(A) original image; (B) prostate reference segmentation (green contour); (C) segmentation of the prostate_whole_cnn node (the baseline results; yellow contour); (D) SimpleMind segmentation of the prostate_whole node.
Fig 7.
The SimpleMind semantic network (SN) for the ETT.
The SN of a Node List file listing the nodes in the SN and the corresponding set of human readable node files. (A) The endotracheal tube SN Node List file. (B) The et_zone_1 node file. (C) The et_tip_correct node file. (D) The et_tube_correct node file. (E) Graph visualization of the endotracheal tube SN.
Fig 8.
Visualizations of the nodes involved in the knowledge-supported machine reasoning to determine correct ETT placement.
(A) Original chest x-ray image; (B) trachea region overlayed on an enhanced image; (C) ETT safe zone (et_zone) and tube tip location (et_tip) showing the tip within the safe zone; (D) the final output of the system as presented to the physician with the green overlay indicating the system determination of correct tube placement.
Fig 9.
Visual example of nodes leading to determining correct ETT placement, determined by knowledge-supported machine reasoning.
(A) Original chest x-ray image; (B) trachea region overlayed on an enhanced image; (C) ETT safe zone (et_zone) and tube tip location (et_tip) showing the tip outside the safe zone; (D) the final output of the system as presented to the physician with the red overlay indicating the system determination of incorrect tube placement (tip too low relative to the carina).
Fig 10.
Visual example of nodes leading to determining incorrect ETT placement, determined by knowledge-supported machine reasoning.
(A) Original chest x-ray image; (B) trachea region overlayed on an enhanced image; (C) ETT safe zone (et_zone) and tube tip location (et_tip) showing the tip outside the safe zone; (D) the final output of the system as presented to the physician with the red overlay indicating the system determination of incorrect tube placement (tip too high relative to the carina).
Fig 11.
The SN node for prostate_apex_focused_cnn.
The prostate_apex_focused_cnn semantic network node showing parameters exposed for optimization relating to DNN input normalization and learning.
Fig 12.
Evolution of fitness performance of chromosomes (i.e., different parameter sets) over time within the GA optimization in prostate segmentation.
The y-axis represents the fitness score for each chromosome in a population, representing the weighted average of the prostate segmentation performance of the apex, base, and whole prostate regions, averaged across the cases of the KNoLO tuning set. For comparison, the red dotted line demarcates the performance of the prostate segmentation SN using only hand-tuned parameters by a data scientist. Each generation (iteration) has 30 chromosomes, with the green points representing the highest fitness chromosomes and the red points representing the lower fitness chromosomes. The highest fitness score was 0.833.
Fig 13.
Example 2D image slices and their associated pixel intensity histograms before and after preprocessing.
Through KNoLO, an optimal combination of preprocessing steps and input channels was found for each prostate region specific CNN, resulting in the displayed example slices.
Table 1.
Image preprocessing steps selected by the KNoLO optimizer for each DNN input channel for the three prostate regions.