Table 1.
Parameters of the VTK optical and lighting model.
Fig 1.
The Location Search technique.
A flowchart showing the steps of the Location Search registration technique.
Fig 2.
Slicing the surface mesh in step 3 of Location Search.
(a) The virtual endoscopic image at a path position in the left side of the nasal cavity looking towards the pharynx. On the left is the medial wall of the nasal cavity, and on the right is the inferior concha. (b) The rendered image of the slice outline. Both sides of the nasal cavity can be seen, as well as the maxillary sinuses. The virtual endoscope’s position is shown as a red dot. (c) The outline has been flood-filled from the virtual endoscope’s position and unconnected regions have been discarded. (d) The centroids of the k-means clusters are shown as blue dots. This image was cropped slightly to fit the template.
Fig 3.
The clay phantom with radiopaque markers.
(a) A picture of the marker phantom. The inner diameter is about 5 cm. (b) An endoscopic video frame from inside the phantom with four of the markers visible. (c) The corresponding virtual endoscopic image.
Fig 4.
The clay phantom with bolus material.
(a) A picture of the bolus phantom. This phantom is smaller than the marker phantom, with an inner diameter of about 2 cm. (b) An endoscopic video frame from inside the phantom with the bolus visible on the lower right. (c) The corresponding virtual endoscopic image.
Fig 5.
Virtual endoscope path creation in the marker phantom.
A coronal CT slice shows the manually-selected virtual endoscope path through the marker phantom used for the Location Search registration technique.
Fig 6.
The measurement of CT-space marker positions.
(a) A video frame with four markers visible. Their manually-selected pixel locations are overlaid in red. (b) The corresponding virtual endoscopic image rendered at the registered coordinates. The manually-selected pixel locations are overlaid in red and the image-plane projections of their true 3D locations are overlaid in blue. The average 3D error for these four measurements was 0.29 cm, and the average 2D error was 11 pixels.
Fig 7.
The measurement of the CT-space bolus surface.
(a) A video frame showing the bolus material with the manually-drawn contour overlaid in red. The contour contains over 28,000 pixels, but downsampling reduced this to 613 pixels spaced evenly within the area. (b) The corresponding virtual endoscopic image rendered at the registered coordinates. The manually-drawn contour is overlaid in red and the image-plane projection of the bolus volume is overlaid in blue. The MAD for this frame was 0.23 cm and the Dice coefficient between the 2D contours was 0.82.
Fig 8.
Virtual endoscope path creation in a patient.
A sagittal CT slice shows the manually-selected virtual endoscope path through a patient used for the Location Search registration technique. The nose is cropped out due to the field-of-view reduction described in section 2.4.3.
Fig 9.
Obtaining ground-truth endoscope coordinates via camera resectioning.
(a) A video frame showing the epiglottis and glottis with manually-selected points overlaid in blue. (b) A virtual endoscopic image rendered at the initial guess for the endoscope coordinates with the manually-selected corresponding points overlaid in red. These points are projected into the surface mesh to get their 3D positions. (c) The virtual endoscopic image rendered at the ground-truth endoscope coordinates. These coordinates are obtained by moving the virtual endoscope to minimize the pixel distances between the video frame points (overlaid in blue) and the image-plane projections of their 3D coordinates (overlaid in red).
Fig 10.
Results of the two registration techniques in the marker phantom.
The average 3D point errors for each registration frame in video sequence 1 (left) and video sequence 2 (right) are shown for both registration techniques vs. the frames’ time points in the video. In sequence 1, the virtual endoscope became lost during Frame-To-Frame Tracking, resulting in very large errors for the last seven frames. In sequence 2, Location Search failed on two frames, resulting in very large errors.
Fig 11.
Results of the two registration techniques in the bolus phantom.
The mean absolute distances (MADs) between the measured and ground truth bolus contours for each registration frame in video sequence 1 (left) and video sequence 2 (right) are shown for both registration techniques vs. the frames’ time points in the video. Location Search performed better than Frame-To-Frame Tracking in this phantom, especially on sequence 1.
Fig 12.
Structural differences in the pharynx between the two endoscopic examinations for patient 1.
(a) A frame from video 1 showing the epiglottis and glottis. (b) A similar frame from video 2. The pharynx appears narrower, and the posterior wall of the pharynx (at the top of the image) produces a large bright area in many of the registration frames that could not be reproduced in the virtual endoscopic images. (c) A virtual endoscopic image showing the epiglottis and glottis. The surface mesh appears wider than the pharynx in both videos, but the differences are particularly apparent when compared to video 2.
Fig 13.
Registration failure caused by structural differences between the endoscopic video and the virtual endoscopic images.
(a) A registration frame from video 2 showing the epiglottis and glottis. (b) The virtual endoscopic image rendered at the ground truth coordinates obtained via camera resectioning. (c) The virtual endoscopic image rendered at the registered coordinates obtained via Location Search. The similarity measure matched the large bright area in the registration frame to the bright region seen in this virtual image inferior to the epiglottis. The distance between the ground truth and registered coordinates was 3.66 cm.
Fig 14.
Virtual endoscopic images lacked anatomical detail in patient 3.
(a) An endoscopic video frame showing the epiglottis and glottis. (b) A virtual endoscopic image in the same region. The epiglottis is poorly defined and attaches to the posterior wall at the top of the image. The region in front of the epiglottis has very little structure in the virtual image as well. The glottis is also poorly defined, and, unlike in the video frame, it is offset anteriorly and can barely be seen behind the epiglottis.
Fig 15.
Three examples of successful registrations in patient 3.
The top row shows three registration frames, and the bottom row shows the corresponding virtual endoscopic images rendered at the registered coordinates obtained via Location Search. None of these is a perfect match, and the virtual endoscope appears to be farther from the epiglottis in the virtual images than the camera was in the video frames. However, even with the lack of detail in the surface mesh, Location Search was able to find reasonable matches for these video frames.