Automatic Detection and Reproduction of Natural Head Position in Stereo-Photogrammetry

The aim of this study was to develop an automatic orientation calibration and reproduction method for recording the natural head position (NHP) in stereo-photogrammetry (SP). A board was used as the physical reference carrier for true verticals and NHP alignment mirror orientation. Orientation axes were detected and saved from the digital mesh model of the board. They were used for correcting the pitch, roll and yaw angles of the subsequent captures of patients’ facial surfaces, which were obtained without any markings or sensors attached onto the patient. We tested the proposed method on two commercial active (3dMD) and passive (DI3D) SP devices. The reliability of the pitch, roll and yaw for the board placement were within ±0.039904°, ±0.081623°, and ±0.062320°; where standard deviations were 0.020234°, 0.045645° and 0.027211° respectively. Conclusion: Orientation-calibrated stereo-photogrammetry is the most accurate method (angulation deviation within ±0.1°) reported for complete NHP recording with insignificant clinical error.


Introduction
Recently, a new method for recording Natural head position (NHP) [1][2][3][4][5][6][7][8][9][10] in stereo-photogrammetry (SP) was suggested [11]. The idea was to use the physical references of a pre-captured digital model of a board to correct the orientation of subsequent facial surface models. The board was placed in a way that it was vertically placed and parallel to a hanging mirror at the center of the SP devices. The mirror was an external reference for patients to reproduce their own NHP by looking straight into the mirror at their own eyes in a balanced position [12]. When the patients reproduced their own NHP, this method was able to record this posture accurately. The achieved overall accuracy of physical references derived from the board [11], to the axes defined by gravity and mirror orientations, was within ±0.2°; where standard deviation was around 0.11°. These were considered to be clinically insignificant. The stereophotogrammetric NHP method possesses many unique advantages over traditional approaches: 1) radiation-free; 2) patients can be in their own NHP for capturing of facial surface models without the need for any markers/sensors; 3) only one reference board recording is needed after camera calibration [13][14]. We can further use the obtained orientation as references for other imaging modalities such as CT scans in surgical planning (e.g. 3dMDvultus, 3dMD, USA). Although high accuracy can be achieved with this approach, there are some manual steps in correcting the orientation of patients' facial surfaces. To simplify the procedure in order to minimize operator variability, we developed an automatic orientation calibration method in this paper. The method consists of two components: 1) a specially designed reference board; and 2) a computation method in physical references detection and patients' facial surfaces correction. The aim of this paper was to report this technique and its reliability.

Material and Method
The individuals (appears in Fig 1) in this manuscript have given written informed consent to publish these case details. In fact, they are the first two authors of the manuscript. Consent forms for publication are also signed and keep securely. In this study, the 3D surface scans were performed using 3dMDface (3dMD USA) and DI3D (Dimensional Imaging, UK). The former one is an active SP device [15,16] which projects infrared pseudo random patterns onto the subjects for 3D reconstruction. The latter one is a passive SP device [17] which does not project patterns but requires that the surfaces contain sufficient texture contrast in order to work. For these SP devices, different methods were used for their geometric camera calibration. Orientation of the captured surfaces usually followed the calibration targets' orientation which was manually placed, without aligning to any physical references (e.g. true vertical). For 3dMD, the reconstructions were tilted backwards approximately 45°. For DI3D, the reconstructions appeared to be in the correct orientation, but were found to be 1°-4°deviated from all reference axes. These orientation biases remained constant until the next geometric camera calibration. Due to the manual placement of the geometric calibration targets, the orientation biases for each calibration were usually different.
In the proposed method, the materials and tools are listed as follow:

A. Reference board alignment
To enable human subjects to be in their own NHP, a mirror was installed at the center of the SP device. The subjects were in their NHP when looking straight ahead at their own eyes through the mirror [1][2][3][4][5][6][7][8][9][10][11]. Physical references, i.e. the true verticals and mirror plane heading, were needed to fully characterize subjects' posture. They were aligned with a customized reference board which was then held with a vice clamp and was in turn mounted on a tripod equipped with a 3D geared head and a two-way focusing adjuster (Fig 2). The 3D geared head and the adjuster were used to fine tune the angulation and horizontal positioning respectively. A card paper was printed (HP LaserJet 100 Color MFP M175a, A4 180gsm plain paper) with colored patterns of two red and green discs (Fig 2) along the vertical and horizontal axes respectively. Dotted lines along the axes were used as guides for alignment with laser beams. The card paper was designed using Adobe Photoshop CS6 in 300dpi. The dimension parameters of the discs were: 450px (1.5") in diameter; center-to-center distances of red discs and green discs were 1430px (4.76") and 1200px (4") respectively; 9 vertical laser alignment marks: 20x10mm. This dimension was selected so that the discs would span the whole field of view of the SP devices. In order to transfer the physical references to the board, it was required that the indication of true verticals and mirror heading were available at the capture position. For the true verticals, the alignment lasers project vertical planes automatically. For the mirror heading, we adopted the same method as described in the original SP NHP recording method [11] for placing landmarks in the 3D capturing room in relating the NHP alignment mirror plane to the capturing position (position of board in Fig 3). The alignment laser (Bosch GLL3-80P, Germany) was first placed in the capturing position. It was adjusted in such a way (Fig 4 in [11]) that when the laser was turned on, the laser line on the back wall directly from the vertical laser plane (V1) would overlap with the reflected laser plane V1 from the mirror. This vertical laser plane was therefore perpendicular to the plane of the mirror. The position of the second vertical laser plane V2, which was engineered to be perpendicular to the first laser plane V1, was then marked onto the side walls. These landmarks were marked onto the side walls with a Set up of the reference board. The reference board was taped with color patterns on the front side facing the SP device cameras. Side grooves were crafted by using surface gauge at the side and top edge. Short paper tags were taped near the edges for angulation matching with laser beams. It was held by a vice clamp, connected to a two-way focusing slider for fine horizontal positional adjustments; and 3D geared head for angulation adjustments. Laser beams A (bottom right) and B1-B2 were projections from V2 which was aligned to be parallel to the mirror plane. Laser beam C1-C2 and D1-D2 were projections from front alignment lasers. Yaw angle was aligned with A-B1 (right) and top board edge; pitch angle was aligned with B1-B2 and side groove; roll angle was aligned with C1-C2, D1-D2 and dotted lines on card paper. pencil as dotted lines (magenta and blue boxes in Fig 3) such that any laser beam deviations could easily be identified at their projection edges. Hence, the mirror plane was transferred to the capturing position with the indications of side wall landmarks. The alignment laser was then moved to the left (or right) side in order to make space for 3D capture of the board ( Fig  3). It was also aligned with the side wall landmarks and elevated to a position such that the laser was able to light up the top and side of the board edges as shown in Fig 4. Moreover, two additional alignment lasers were placed next to the 3D camera rig for indicating true verticals (V2', V2") on the reference board pattern. The position and angulation of the board were adjusted to match with the laser beams. Laser beams A-B1 and B1-B2 (Fig 2) were projections from V2 (Fig 3), which was aligned to be parallel to the mirror plane. Laser beam C1-C2 and D1-D2 were projections from the front alignment lasers.
The reference board angulation was aligned with the laser beams using the following procedure (Fig 2),

Procedure 1
1. Align yaw angle with A-B1 and top board edge.
2. Align pitch angle with B1-B2 and side edge. It was also cross validated with the agreement of the front alignment lasers and the patterns.
3. Align roll angle with C1-C2, D1-D2 and dotted lines on the card stock. After alignment, 3D capture of the reference board was performed with the lasers turned off.
The captured digital mesh model of the board was saved in Wavefront (OBJ) format for subsequent calculations. The board was then removed to make space for subsequent facial captures.

B. Digital processing on orientation detection and correction
After the device geometric camera calibration, the aligned reference board was scanned with the SP device. The digital processing on the captured digital mesh model consisted of two parts: 1) Physical reference detection; and 2) orientation correction. In the first part, axes corresponding to the physical references were detected from the reference board digital mesh models and saved. In the second part, the saved parameters were used to correct subsequent patients' facial models. We implemented these two processing steps in Matlab (Mathworks, USA).
(i) Physical reference axes detection. Following procedure 1, the red discs were aligned along with the true verticals (roll and pitch), whereas green discs were aligned with the true horizontal (roll and yaw). Moreover, the discs were on a vertical plane (yaw) which was parallel to the mirror. This is because the board has been aligned with the laser plane V2 (Fig 3). The 3D vectors, v x and v y , connected through the centroids of each group of the discs indicated the true horizontal and vertical axes. As the SP devices do not consider physical orientation, these axes were tilted (Fig 1, upper left) in the 3D reconstructed surfaces. To extract these vectors (Fig 5), the associated texture colors of the 3D surfaces were first converted into CIELAB color space [18]. 3D vertices of the green and red discs were segmented according to the chromaticity components a Ã and b Ã of the associated texture color. In our configuration, the angle of chromaticity vector (a Ã , b Ã ) of red and green colors were distributed at 0.2 rad (±0.4 rad) and π-0.6 rad (±0.4 rad) respectively. For the adopted SP devices, lighting and exposures were under well-control. The captured images were very clean. Camera distortions and perspective effects were effectively compensated by the SP devices for enabling the 3D surface reconstruction. Therefore, discs centroids could reliably be calculated by simply taking the mean of the segmented 3D vertices.
(ii) Digital mesh model correction. Assuming that the SP devices were able to reconstruct the 3D surface with negligible distortion errors, it sufficed to adopt a rotation transformation Setting of 3D camera room. The side laser was elevated to a higher position than the reference board such that the laser plane V2 can beam up the side (pitch) and top edge (yaw) of the board. It was aligned to the side wall landmarks such that V2 was parallel to the NHP mirror plane. Another alignment lasers were placed next to the 3D camera rig for projecting laser beam indicating true verticals for roll angle. for the orientation correction, which corrected the physical reference axes v x and v y back to the X and Y axes in 3D Euclidean space. That is, in homogenous coordinates, where b is the center of the four discs; 0 is 1x3 matrix with all elements zero and R is the rotation matrix, where u y is the unit vector of v y ; and u x is obtained from Gram-Schmidt process on u y and v x . Both unit vectors u x and u y were set to be positive in x and y components respectively. The correction of each vertices v: = (v 1 ,v 2 ,v 3 ,1) T , of the reference board and patients' digital mesh model were computed by, where we denote w as the corrected vertices. Since the SP devices were found to have high reproducibility from previous studies [13][14], this matrix is valid for the subsequent captured face models until next geometric camera calibration.

A. Accuracy analysis
The accuracy of the proposed method depended on several components: the accuracy of the alignment lasers, reliability of 3D surface reconstruction, reference board placement and computer orientation axes detection. For the adopted alignment lasers (Bosch GLL-3-80P), the accuracy was rated to be within: ±0.2mm/m (±0.011459°) from the true verticals/horizontal. They were calibrated in factory using devices with traceable accuracy. The angulation accuracy for each alignment steps with alignment lasers was read as ±0.011459°to the true verticals/horizontal. In the following experiments, the reference board was aligned with the laser beams projected on its marks. The pitch and roll angulations were directly aligned with the alignment laser (angulation accuracy ±0.011459°). The yaw angle was aligned to the side wall landmarks which were inferred from the NHP alignment mirror plane (angulation accuracy ±0.022918°). The overall accuracy of the proposed method was the sum of laser alignments and the reliability of the reference board placement and orientation axes detection. We used the following procedures to find out the reference axes detection reliability,

Procedure 2
1. Side walls landmarks were set and the NHP alignment mirror were kept fixed.
2. The SP device was calibrated.
6. Compute all orientation axes u x and u y with the simulation program.
7. Calculate angulation deviations of the axes in pitch, roll and yaw angles relative to the mean axes.
Assuming the alignment lasers were in good condition; mirrors, cameras and walls landmarks were fixed during the tests. Each laser beams and alignment patterns were matched appropriately. Patterns and the reference board were also checked that they were not significantly deformed due to handling and environment conditions. From the above procedure, we were able to characterize the overall accuracy of board placements in terms of orientations deviation. Components of the orientation axis u y , were used to calculate the roll and pitch angles with y roll ¼ arctanðu y;2 =u y;1 Þ and y pitch ¼ arctanðu y;2 =u y;3 Þ respectively. The yaw angle was calculated with y yaw ¼ arctanðu x;1 =u x;3 Þ where we denote the detected orientation axes as u x :=(u x,1 ,u x,2 , u x,3 ) T and u y :=(u y,1 ,u y,2 ,u y,3 ) T .

B. Validation with dummy plastic head
Validation of the proposed method was illustrated in Fig 6. An alignment laser was placed at the center of the SP device with the laser beams (vertical and horizontal) aligned to be perpendicular to the mirror plane. The projected laser beams on the plastic head were indicated with arrow stickers. The captured plastic head surface was corrected using the detected axes. In the figure, we can see that the captured surfaces were corrected as they would appear in the physical world. The arrow stickers were pointing to the true horizontal and verticals.

C. Axes detection reliability
Besides the plastic head validation, we investigated the axes detection reliability using the active and passive SP devices respectively. They were checked with multiple shots on the still aligned reference board with Procedure 2. The angular deviations in pitch, roll and yaw were found to be within ±0.01199°, ±0.16296°, ±0.007214°respectively; where standard deviations were 0.00433°, 0.07639°and 0.00287°respectively for 3dMD (M = 60). For DI3D (M = 30), the deviations in pitch, roll and yaw were within ±0.07393°, ±0.16327°, ±0.04361°respectively; and standard deviations were 0.02831°, 0.06542°and 0.02385°. The descriptive statistics of the experiments are shown in Table 1. Higher angulation deviation in roll was found. It is probably due to sampling error of disparity matching during surface reconstruction. The axes detection deviations can be reduced by averaging the detected axes from multiple shots. Averaging of five shots was found to be sufficient to reduce the roll angulation errors to similar level of pitch and yaw's.

D. Reproducibility
The inter-operator reproducibility was tested among three operators (TCH, JL, TSL). We performed the reference board alignment for a total of N = 75 (M = 1) in 5 days. The number of trial for the operator 1, 2 and 3 are respectively 49, 9 and 9. Operator 1 performed 52 alignments on the first 2 days and the other operators performed 12 and 11 alignments respectively on the fifth day. The SP device was firstly calibrated on the first day. No more geometric camera calibration was done in this session. Eight of the captures were excluded due to bad exposures. Three re-alignments of laser to side walls landmarks were performed at the first, 38 th and 53 rd trials. Roll angle was aligned with one front alignment laser. Since the SP device possesses high reproducibility and only one geometric camera calibration was done at the beginning, we were able to see the reproducibility among operators whether contain significant bias in reference recording. In Fig 7, it shows the angulation deviations to the mean for the trials. The angulation deviation of board placement in pitch, roll and yaw were within ±0.075°, ±0.18°, ±0.12°respectively; and standard deviations were 0.02335°, 0.09239°and 0.04249°T respectively. The descriptive statistics of the experiments are shown in Table 2. We can see that the reproducibility was very high even several side laser alignments were performed. One sample t-test was performed on the results of each angulation deviation of pitch, roll and yaw from the three operators. The null hypothesis was that the mean angulation deviations were not different to 0°. From Table 3, the mean angulation deviation was not significantly different statistically to 0°with 95% CI, p > 0.05 except for operator 2 in yaw angle. Despite of the fact that significant difference was found for operator 2, the range of angulation deviation for all operators was still very small: i.e. ±0.12°. The largest mean differences were 0.014°±0.024°in pitch, 0.051°±0.090°in roll and 0.053°±0.042°in yaw.

E. Reliability analysis on active and passive SP devices
To measure the reliability for the active and passive SP devices, we performed (TCH, SWZ) reference board alignments with M = 5. That is, the detected axes were averaged with five shots in still for each alignment. In Figs 8 and 9, it shows the angulation deviations for N = 30 and 33 with averaged axes for using 3dMD and DI3D respectively. For 3dMD, the deviation ranges in pitch, roll and yaw were within ±0.039904°, ±0.081623°, ±0.062320°; and standard deviations 0.020234°, 0.045645°and 0.027211°respectively. For DI3D, the deviation ranges in pitch, roll and yaw were within ±0.098020°, ±0.195240°, ±0.166327°; and standard deviations 0.038850°, 0.079088°and 0.071816°respectively. The descriptive statistics of the experiments are shown in Table 4.

Discussion
Traditional approaches to obtaining NHP involve capturing physical references and the patients' images at the same time. Methods include placing markers on subjects' face [19,20]; holding orientation sensors with bite-jig [21,22]; wearing eyeglasses mounted with inclinometers [23]; or hold spirit level equipped face bow [24] etc. The best accuracy achieved was ±1.1° [ 21,22]. In using SP device, the proposed method captures physical references and patients' images in separate scans [11]. This eliminates operator variability on applying references to patients and their adverse effects patients' NHP. The major difference of the proposed approach to the traditional one is that, once the calibrations are done, the subsequent 3D scans with NHP are the same as normal. The patients can present their NHP in the most natural way.
In the proposed approach, two (or three) alignment lasers were used in the angulation adjustment steps. This enables simultaneous alignment in pitch, roll and yaw angles. Along with the adoption of 3D geared head and focusing adjuster, typically only 2.5 minutes is required for the whole setup. After aligning the reference board and 3D capture, the orientation axes are detected and saved. Subsequent patients' facial models are then corrected automatically from the saved axes. There is no operator variability in the digital processing phase.
The patterns for reference axes detection were colored discs. They span almost 3/4 of the whole field of view with around 800-1500 vertex points for each orientation axes. The same alignment method and programs were used for the experiments with both of the SP devices. The only difference was that we added some textures on the colored discs pattern for DI3D. From the experiments, the reliability was improved by taking several shots of the board in still and averaging the detected axes. This can reduce the variability of orientations that is probably due to noise and data sampling. It is possible because the reference board can be kept still allowing multiple captures. On the other hands, other combinations of pattern are possible provided that the reference board is flat enough and the patterns are detectable by the SP device.
Integration of the orientation calibration to geometric camera calibration is also possible, provided that physical references are transferred and applied appropriately. In this paper, we demonstrated the idea of "orientation calibration" and suggested one solution using the 3D vertices and texture in the physical reference recording for NHP. It is compatible to the current SP devices (3dMD & DI3D) which uses different methods of geometric camera calibrations.

Conclusion
In this paper, we presented a highly accurate and repeatable protocol for capturing physical references for recording NHP in stereo-photogrammetry. The suggested additional orientation calibration step is performed after the device geometric camera calibration. No devices or markings are needed to be placed on patients. It can be applied in most of the commercially available SP devices. Orientation-calibrated stereo-photogrammetry is the most accurate method (angulation deviation within ±0.1°) reported for complete NHP recording with insignificant clinical error.
Supporting Information S1 File. ZIP archive of The reference boards in digital form (Adobe psd and pdf). (ZIP)