Image registration

The image registration [3–5] is the problem of finding a transformation T that aligns one or more images to a reference image. This transformation transforms the points of an image to the same location of the points in the reference image. Intra-subject medical image registration aligns images of the same patient while inter-subject registration aligns images of different patients. Fig 2 shows the main registration components: a fixed and a moving image, a cost function, an optimiser, and a transformation. The first transformation handles the issue of different resolutions by mapping both input images to a virtual domain in the images’ physical space. This takes place before the optimisation process starts. In this implementation, medical image registration is not performed in the images’ space [6].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The main components of image registration: Input images, cost function, optimiser, and a transform.

The virtual domain transform deals with the issue of different image resolution.

https://doi.org/10.1371/journal.pone.0264449.g002

Mathematically, the transform is a function of the image points P and the transform’s parameters μ where it takes an image point (or a vector of points) and outputs the image transformed point (or a vector of transformed points). We can write this transform as T(P, μ) where P is the image coordinates vector. The second input to the transformation’s function T is μ which is a vector that contains the transformation’s parameters.

The transformation transforms an image called a moving image I_M(P) to another image called a fixed image I_F(P), where P is a vector of all image point coordinates. The image itself can be considered a function of points where the output is the intensity (or color) values of the point locations.

Finding the parameters of this transformation is a challenging task. Hence, the general registration problem is still unsolved and many papers are published every year trying to solve different aspects of this problem.

Optimisation techniques [7] are usually used to find these parameters. An optimiser, such as Gradient Descent (GD) [8], tries to find these parameters by minimising a cost function of the input images. Popular image registration cost functions use similarity metrics such as Mean Squared Error S_MSE(I_F, I_M) [9] or Mutual Information S_MI(I_F, I_M) [10]. Note that by minimising the similarity metric value, we are maximizing the similarity between the fixed and the transformed moving images.

The optimisation problem is stated as follows: (2) where μ is the transformation’s parameters’ vector, I_F and I_M are the fixed and the moving images, P is the points’ coordinate vector, S is the similarity metric function, and T(P, μ) is the transformation’s function.

In the GD optimiser, the new parameters are updated using the derivative of the cost function with respect to the transformation’s parameters, e.g.: (3) where k is the optimisation iteration, λ represents the step-size or the learning rate of the GD. The derivative of the similarity metric with respect to the transformation’s parameters is required for Eq (3). Using the chain rule, this derivative can be divided into two main parts: (4) where is the first order derivative of the similarity metric S with respect to the transformation’s parameters μ at k optimisation’s iteration, p_j is an image pixel coordinate, and n is the total number of the image’s pixels. The first part of the right-hand side in Eq (4) requires the derivative of the similarity metric and the transformed moving image. The second part is called J_μ and it is similar to the Jacobian matrix of the transformation as it represents the transformation’s partial derivatives at each image point with respect to the transformation’s parameters. The term J_μ is computed at each optimisation iteration. For a large image size and complex transformation, the computation requires a lot of time to complete. To solve this issue, the Stochastic Gradient Descent (SGD) optimisation approach [11, 12] uses samples instead of the whole image. Sometimes these samples are taken randomly. The stochastic approach allows fewer computations and provides better results in practice. The parameters updating rule of SGD is: (5) where and represent samples from the input images I_F and I_M.

Aim of the work

The aim of this work is to propose and evaluate a new relatively easy and fast automatic cochlear image registration and fusion method utilizing simple computer hardware and software to align and fuse pre-and postoperative multimodal cochlear images in cochlear implant (CI) patients.

Ethics statement

The use of all human datasets in our experiments were approved by the Ethics Committee of the Rheinland-Pfalz state, Germany (Landesaerztekammer Rheinland-Pfalz, Koerperschaft des oeffentlichen Rechts) with request number 2021–15895-retrospektiv.

We conducted a retrospective review from 2014 to 2020 using our institution’s CI database to retrieve the preoperative and postoperative imaging data of patients who underwent CI surgery. Exclusion criteria were the presence of congenital cochlear anomalies or severe artifacts distorting the image quality.

Dataset

The study included 41 CI patients of different genders and ages with a total of about 131 preoperative and postoperative imaging studies from different modalities (MR, MDCT and CBCT). All the patients’ information was anonymised.

All preoperative MR images were done on a Siemens Avanto 1.5 Tesla device. The sequence used was transverse T2-spc 3D sequence with the following specifications: Repetition time (TR) 9.1 ms Echo time (TE) 3.62 ms, Flip angle 80 degrees; pixel bandwidth 130Hz/Px; thickness 0.8 mm; voxel size 0.7x0.5x;0.8 mm; FOV selection 200 mm; Matrix size 384x384; SNR 1.0; Acquisition time 3:30 min. Each used image has a size of 384 × 512 × 64 voxels with 0.39x0.39x × 0.7 mm spacing.

All preoperative MDCT images were done on a Siemens Sensation Cardiac 64 Slice. Each image was taken with these parameters: KV = 120; mAS = 140; matrix = 512x512; FOV = 65–75 mm; slice thickness = 0.6 mm; Kernel = U90 μ. Each MDCT image has a size of 512 × 512 × 58 voxels with 0.12x0.12x1.0 mm spacing.

Preoperative and postoperative CBCT images were done on a Morita 3D Accuitomo 170 (J. Morita). Each ear was imaged separately using the following parameters: 90-kV tube voltage; 5-mA current; a high-resolution mode (Hi-Res J.Morita) with a rotation of 180 degrees; voxel size of 0.125 mm and FOV of 80x80x80 mm. The final postoperative CBCT images have a size of 485 × 485 × 121 voxels with 0.12 × 0.12 × 0.5mm spacing and the final preoperative CBCT images have a size of 483 × 483 × 161 voxels with 0.12 × 0.12 × 0.3 mm spacing. These CBCT images were probably cropped as they don’t have isotropic spacing.

ACIR: Automatic Cochlea Image Registration

The multi-registration Automatic Cochlea Image Registration (ACIR-v1) was proposed in [13] to provide practical cochlea image registration in a few seconds. It combined different standard techniques and tuned parameters in a customized way that worked for cochlea images. It combined different registration elements in a hierarchical approach of two stages. In the first stage, a rigid registration was used while in the second stage, a non-rigid B-spline registration was used. Moreover, the metric was based on the input image types i.e S_MSE for monomodal images i.e CBCT and CBCT, and S_MMI for multimodal images e.g. MDCT and MR, or MDCT and CBCT. Even though CBCT and MDCT are very similar, in practice, CBCT and CT had better image registration result when S_MMI was used. It used the Adaptive Stochastic Gradient Descent (ASGD) as an optimiser [14]. The step-size λ, in Eqs (3) and (5), was an important optimisation factor that had to be set manually. It had a large influence on the GD optimisation results. This factor is data-dependent, so finding a suitable value for different problems was challenging. ASGD is an optimiser that adapts the step-size factor λ automatically, using an image-driven mechanism, to predict its value. It replaces λ with a parameter δ that defines the maximum incremental displacement allowed between the optimisation’s iterations. The δ parameter was computed using the voxel size of the input image. The mean voxel spacing (in mm) was found to be a good value.

The cochlear image registration problem in this study is an intra-subject image registration problem. Hence, it belongs to rigid image registration problems. For such images, the same transform applies to all pixels in the moving image. Finding the correct transform for any part of the moving image may solve the problem for other parts in the image. Some areas in these images have clear structures and less noise. Cropping the original images to one of these small areas and registering them is an efficient way to produce a transform that registers the original images. Another method, ACIR-v2, which uses prior knowledge of the images’ characteristics was introduced in [15]. ACIR-v2 uses the ASGD optimiser to minimise negative Mattes’ mutual information S_MMI similarity metric of two images [16] by modifying 3D rigid transformation’s parameters.

ACIR-v1 and ACIR-v2 were GD methods. In GD, the second order derivative of the similarity metric function, also called the Hessian H, was the identity matrix I, hence, it was omitted from (3). In some cases where the Hessian matrix is ill-conditioned, this assumption leads to an inefficient optimisation result. In this case, one should use a Newton optimiser method where the second-order partial derivatives of the cost function are considered. This approach is very computationally expensive as the computation of the Hessian requires a lot of time and memory capacity.

The proposed idea

Quasi-Newton methods [7] use an approximation of the Hessian of the cost function which reduces the computation time. The update function in quasi-Newton optimiser is: (6) where k is the current optimisation iteration, is the first order derivative of the similarity metric function at iteration k, λ is the step-size and is a symmetric positive definite approximation of the Hessian matrix. A popular quasi-Newton update rule is Broyden-Fletcher-Goldfarb-Shanno (BFGS) [7] which uses the first-order derivatives to update the inverse Hessian directly. This produces a linear rate convergence. The BFGS update rule is described in Eq (7): (7) where , I is the identity matrix, , , is the inverse matrix of , V^T is the transpose of V, and μ^T is the transpose of μ.

This still requires a long computation time and large memory. The memory issue is solved by the Limited-memory BFGS (LBFGS) [17] method which saves only a few numbers of the previous Hessian approximations. The computation time issue is solved using a stochastic approach.

This paper introduces the Automatic Cochlea Image Registration (ACIR-v3) method which is based on the stochastic quasi-Newton with Limited-memory BFGS (s-LBFGS) updating rule.

The proposed method replaces the ASGD optimiser in ACIR-v2 with the s-LBFGS optimiser [18]. The s-LBFGS combines the stochastic approach with the LBFGS approach. This allows for faster convergence and more robust results.

Evaluation method and statistical analysis

The ACIR-v3 method was compared to its previous versions (ACIR-v1, ACIR-v2), and three other optimisers i.e. ASGD, Fast Adaptive Stochastic Gradient Descent (FASGD) [19], and Fast Preconditioner Stochastic Gradient Descent (FPSGD) [20]. Since the result time recorded might differ according to the hardware used, all the experiments were done using the same hardware. The hardware used was a computer equipped with AMD Ryzen 3900 CPU, a 32 GB memory and a Nvidia RTX2080Ti graphics card. We used the same parameters for all methods and the original implementations of ACIR-v1, ACIR-v2, ASGD, s-LBFGS, FPSGD by their authors which are provided as an open-source in 3D Slicer software and Elastix 5.0.0 toolbox [21, 22].

In the ASGD method, we used the original images and no cropping was involved. The ASGD parameters were: rigid transformation, no multi-stages, and no multi-resolution. The optimiser was changed in order to make fair comparisons between FASGD, FPSGD, ACIR-v2, and ACIR-v3. The comparison between the methods included three criteria: the image registration accuracy, the required time to align a pair of images, and the robustness of the method. The time was recorded using a fixed number of iterations (n = 100) and the pre-processing and post-processing steps, e.g. image cropping, were included. The robustness of the method was evaluated using the percentage of the missing results according to Eq (8): (8) where N_success is the number of cases where the method produced a valid result, and N_fail is the number of cases where the method failed to produce a valid result e.g. the optimisation stopped.

Algorithm 1: ACIR-v3

1 Input: two cochlea images I_F(P), and I_M(P);

2 Output: a registered and fused image I_result(P);

3 Locate the cochlea locations in input images;

4 , ;

5 Set transform T = 3D rigid transform;

6 Set k = 0.;

7 While is large and k < 100 do;

8  μ = update the old μ using s-LBFGS.;

9  Set k = k + 1;

10 Transform the moving image, I_result(P) = I_M(T(P, μ));

11 Fuse the result, I_result = I_result + I_F

Two neuroradiology experts added two 3D landmark points (fiducial points) (in consensus) to all the imaging studies using the 3D Slicer software version 4.10 [23, 24]. The two landmark points represent the round window and the cochlear apex, see Fig 3. These landmarks were saved for each imaging study and were used later for validation of the image registration results. After aligning the images, the landmarks of the moving image were transformed using the same transformation. Thereafter, RMSE in mm was measured between these transformed landmarks and the related fixed image landmarks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cochlea 3D models and two landmarks.

This model is generated in our lab by manual segmentation of a high quality μCT image using 3D Slicer software. The red colour represent scala media and scala vestibuli. The green colour represent the scala tympani.

https://doi.org/10.1371/journal.pone.0264449.g003

For justification, the results were repeated 3 times and the average values were used. The results have been divided into 4 groups based on image modalities to obtain more details on how each method performs on a specific modality. This allowed for more detailed evaluation as some methods may be biased to specific types of images. These groups were (CBCT, CBCT), (CBCT, MR), (CBCT, MDCT), and (MR, MDCT). In addition to these group results, the total results were presented to give a global evaluation of each method. The average RMSE and average time calculated for each registration process in each of the 6 methods were then used to compare (ACIR-v3) to each of the other 5 methods using independent T-tests.

Other optimisers (used for comparisons)

FASGD is another adaptive stochastic optimiser that estimates the step-size automatically using the observed voxel displacement. During the computation of the step-size, two ASGD parameters are fixed in FASGD, while they have to be computed in each iteration in ASGD. This makes FASGD faster but less adaptive than ASGD.

Preconditioned Stochastic Gradient Descent (PSGD) [25] is proposed to improve the rate of convergence of GD methods. It adds a preconditioning strategy to Robbins-Monro stochastic gradient descent (SGD_RM) [11] and ASGD methods. The updating rule for PSGD is: (9)

The equation is similar to Eq (5), the only difference is adding the preconditioner matrix M which is a symmetric positive definite matrix with size equal to the number of the transformation’s parameters, i.e. |μ|. When M is the identity matrix, we get the standard SGD method as in Eq (5). In [25] the step-size is defined to be a non-increasing and non-zero sequence to guarantee convergence: (10) where t₀ = t₁ = 0, , is an approximation of , M is the preconditioner matrix, A = 20 is a decay speed factor, and η is a noise factor and is defined as: (11) where g is the exact gradient and ϵ is a random noise added to the exact gradient. Unfortunately, PSGD is limited to monomodal image registration where the fixed and the moving images are the same type e.g. MDCT. The FPSGD solved this issue and can be used for multimodal image registration. The authors mention that FPSGD is 2 to 5 times faster than SGD methods while retaining the same accuracy [20].

FPSGD estimates the diagonal entries of a preconditioning matrix M on the distribution of voxel displacements. This rescales the registration similarity metric and allows for more efficient optimisation. Let M be a preconditioning matrix with the same length of the parameter vector μ, and m_i is an element in the diagonal of M that can be computed using Eq (12): (12) where δ is a pre-defined value representing the maximum voxel displacement, E||X|| is the expectation of the l₂ norm, Var||X|| is the variance of the l₂ norm, is the i_th column in the Jacobian matrix which is mentioned earlier in Eq (4), and ϵ is a small number to avoid dividing by zero.

Limitation

The image registration field is very rich. There are many methods published every year and one cannot cover all the related work in one paper. Hence we selected related work close to our problem which involves rigid transformation and cochlea images. However there is some interesting work which can be used for problems involving non-rigid image registration and different datasets as in [31–35]. Deep learning methods are increasing in popularity in medical image registration e.g. as in [36, 37]. However, they require expensive specialized hardware and thousands of medical images for training which are often unavailable. Moreover, the accuracy of these methods is poor in some tasks [38].