2.1 Retinal autofluorescence imaging

As light penetrates the retina, it is largely unscattered and only locally absorbed by retinal chromophores (first, hemoglobin within large retinal vessels, then macular pigments largely in the photoreceptor axons, then the unbleached opsins within the photoreceptor outer segments) before reaching the lipofuscin granules in the RPE. The emitted fluorescence from unoxidized lipofuscin bisretinoids can be excited by a range of visible wavelengths (blue to yellow) to yield long-wavelength emission in the red, which is detected noninvasively by the fluorescence imaging camera. Let Λ and λ be the excitation and emission wavelengths, respectively, and AF(Λ, λ) denotes the measured autofluorescence. The Beer-Lambert law for the double-path yields (1) where D(Λ) and D(λ) are the optical densities of the underlying tissue at wavelengths Λ and λ, respectively, Φ(Λ, λ) is the fluorescence efficiency of lipofuscin, and I(Λ) is the radiant power of the excitation light.

We use two different imaging techniques: First, to measure macular pigment, we developed autofluorescence imaging of the human retina at varying emission and excitation wavelengths by modifying standard fundus cameras [34]. Secondly, to subsequently quantify rod rhodopsin, we record images at a commercial cSLO camera (Heidelberg Retinal Angiograph 4.0) that delivers an average of ≈ 3μW/mm² at 488nm by rapidly scanning a small laser beam (10μm) over the retina. The intensity of excitation in the cSLO is > 100-fold less than in the fundus camera so that each cSLO image is acquired over a 100ms scan with an incremental rhodopsin bleaching (≈ 1%). After ≈ 40s of cSLO imaging, the rhodopsin is completely bleached (> 98%), and its initial attenuation of the excitation light at 488nm is removed. To map the rod rhodopsin from the magnitude of the brightening of the autofluorescence in registered movies, we shall study the quantitative models of bleaching and macular pigment absorption in more detail.

2.2 Rod rhodopsin models

2.3 Macular pigment model

To improve accuracy in rod rhodopsin mapping, we quantify macular pigments from multi-spectral autofluorescence image sets. In the past, we developed noninvasive multi-spectral autofluorescence imaging of the human retina by adding selected interference filter sets to standard fundus cameras, cf. [34, 37–43]. By exciting the fluorescent lipofuscin granules within the RPE, the Beer-Lambert model for the double path penetration enables us to effectively measure the spatial macular pigment distribution from a set of multi-spectral autofluorescence images. Quantitative measurements of macular pigments based on two-wavelength autofluorescence images have been introduced by Delori et al. in [33, 44]. To more robustly analyze the spatial macular pigment distribution, we introduce a multiple-wavelength model that enables more effective self-consistency tests.

Let AF_f(Λ, λ) and AF_p(Λ, λ) be the autofluorescence measured at the fovea and the perifovea, respectively. While AF_f depends on the specific location within the fovea, the term AF_p is often replaced by a circular average at 6 degrees [33]. We denote the optical density of the foveal and perifoveal tissue by D_f and D_p, respectively. Let Φ_f and Φ_p be the fluorescence efficiencies of lipofuscin in the foveal and perifoveal regions. According to Eq (1), the foveal and perifoveal autofluorescence are given by We can assume that the fluorophore at the fovea has the same composition as that at the perifovea (constant shape of its spectrum over ), and that foveal-perifoveal differences in absorption by other pigments (retinal blood, visual pigments, and RPE melanin) are negligible. We also neglect the low macular pigment concentration in the perifoveal region, so that the optical density of macular pigment D_MP at 460nm is the difference D_MP(460nm) = D_f(460nm) − D_p(460nm), [33, 44]. We use the relative extinction coefficient k_MP of macular pigment, scaled to k_MP(460nm) = 1, such that D_MP(λ) = k_MP(λ)D_MP(460nm), and we obtain We choose n pairs of excitation and emission wavelengths and weights such that We apply the above equations for each wavelength pair (Λ_j, λ_j), multiply by ω_j, and add them up to obtain Since the foveal-perifoveal differences in fluorophore composition are negligible, the ratio does not depend on j. Therefore, we can determine D_MP(460nm) by (5)

If we only choose two excitation wavelengths Λ₁ = 480nm and Λ₂ = 520nm with the weights 1 and −1, respectively, then we obtain the formula originally proposed in [33]. Our formula (5) enables several tests for self-consistency by removing or adding wavelength pairs and by changing the weights.

2.4 Combined model

To measure rod rhodopsin in the human retina, we first quantify the macular pigment density D_MP(460) from multi-spectral autofluorescence image sets according to formula (5). The rod rhodopsin density can then be quantified from cSLO autofluorescence movies using the bleaching model. The macular pigment density is a parameter of the bleaching model whose exact specification increases the accuracy. By combining Eq (2) with the simplification of Eq (4), we derive the complete bleaching model (6)

Note that each flash of the fundus camera in macular pigment measurements fully bleaches rhodopsin so that our subsequent multi-spectral images are not affected by the bleaching kinetics and hence do not depend on the rod rhodopsin distribution.

2.5 Bleaching protocol

For recording rhodopsin bleaching cSLO movies, we follow a simple clinical protocol in which the human subject wears vermillion sunglasses (rod-protecting) while waiting and the photographer performs focus adjustment using the infrared reflection imaging (nonbleaching) in the cSLO. A 488nm excited autofluorescence movie (≈ 8 frames per sec, 1 minute long) is started that is recorded from the start with blinks every 10s, which refresh the tear film layer on the cornea. The average photon flux bleaches the rod rhodopsin after > 25s. We record the cSLO movie until steady-state rhodopsin bleaching.

3.1 Numerical rhodopsin extraction

The recorded cSLO movie yields the autofluorescence AF(Λ, λ, t) over time in Eq (6). For notational convenience, we shall denote the model parameters by (7) The extinction coefficients k_MP and k_rh are derived from the literature. We find the rhodopsin density ϱ_rh(0) by numerically determining the parameters α, β, and γ in (8) where T₀ ≈ 0 is the time of initial exposure to light. Note that we have bleaching curves f(t), t ∈ [T₀, T], associated to AF(Λ, λ, t), for each pixel location (x, y). Computed parameters are optimized within the declared model, but may differ from the actual physiological parameters α, β, γ by a small error margin, or differ in certain areas due to physiological phenomena that are not present in the model. So, to avoid confusion, we will denote the numerically recovered parameters by , , and .

Given a set of autofluorescence measurements f(t) satisfying the model for AF(Λ, λ, t), the numerical scheme approximates the bleaching process in the form of the double-exponential law in Eq (8). To avoid the use of double-exponents, we consider a “mean-square-log” deviation between f(t) and AF(Λ, λ, t), (9) and numerically determine the three model parameters , , and that minimize E. We want to point out that this energy is different from the one used in our earlier preliminary work [38, 39], where we directly enforced f(t) ≈ ae^−ce−bt. In the present approach Eq (9), we would like to be close to 1, which implies that should be close to 0.

Remark 3.1 The remainder of the mathematical section is dedicated to develop and validate an iterative algorithm that determines the global minimizer of E in Eq (9). We shall see that the Hessian matrix of E is positive definite throughout, so that zeros of the first partial derivatives not only yield local minimizers but indeed the global minimizer. Therefore, the iterative scheme will indeed converge towards the global minimizer of Eq (9).

3.2 Implementation details

To find the actual minimizers of E, we set p = ln(a) and use a gradient descent algorithm. The latter algorithm can equivalently be derived from solving the Euler-Lagrange equations associated to Eq (9), (10) with the use of an artificial time marching [45]. Indeed, the gradient descent of E can be formulated as a discretization of the following system (11) where a, b and c are now supposed to be functions depending on some artificial time parameter τ. More precisely, the steady-states of Eq (11) satisfy Eq (10) and yield the minimizer of E in Eq (9), so that we can set (12) cf. [45]. To actually compute the steady-states lim_{τ → ∞} a(τ), lim_{τ → ∞} b(τ), and lim_{τ → ∞} c(τ), we use a so-called semi-explicit finite difference scheme [46] that leads to where τ_p, τ_b, and τ_c are the individual step widths and initial values p₀, b₀, and c₀ that still need to be determined. As usual, the integral shall be replaced with the finite sum over the measured time points t₁, …, t_m, i.e., only are available, and the index n+1 indicates the updated value of the solution at the next step, the index n corresponds to the “current” parameter values that were updated at the previous step. The second equation is explicit, i.e., we use b_n on the right-hand side. The first and third equations can be implicit, i.e, the index n+1 is used on the right-hand side, because we are still able to solve the equations by (13) (14) (15) According to Eq (12), the minimizers of Eq (9) are computed from the iterations (13), (14), (15) as (16) We shall verify the legitimacy of the above iterative scheme in Section 3.3.

Before, we still need to discuss the choice of the initial values p₀, b₀, and c₀ for the respective model parameters. To determine their order of magnitude, we utilize a spatially averaged bleaching curve, so that the fitting is numerically stable, and we then use the resulting steady state solutions as initial values for consequent pixelwise processing. The fitting routine is, in fact, applied to the modified image stack, in which each pixel is averaged over its surrounding 8 × 8 pixels. The latter effectively reduces noise, including local distortions due to the eye movements and the mutual image registration.

We also see the importance to determine the macular pigment density a-priori because it influences the spatial distribution of the initial value p₀ = ln(a₀) that corresponds to α in Eq (7). In other words, the macular pigment measurements enable us to use more spatially accurate initial values of p₀ yielding faster convergence of the numerical algorithms, so that overall accuracy is improved.

From the biophysical point of view it appears reasonable to assume that β, being the quotient between the steady-light illuminance I and the bleaching constant L, is constant throughout the image. Thus, we only minimize over p and c, and we shall support this simplification in Section 6.1.

3.3 Mathematical validation of the energy minimization

If we restrict the values of parameters to a closed bounded set of the form a ∈ [0, A], b ∈ [0, B], c ∈ [0, C], then E(a, b, c) in Eq (9) attains its nonnegative minimum. Since we keep the parameter b fixed, we only consider the first and third equations in Eq (11), which are (17) According to a generalization of the Cauchy-Picard-Lindelöf Theorem any initial condition p(0) = p₀ and c(0) = c₀ yields a unique solution of this system of ordinary differential equations for all time provided that all second order partial derivatives of the right hand sides are uniformly bounded. Indeed, the second order partial derivatives in this case are (18) (19) (20) so that the boundedness condition holds. Thus, the system (17) has solutions p(τ) and c(τ) defined for all artificial time τ. We now would like to claim that they converge towards a steady-state as the artificial time τ tends to infinity, and that and , consistently with Eq (12), yield the minimizer of E.

To be precise, if we fix b, then Eq (17) are necessary conditions a minimizer of E must satisfy. To show that the parameters found by solving the system (17) indeed yield the minimum of E, we need to check that the Hessian matrix of E is positive definite. The latter, together with Eq (17), is a sufficient condition and equivalent to Since is always satisfied, we only need to check on the determinant. Its definition and the Cauchy-Schwartz inequality applied to the two functions e^−bt and 1 yield The inequality in these computations becomes an equality only if the two functions e^−bt and 1 are linearly dependent, which would require b = 0. We fix b > 0 anyway, so that the Hessian of E is positive definite. Therefore, there is only one critical point of the energy E in Eq (9), which must be the global minimum. According to Eq (16), the gradient descent (13) and (15) yields this minimizer as n tends to infinity.

4.1 Detecting blood vessels

Retinal blood vessel detection has already been widely addressed in the literature [48–56], and, in principle, we could choose one of the standard tools, see also [57, 58] for related approaches. Nevertheless, we shall describe two semi-automated methods producing a mask of the pixels to be inpainted that directly evolves from the rhodopsin model itself.

1. Let f(x, y, t) denote our image stack, where (x, y) keeps track of the pixel location. For each fixed pair (x, y), the numerical minimization algorithm provides the outputs , , , as well as the number of iterations N(x, y) required for the gradient descent to converge. The “image” or “matrix” N contains edge information that enables tracing contours including the edges of the blood vessels. Thus, the speed of convergence of the numerical minimization scheme provides a tool to detect the retinal blood vessels.
2. The second method relies on the error between measured intensities and reconstructed autofluorescence from the computed model parameters. In other words, we aim to detect the pixel locations (x, y) where the cSLO measurements f(x, y, t) strongly deviate from the proposed bleaching model Eq (8). For instance, we can check the magnitude of . Here, instead of the squared logarithmic deviation in , we use the total L₁ deviation at each pixel location, i.e, where t₁, …, t_m denote the time points associated to the frames in the recorded cSLO movie. Thresholding of the values between varying percentiles enables the identification of pixels, where the model does not fit the actual measurements, hence detecting the blood vessels. We denote those pixel locations by Ω.

4.2 Inpainting technique

Image inpainting is an active mathematical research field and several techniques have been proposed in the literature [59–69]. Here, we need image inpainting to remove the blood vessels as occlusions from the rhodopsin distribution map. Due to its robustness against measurement noise, we shall use a refined variational method developed in [37, 47] for binary images, briefly described in the following. Since the cSLO camera output is 8-bit grayscale, we use bit-wise processing (work with 8 binary images separately) as outlined in [47], which further suppresses distortions effectively. Mathematically, we identify the images with binary functions g:[0, 1]²\Ω → {0,1}, and Ω ⊂ [0, 1]² is the location of the retinal blood vessels to be inpainted, so that we are looking for values of g on Ω. We compute the desired function as a minimizer of the modified (wavelet) Ginzburg-Landau energy, (21) and refer to [47] for the detailed motivation, see also [65]. Briefly, the first term forces the minimizer to be close to g on [0, 1]²\Ω, so that only minimal changes in intensities outside blood vessels are allowed. The second and third term refer to the underlying Ginzburg-Landau model for u, meaning regularity conditions on u and the way g can be extended to Ω. For the reader familiar with such techniques, the term denotes the Besov 1-2-2 norm computed using a wavelet expansion of u, and the details are described in [47]. Such variational methods based on optimization commonly yield much better results than more elementary interpolation techniques.

The pixels of retinal vessels and hence the missing area Ω are not perfectly well-defined due to measurement inaccuracies or micro-movements of the eye during the recordings and resulting image registration artifacts. To compensate for this, we replace the first term in Eq (21) with (22) where the mask χ_Ω : [0, 1]² → [0, 1] smoothly changes from 0 to 1 at the “boundary” of Ω in [0, 1]², thus gradually assigning a lower weight to the forcing u(x, y) ≈ g(x, y) close to the unknown area. The final rhodopsin map then is a result of a two-step procedure. At the first step, three versions of the map were recovered using inpainting with slightly varying parameters of ϵ, μ in Eq (21) and smoothening of the mask χ_Ω in Eq (22)—meaning the information near the vessels was given different levels of priority each time. At the second step, the average of the resulting maps was subject to adaptive binary wavelet thresholding, which eliminates further pixel artifacts, see [47] for details about the technique.

6.1 Numerical algorithm

The parameter β is a fundamental biophysical constant of rhodopsin-photon absorption and thus is presumed to be spatially invariant in our analysis, so that we only optimize over p and c [18, 35]. We shall validate this approach by observing that the optimization over all 3 parameters leads to similar results. It should be mentioned that computed parameters , , and turn out positive without explicit enforcement. When solving Eq (10) for all 3 parameters by applying Eqs (13), (14), and (15), we see only minor variations of b(x, y) away from the blood vessels, the fovea and the optic disc. More precisely, we computed an average of the intensity curves over a large annulus-like area (5–8 degrees) away from the fovea and ran the fitting routine for the averaged experimental curve. A representative histogram of the recovered values of b, cf. Fig 1, shows strong concentration around one value, which we can then assign to b₀.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Histogram of computed parameter b.

We optimized all 3 parameters p, b, and c at the same time and plotted a representative histogram of parameter b. The counts cluster tightly around the mean, enabling us to assign the highest count to b₀, keep it fixed throughout the image and only optimize over the remaining two parameters p and c.

https://doi.org/10.1371/journal.pone.0131881.g001

6.2 Inpainted rhodopsin maps

The main objective of the present methodology paper is to develop the entire cycle of rod rhodopsin quantification, ranging from the biophysical model, the clinical measurements, and all the way to the mathematical optimization procedures and image analysis tools. Indeed, after retinal images are acquired and registered, building the rod rhodopsin maps from cSLO measurements is a 5-step procedure:

1. macular pigment:
   We spatially quantified macular pigment from multi-spectral autofluorescence image sets refined through self-consistency tests in Fig 2.
2. initial values:
   While Fig 3a shows a typical bleaching curve of a single pixel, we compute initial values a₀, b₀, and c₀ from averages over a large spatial area in the autofluorescence movie, cf. Fig 3b. The parameter b₀ is determined as the maximal count in the underlying histogram as described in Section 6.1, and a₀ is spatially modified according to the macular pigment maps.
3. preliminary rhodopsin map:
   We then apply the gradient descent to perform the minimization on 8 × 8 pixel patches to reduce noise. Parameters for pixels near the fovea or near the major blood vessels show atypical behavior and the fitting changes qualitatively, Figs 4, 5. When ignoring such areas, plotting the parameter yields a preliminary rhodopsin map in Fig 6a.
4. vessel detection:
   We detect retinal blood vessels to remove them from the preliminary rhodopsin maps, see Fig 6b.
5. inpainting:
   We inpaint the blood vessels to obtain the final rod rhodopsin map in Fig 7. The deviation between the preliminary rhodopsin map and those after image inpainting, assessed away from the mask, are below 5% in the amplitude, and < 1% in the mean-square error.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Macular pigment measurement.

Two bands of a multi-spectral autofluorescence image set are shown and a representative macular pigment map derived from formula (5), where we optimized the map over several choices of weights to maximize self-consistency of the measurement. The macular pigment is concentrated in the macula and rapidly decays with distance to the center of the fovea. Although vessels and optic disc can still be recognized visually, the pixel magnitudes are so small that their contribution in the further steps is negligible.

(a) blue excitation

(b) yellow excitation

(c) spatial map of MP density

(d) horizontal and vertical MP profiles, averaged over small stripes.

https://doi.org/10.1371/journal.pone.0131881.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Bleaching curve and its fit (I).

Temporal sequences of the intensity values (blue) and the corresponding fit (red) through minimizing E in Eq (9). Due to the low signal to noise ratio in cSLO measurements, there are large variations, but we can still recognize the overall trend. By averaging the cSLO movie over an angulus of 3 degree width, the noise and hence the variation are suppressed and the fit becomes tighter.

(a) typical temporal sequence with its fitted curve at one pixel

(b) temporal sequence from averaging intensities over an annulus (5–8 degrees) and fitted curve.

https://doi.org/10.1371/journal.pone.0131881.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Bleaching curve and its fit (II).

The fit near the fovea does not show the typical bleaching behavior because there are almost no rods present. Instead, the minimization procedure leads to a linear fit. By averaging an 8 by 8 pixel region outside the fovea, we derive a typical bleaching curve with a good fit. All further computations are computed using this 64 pixel averaging to effectively suppress the noise in the cSLO measurements.

(a) non-averaged image stack, curve chosen near the fovea, parameters

(b) averaged over square neighborhoods (8 × 8 pixels).

https://doi.org/10.1371/journal.pone.0131881.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Local fitted curves.

Bleaching curves computed from fitted parameters averaged over several small spatial regions. The blue curve is the fitting for the intensity vector averaged over the annulus-like area between two blue ovals shown in (a). Green, black, and purple curves show similar bleaching kinetics. The yellow curve corresponds to the region close to the fovea where not much rhodopsin is expected and the fit becomes a steady line. Blood vessels do not exhibit the bleaching model although the fit (red) still shows some bleaching behavior. Nevertheless, the intensity values appear far off.

(a) regions marked on the first image of the stack

(b) respective fitted bleaching curves computed using the vectors of the averaged intensities.

https://doi.org/10.1371/journal.pone.0131881.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Rod rhodopsin measurement (I) and Blood vessel detection.

We determined b ≈ 0.04 through the largest count in the histogram, which corresponds to the fit associated to an average of a large annulus area. The gradient descent minimization scheme led to the optimized parameters and , where the macular pigment distribution was used to refine the initial value a₀. Blood vessels show up as artifacts in the resulting rhodopsin maps and must still be removed.

We detect blood vessels through a scheme that evolves from the computational minimization of E related to the rhodopsin model. Spatial regions where the numerical algorithm converges significantly slower than in other image parts shall be identified as blood vessels. The number of required iterations leads to an image which can simply be thresholded to identify retinal blood vessels.

(a) spatial map of parameter optimal within the model. Brighter means increase in rhodopsin density, but brightest spots occur as registration artifacts along blood vessels. Therefore, the grayscale colormap is suppressed.

(b) extracted mask of blood vessels derived from thresholding the number of iterations needed for convergence in the optimization scheme. Simple pixel growth in the mask would lead to complete coverage of the vessels.

https://doi.org/10.1371/journal.pone.0131881.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Rod rhodopsin measurement (II).

To derive the final rhodopsin map, we first detect the blood vessels. Instead of a binary vessel mask, we use a gradual interface at vessel borders resulting in a smooth vessel mask as proposed in Eq (22). Inpainting based on Eq (21) was used to remove the retinal blood vessels from the final rhodopsin map leading to a smooth rod rhodopsin map. We show the computed parameter γ, which is the sum of emission and excitation rhodopsin absorbance. Since the emission rhodopsin absorbance (590nm-600nm) is neglible, γ indeed is the rhodopsin absorbance at 488nm. The center of the fovea lacks rhodopsin whose density increases when moving apart from the center. Consistently with the rod distribution described in [8], rod rhodopsin increases most rapidly along the superior vertical meridian and increases least rapidly along the nasal horizontal meridian. Although we do not see a connected hot spot of highest rod rhodopsin density, we observe larger and more connected areas of highest rhodopsin density in the superior retina than in the inferior retina, again, consistent with the rod distribution in [8].

(a) cropped rhodopsin map from Fig 6a to be inpainted

(b) smooth mask χ used in the spatial consistency forcing term

(c) the final output showing the distribution of the rhodopsin optical density γ, i.e., the rhodopsin absorbance at 488nm. Units are suppressed to indicate that we compute its distribution rather than absolute quantitative optical densities as we are not able to validate overall amplitudes due to background in our image sets.

(d) horizontal and vertical rhodopsin profiles, averaged over small stripes.

https://doi.org/10.1371/journal.pone.0131881.g007

6.3 Rhodopsin maps compared to rod photoreceptor topology

We compare our rhodopsin distribution maps with typical rod topology as characterized in [7–9]. Note that we compare the distribution meaning comparison of actual densities in a qualitative but not quite quantitative fashion. Thus, local rhodopsin changes can be observed but the overall amplitude of our maps is arbitrary to some extent, so that we suppress units in our figures. There are no rods nor rhodopsin present in the fovea, and rhodopsin density increases with distance to the foveal center, Fig 7c. As observed in rod photoreceptor cells [8], the rhodopsin density increases most rapidly along the superior vertical meridian and less rapidly along the nazal horizontal meridian, see Fig 7d.