The origins of Fisher information are in its use as a performance measure for parametric estimation. We augment this and show that the Fisher information can characterize the performance in several other significant signal processing operations. For processing of a weak signal in additive white noise, we demonstrate that the Fisher information determines (i) the maximum output signal-to-noise ratio for a periodic signal; (ii) the optimum asymptotic efficacy for signal detection; (iii) the best cross-correlation coefficient for signal transmission; and (iv) the minimum mean square error of an unbiased estimator. This unifying picture, via inequalities on the Fisher information, is used to establish conditions where improvement by noise through stochastic resonance is feasible or not.
Citation: Duan F, Chapeau-Blondeau F, Abbott D (2012) Fisher Information as a Metric of Locally Optimal Processing and Stochastic Resonance. PLoS ONE 7(4): e34282. doi:10.1371/journal.pone.0034282
Editor: Matjaz Perc, University of Maribor, Slovenia
Received: February 1, 2012; Accepted: February 25, 2012; Published: April 6, 2012
Copyright: © 2012 Duan et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work is sponsored by the NSF of Shandong Province, China (No. ZR2010FM006). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Fisher information is foremost a measure of the minimum error in estimating an unknown parameter of a probability distribution, and its importance is related to the Cramér-Rao inequality for unbiased estimators , . By introducing a location parameter, the de Bruijn's identity indicates that the fundamental quantity of Fisher information is affiliated with the differential entropy of the minimum descriptive complexity of a random variable . Furthermore, in known weak signal detection, a locally optimal detector, acting as the small-signal limited Neyman-Pearson detector, has favorable properties for small signal-to-noise ratios . With sufficiently large observed data and using the central limit theorem, it is demonstrated that the locally optimal detector is asymptotically optimum and the Fisher information of the noise distribution is the upper bound of the asymptotic efficacy –. For weak random signal detection, the second order Fisher information is also associated with the maximum asymptotic efficacy of the generalized energy detector –.
However, the fundamental nature of Fisher information is not adequately recognized for processing weak signals. To extend the heuristic studies of –, in this paper, we will theoretically demonstrate that, for a weak signal buried in additive white noise, the performance for locally optimal processing can be generally measured by the Fisher information of the noise distribution. We show this for the following signal processing case studies: (i) the maximum output signal-to-noise ratio for a periodic signal; (ii) the optimum asymptotic efficacy for signal detection; (iii) the best cross-correlation coefficient for signal transmission; and (iv) the minimum mean square error of an unbiased estimator. The physical significance of Fisher information is that it provides a unified bound for characterizing the performance for locally optimal processing. Furthermore, we establish the Fisher information condition for stochastic resonance (SR) that has been studied for improving system performance over several decades –. In our recent work , it is established that improvement by adding noise is impossible for detecting a weak known signal. Here, based on Fisher information inequalities, we further prove that SR is not applicable for improving the performance of locally optimal processing in the considered cases (i)–(iv). This result generalizes a proof that existed previously only for a weak periodic signal in additive Gaussian noise , . However, beyond these restrictive conditions, the observed noise-enhanced effects –, , – show that SR can provide a signal processing enhancement using the constructive role of noise. The applications of SR to nonlinear signal processing are of practical interest.
In many situations we are interested in processing signals that are very weak compared to the noise level , , . It would be desirable in these situations to determine an optimal memoryless nonlinearity in the following study cases.
Output signal-to-noise ratio for a periodic signal
First, consider a static nonlinearity with its output(1)where the function is a memoryless nonlinearity and the input is a signal-plus-noise mixture . The component is a known weak periodic signal with a maximal amplitude () and period . Zero-mean white noise , independent of , has probability density function (PDF) and a root-mean-square (RMS) amplitude . It is assumed that has zero mean under , i.e. , which is not restrictive since any arbitrary can always include a constant bias to cancel this average . The input signal-to-noise ratio for can be defined as the power contained in the spectral line divided by the power contained in the noise background in a small frequency bin around , this is(2)with indicating the time resolution or the sampling time in a discrete-time implementation and the temporal average defined as . Here, we assume the sampling time and observe the output for a sufficiently large time interval of () . Since is periodic, is in general a cyclostationary random signal with period . Similarly, the output signal-to-noise ratio for is given by(3)with nonstationary expectation and nonstationary variance .
In the case of , we have a Taylor expansion of the expectation at a fixed time as(4)where we assume the derivatives and exist for almost all (similarly hereinafter) , . Thus, we have(5)where and , compared with , can be neglected as () , . The above derivations of Eqs. (4) and (5) are exact in the asymptotic limit for weak signals, and have been generally adopted in , .
Substituting Eqs. (4) and (5) into Eq. (3), we have(6)where the expectation is simply the Fisher information of the noise PDF , , and the equality occurs as(7)by the Cauchy-Schwarz inequality for a constant , .
Noting Eqs. (2) and (6), the output-input signal-to-noise ratio gain is bounded by(8)with equality achieved when takes the locally optimal nonlinearity of Eq. (7). Here, for a standardized PDF with zero mean and unity variance , the scaled noise has its PDF and the Fisher information satisfies , . It is known that a standardized Gaussian PDF has the minimal Fisher information and any standardized non-Gaussian PDF has the Fisher information . It can be seen that, the linear system has its output signal-to-noise ratio in Eq. (3). Thus, the output-input signal-to-noise ratio gain in Eq. (8) also clearly represents the expected performance improvement of the nonlinearity over the linear system .
Optimum asymptotic efficacy for signal detection
Secondly, we consider the observation vector of real-valued components by(9)where the components form a sequence of independent and identically distributed (i.i.d.) random variables with PDF , and the known signal components are with the signal strength . For the known signal sequence , it is assumed that there exists a finite (non-zero) bound such that , and the asymptotic average signal power is finite and non-zero, i.e. . Then, the detection problem can be formulated as a hypothesis-testing problem of deciding a null hypothesis () and an alternative hypothesis () describing the joint density function of with(10)Consider a generalized correlation detector(11)where the memoryless nonlinearity has zero mean under , i.e. . In the asymptotic case of and , the test statistic , according to the central limit theorem, converges to a Gaussian distribution with mean and variance under the null hypotheses . Using Eqs. (4) and (5), is asymptotically Gaussian with mean and variance under the hypothesis .
Given a false alarm probability , the asymptotic detection probability for the generalized correlation detector of Eq. (11) can be expressed as , (12)with and its inverse function , . Thus, for fixed and (since the signal is known), is a monotonically increasing function of the normalized asymptotic efficacy given by (13)with equality being achieved when in Eq. (7). This result also indicates that the asymptotic optimal detector is just the locally optimal detector established by the Taylor expansion of the likelihood ratio test statistic () in terms of the generalized Neyman-Pearson lemma , .
Interestingly, with achieved by a linear correlation detector ( in Eq. (11)) as a benchmark , , the asymptotic relative efficiency(14)provides an asymptotic performance improvement of a generalized correlation detector over the linear correlation detector when both detectors operate in the same noise environment , .
Next, consider the weak random signal components has PDF with zero mean and variance in the observation model of Eq. (9) , . Here, the signal components are i.i.d. Then, this random signal hypothesis test becomes for determining whether the random signal is present or not. Consider a generalized energy detector(15)where we also assume , and then . Furthermore, in the asymptotic case of , the expectation (16)Thus, the efficacy of a generalized energy detector is defined as (17)where is treated as the signal strength parameter and is the second order Fisher information , . It is noted that the equality of Eq. (17) is achieved as for a constant . Given a false alarm probability , the asymptotic detection probability for the generalized energy detector of Eq. (15) is a monotonically increasing function of the efficacy –.
Cross-correlation coefficient for signal transmission
Thirdly, we transmit a weak aperiodic signal through the nonlinearity of Eq. (1) . Here, the signal is with the average signal variance , the zero mean and the upper bound A (). For example, can be a sample according to a uniformly distributed random signal equally taking values from a bounded interval. The input cross-correlation coefficient of and is defined as , (18)Using Eqs. (4) and (5), the output cross-correlation coefficient of and is given by(19)which has its maximal value as of Eq. (7). Then, the cross-correlation gain is bounded by(20)
Mean square error of an unbiased estimator
Finally, for the observation components , we assume the signal are with an unknown parameter . As the upper bound (), the Cramér-Rao inequality indicates that the mean squared error of any unbiased estimator of the parameter is lower bounded by the reciprocal of the Fisher information ,  given by(21)which indicates that the minimum mean square error of any unbiased estimator is also determined by the Fisher information of a distribution, as is given.
Therefore, just as the Fisher information represents the lower bound of the mean squared error of any unbiased estimator in signal estimation , , the physical significance of the Fisher information () is that it provides a unified upper bound of the performance for locally optimal processing in the considered signal processing cases.
Aiming to explain the upper bound of the performance for locally optimal processing as Fisher information, we here show an illustrative example in Fig. 1. Consider the generalized Gaussian noise with PDF(22)where , for a rate of exponential decay parameter , . The corresponding locally optimal nonlinearity is and the output-input signal-to-noise ratio gain in Eq. (8) is (solid line), as shown in Fig. 1. For comparison, we also operate the sign nonlinearity and the linear system in the generalized Gaussian noise. The output-input signal-to-noise ratio gain in Eq. (8) of is (dashed line), as shown in Fig. 1. For the linear system , Eq. (8) indicates that (dotted line) for , as plotted in Fig. 1. It is seen in Fig. 1 that, only for , the performance of attains that of the locally optimal nonlinearity of . This is because, the nonlinearity is just the locally optimal nonlinearity for Laplacian noise (), and the Fisher information limit is achieved. Likewise, for Gaussian noise (), the linear system is optimal and the output-input SNR gain . It is noted that the above analyses are also valid for the asymptotic relative efficiency of Eq. (14) and the cross-correlation gain of Eq. (20).
. The output-input signal-to-noise ratio gain versus the exponential decay parameter of the generalized Gaussian noise for the locally optimal nonlinearity (solid line), the sign nonlinearity (red line) and the linear system (dotted line), respectively.
Fisher information condition for stochastic resonance
Stochastic resonance (SR), being contrary to conventional approaches of suppressing noise, adds an appropriate amount of noise to a nonlinear system to improve its performance –. SR emerged from the field of climate dynamics , and the topic has flourished in physics – and neuroscience , , . The notion of SR has been widened to include a number of different mechanisms , , , and SR effects have also been demonstrated in various extended systems –,  and complex networks –, .
An open question concerning SR is that, under the asymptotic cases of weak signal and large sample size, can SR play a role in locally optimal processing? Here, based on the Fisher information inequalities, we will demonstrate that SR is inapplicable to performance improvement for locally optimal processing.
For a given observation , we add the extra noise , independent of the initial noise and the signal , to . Then, the updated data . Here, the composite noise has a convolved PDF(23)where is the PDF of noise . Currently, the weak signal is corrupted by the composite noise , and then the performance measures of locally optimal processing in Eqs. (6), (13), (17), (19) and (21) should be replaced with (). It can be shown by the Cauchy-Schwarz inequality that (24)(25)This is because that, if , then using and the Cauchy-Schwarz inequality (26)Similarly, substituting into Eq. (26), we also obtain of Eq. (25).
Therefore, in asymptotic cases of weak signal and large sample size, Eqs. (24) and (25) show that SR cannot improve the performance of the above four locally optimal processing cases by adding more noise. However, the asymptotic limits of weak signal and large sample size are well delimited, and may not be met in practice. It is interesting to note that, under less restrictive conditions, noise-enhanced effects have been observed in fixed locally optimal detectors , suboptimal detectors , , the optimal detector with finite sample sizes  or non-weak signals , , soft-threshold systems  and the dead-zone limiter detector  by utilizing the constructive role of noise.
We here present an illustrative example of SR that occurs outsides restrictive conditions, where a suboptimal detector is adopted for Gaussian noise. Consider a generalized correlation detector of Eq. (11) based on the dead-zone limiter nonlinearity(27)with response thresholds at . For the generalized Gaussian noise of Eq. (22), the normalized asymptotic efficacy in Eq. (13) of can be rewritten as(28)where is the cumulative distribution function of the standardized generalized Gaussian noise PDF . For a fixed response threshold ( without loss of generality), we plot the the normalized asymptotic efficacy (solid line) of the dead-zone limiter nonlinearity as a function of the RMS amplitude of Gaussian noise (), as shown in Fig. 2. It is clearly seen in Fig. 2 that the SR effect appears, and achieves its maximum at a non-zero level of . If the original Gaussian noise RMS , we can add independent Gaussian noise with its RMS amplitude to increase to the maximum . However, is a suboptimal nonlinearity for Gaussian noise, and the locally optimal detector is the linear correlation detector based on the linear system in Eq. (11). It is seen in Fig. 2 that can not overperform (dashed line), even we can add the appropriate amount of noise to exploit constructive role of noise in .
In this paper, for a weak signal in additive white noise, it is theoretically demonstrated that the optimum performance for locally optimal processing is upper bounded by the Fisher information of the noise distribution, and this is uniformly obtained in (i) the maximum output signal-to-noise ratio ratio for a periodic signal; (ii) the optimum asymptotic efficacy for signal detection; (iii) the best cross-correlation coefficient for signal transmission; and (iv) the minimum mean square error of an unbiased estimator. Based on the Fisher information inequalities, it is demonstrated that SR cannot improve locally optimal processing under the usual conditions. However, outside these restrictive conditions of weak signal and large sample size, improvement by addition of noise through SR can be achieved, and becomes an attractive option for nonlinear signal processing. The analysis in the paper has focused on the simplest case of additive white noise as an essential reference, and an interesting extension for future work is to examine the affect of considering different forms of colored noise , , .
Under the assumption of weak signals, the Taylor expansion of the noise PDF is utilized in Eqs. (4), (5), (16) and (21). The Cauchy-Schwarz inequality is extensively used in Eqs. (6), (13), (17), (19) and (26).
Conceived and designed the experiments: FD. Performed the experiments: FD FCB DA. Analyzed the data: FD FCB DA. Contributed reagents/materials/analysis tools: FCB DA. Wrote the paper: FD. Proofreading: FCB DA.
- 1. Cover TM, Thomas JA (1991) Elements of Information Theory. New York: Wiley.
- 2. Kay S (1998) Fundamentals of Statistical Signal Processing. N.J.: Prentice-Hall, Englewood Cliffs.
- 3. Capon J (1961) On the asymptotic efficiency of locally optimum detectors. IRE Transactions on Information Theory IT-7: 67–71.
- 4. Poor HV, Thomas JB (1978) Locally optimum detection of discrete-time stochastic signals in nongaussian noise. Journal of the Acoustical Society of America 63: 75–80.
- 5. Lu NH, Eisenstein B (1981) Detection of weak signals in non-gaussian noise. IEEE Transactions on Information Theory IT-27: 755–771.
- 6. Kassam SA (1988) Signal Detection in Non-Gaussian Noise. New York: Springer-Verlag.
- 7. Song I, Kassam SA (1990) Locally optimum detection of signals in a generalized observation model: The random signal case. IEEE Transactions on Information Theory 36: 516–530.
- 8. Benzi R, Sutera A, Vulpiani A (1981) The mechanism of stochastic resonance. Journal of Physics A: Mathematical and General 14: L453–L457.
- 9. Zozor S, Amblard PO (2003) Stochastic resonance in locally optimal detectors. IEEE Transactions on Signal Processing 51: 3177–3181.
- 10. Chapeau-Blondeau F, Godivier X (1997) Theory of stochastic resonance in signal transimission by static nonlinear systems. Physical Review E 55: 1478–1495.
- 11. Chapeau-Blondeau F, Rousseau D (2009) Raising the noise to improve performance in optimal processing. Journal of Statistical Mechanics: Theory and Experiment 1: no. P01003.
- 12. Dykman MI, McClintock PVE (1998) What can stochastic resonance do? Nature 391: 344.
- 13. Collins JJ, Chow CC, Imhoff TT (1995) Stochastic resonance without tuning. Nature 376: 236–238.
- 14. Bezrukov SM, Vodyanoy I (1997) Stochastic resonance in non-dynamical systems without response thresholds. Nature 385: 319–321.
- 15. Gammaitoni L, Hänggi P, Jung P, Marchesoni F (1998) Stochastic resonance. Reviews of Modern Physics 70: 233–287.
- 16. Wiesenfeld K, Moss F (1995) Stochastic resonance and the benefits of noise: from ice ages to crayfish and SQUIDs. Nature 373: 33–36.
- 17. Zaikin AA, Kürths J, Schimansky-Geier L (2000) Doubly stochastic resonance. Physical Review E 85: 227–231.
- 18. Perc M, Gosaka M, Kralj S (2008) Stochastic resonance in soft matter systems: combined effects of static and dynamic disorder. Soft Matter 4: 1861–1870.
- 19. Gosaka M, Perc M, Kralj S (2012) The impact of static disorder on vibrational resonance in a ferroelectric liquid crystal. Molecular Crystals & Liquid Crystals 553: 13–20.
- 20. McDonnell MD, Abbott D (2009) What is stochastic resonance? definitions, misconceptions, debates, and its relevance to biology. PLoS Computational Biology 5: no. e1000348.
- 21. Perc M (2007) Stochastic resonance on excitable small-world networks via a pacemaker. Physical Review E 76: no. 066203.
- 22. Perc M (2008) Stochastic resonance on weakly paced scale-free networks. Physical Review E 78: no. 036105.
- 23. Wang Q, Perc M, Duan Z, Chen G (2009) Delay-induced multiple stochastic resonances on scalefree neuronal networks. Chaos 19: no. 023112.
- 24. Gan C, Perc M, Wang Q (2010) Delay-aided stochastic multiresonances on scale-free fitzhughnagumo neuronal networks. Chinese Physical B 19: no. 040508.
- 25. Stocks NG (2000) Suprathreshold stochastic resonance in multilevel threshold systems. Physical Review Letters 84: 2310–2313.
- 26. Kay S (2000) Can detectabilty be improved by adding noise? IEEE Signal Processing Letters 7: 8–10.
- 27. Ward LM, MacLean SE, Kirschner A (2010) Stochastic resonance modulates neural synchronization within and between cortical sources. PLoS ONE 5: no. e14371.
- 28. Duan F, Chapeau-Blondeau F, Abbott D (2011) Fisher-information condition for enhanced signal detection via stochastic resonance. Physical Review E 84: no. 051107.
- 29. Chen H, Varshney PK, Michels JH, Kay SM (2007) Theory of the stochastic resonance effect in signal detection: Part I–fixed detectors. IEEE Transactions on Signal Processing 55: 3172–3184.
- 30. Greenwood PE, Müller UU, Ward LM (2004) Soft threshold stochastic resonance. Physical Review E 70: no. 051110.
- 31. Zeng L, Li J (2011) Residual aperiodic stochastic resonance in Lévy noise. Fluctuation and Noise Letters 10: 223–229.
- 32. Hänggi P, Jung P, Zerbe C, Moss F (1993) Can colored noise improve stochastic resonance? Journal of Statistical Physics 70: 25–47.
- 33. DeWeese M, Bialek W (1995) Information flow in sensory neurons. Nuovo Cimento 17D: 733–741.
- 34. Brown LD (1982) A proof of the central limit theorem motivated by the cramér-rao inequality. In: Kallianpur G, Krishnaiah PR, Ghosh JK, editors. Statistics and Probability: Essays in Honor of C R Rao. North-Holland Publishing Company. pp. 141–148.