Introduction

Given the widespread recognition of diffusion models in the fields of finance and economics, diffusion processes are commonly employed to describe economic variables that evolve over time, such as stock prices, stock yields, and options futures [1,2]. The general diffusion process can be mathematically expressed through the following stochastic differential equation:

(1)

where represents a standard Brownian motion, and are the drift coefficient and diffusion coefficient of the process X_t, respectively. Due to the fact that the diffusion process (1) is driven by a Brownian motion, its sample paths exhibit unbounded variation and are nowhere differentiable, thus posing challenges in modeling integrated and differentiated processes. As is well known, the integral stochastic process offers a superior explanation of modern econometric phenomena that manifest as the accumulation of past perturbations. For instance, in a discrete-time context, consider a unit root process where . Notice that y_t can be written as

where . It is evident that the time series exhibits inherent nonstationarity. To overcome this limitation, a common strategy is to employ differential methods. For instance, for the unit root process {y_t}, first-order differencing − y_t−1 can transform the series into a stationary sequence, which is more convenient for further analysis and modeling.

For the continuous case, reference [3] developed the second-order diffusion process, which is defined by the following second-order stochastic differential equation based on [4,5]:

(2)

where X_t is, by hypothesis, a stationary process, is a standard Brownian motion, and denote the drift coefficient and diffusion coefficient respectively. Furthermore, {Y_t} is a differentiable process that can be expressed as  +  The process (2) is analogous to the unit root model and has been applied in finance and physics. Integrals of diffusion processes play a fundamental role in fields such as finance, engineering, and physics. For instance, in financial contexts, they can model asset returns tied to stock prices or exchange rates, capturing the stochastic evolution of market dynamics, as illustrated in works such as [3]. Additionally, in physical systems, X_t may represent the velocity of a particle and Y_t its corresponding position, while in paleoclimatology, such models help reconstruct paleotemperatures from ice-core data, as demonstrated in [6].

Second-order diffusion processes are fundamental in modeling complex stochastic dynamics across disciplines such as mathematical finance (e.g., capturing stochastic volatility in option pricing) and statistical physics (e.g., describing particle motion with memory effects). The transition probability density quantifies the likelihood of state transitions, and is a cornerstone for risk assessment, prediction, and model calibration.

However, if the unknown functions in the models cannot be accurately estimated, they cannot be further utilized, so the statistical inference of the model (2) is a prerequisite for its practical application. Compared with parametric methods, nonparametric estimation captures nonlinear relationships without pre-defined function forms, providing greater flexibility and robustness. Regarding the nonparametric estimation of the second-order diffusion process (2), it is worth noting that for a fixed sampling interval, we can observe , which is the cumulation effect of all past perturbations. However, the instantaneous value of X in model (2) at time t_i cannot be directly derived from these observations . In addition, the condition distribution of Y is usually unknown, even if the distribution of X is known, thereby nonparametric estimation based solely on {Y_t} is not feasible. For let (where − t_i−1), as demonstrated by [3], with discrete-time observations and the relationship

we can approximate the value of X at using

The accuracy of as a proxy for depends on the magnitude of . Throughout the paper, our estimation procedures depend on the sample

For a review of nonparametric statistical inference of the second-order diffusion model (2), [3] developed the Nadaraya-Watson estimation of and , and [7] proposed local linear estimators for the two infinitesimal coefficients. Combining the advantages of these two methods, [8] constructed the re-weighted estimator for the diffusion coefficient. [9] considered a generalized likelihood ratio test of the diffusion coefficient in the second-order diffusion model, and [10] proposed a new nonparametric estimator to correct the bias of the kernel estimator for diffusion coefficient in this model. Furthermore, [11] improved the conditions of [3] and established the strong consistency of the nonparametric kernel estimator.

Transition probability density function can fully characterize the dynamic process of the second-order diffusion model and help solve practical challenges such as financial asset pricing and portfolio selection. Consequently, the estimation of transition probability density has attracted widespread attention. [12] considered the kernel estimator of conditional density in nonparametric regression model, while [13] presented a local linear smoothing method that can be directly used to estimate the transition probability density of diffusion process. [14] adopted the conditional distribution smoother of [15], introduced a re-weighted Nadaraya-Watson smoother of the conditional density function. [16] proposed a new flexible nonparametric estimation technique for conditional density of time series models, which utilizes a regression-based approach and inherits the convergence rates of the selected regression methodology.

Beyond classical nonparametric approaches like Nadaraya-Watson and local linear estimation, modern methods for density estimation include adaptive bandwidth kernel estimation, log-spline density estimation, and wavelet density estimation. Adaptive bandwidth kernel estimation dynamically optimizes smoothing scales based on local data features [17], which can improve local estimation accuracy. However, in the second-order diffusion process, the random fluctuations of state variables (e.g., heteroscedasticity induced by the diffusion term ) can easily lead to excessive iteration of bandwidth selection, amplifying the negative effects of discrete observation errors, and making it difficult to balance the smoothing requirements in both conditioning variable x (current state) and response variable y (future state) directions. Log-spline density estimation achieves nonnegative density constraints through logarithmic transformation and spline basis function approximation, and demonstrates notable advantages in handling multimodal distributions and boundary issues [18]. Nevertheless, it requires preset fixed spline knots and cannot adaptively capture the locally time-varying structure of the transition probability density in second-order diffusion processes (e.g., abrupt changes in density morphology at the turning points of the drift term). Wavelet density estimation relies on multiscale decomposition to optimize the processing of abrupt data [19]. However, its theoretical convergence depends on the stationarity of data and the global completeness of orthogonal basis functions, making it difficult to adapt to the nonstationarity of second-order diffusion processes caused by the drift term and diffusion term . Furthermore, it tends to introduce additional high-frequency noise in the approximation error of state variables (e.g., the deviation between the observed and the true ). The limitations of these methods in terms of scenario adaptability for transition probability density estimation of second-order diffusion processes highlight the necessity of developing targeted estimation methods.

In this paper, we develop a novel nonparametric estimation for the transition probability density function in second-order diffusion process. However, estimating the transition probability density of second-order diffusion processes is inherently challenging. Nonparametric estimation requires balancing bias and variance, and traditional methods have key limitations while laying the foundation. The Nadaraya-Watson estimator treats boundary points and interior points uniformly, which cannot explain the unique boundary conditions of the state space for the second-order diffusion process, resulting in boundary distortion. The fixed bandwidth limitation further limits its adaptability; A single bandwidth cannot capture the local and global features of transition probability density optimally, resulting in over smoothing (masking fine-grained transition patterns) or under smoothing (introducing too much noise). In addition, when the estimator strives to conform to the specific curvature and boundary behavior inherent in second-order diffusion process dynamics, significant asymptotic deviations may occur. The local linear estimation method solves some bias problems through local polynomial fitting, but it sacrifices a key characteristic of density estimation: nonnegativity. According to the definition, density estimation must be nonnegative, but local linear methods may generate false negative values in areas of low data density or complex boundary conditions in the pursuit of reducing bias, making them unsuitable for estimation of transition probability density in second-order diffusion processes.

To overcome these limitations, we turn to the re-weighted method, which has shown promise in other estimation contexts but has never been tailored to the unique requirements of transition probability density estimation for second-order diffusion processes. Our extension of the re-weighted method was initially proposed by [20] in a more general estimation framework. The re-weighted method we propose is not a mere transplantation of existing techniques but a specialized adaptation for transition probability density estimation of second-order diffusion processes. We recognize that the structure of second-order diffusion processes, with their specific stochastic differential equations governing state transitions, imposes different requirements on the estimator. The proposed estimator combines the bias advantage of local polynomial smoothing with the interpretability of classical kernel methods, while ensuring the nonnegativity of density estimation.

The re-weighted method has similarities with the Nadaraya-Watson smoothing method, but it introduces a new weighting scheme that subtly improves the Nadaraya-Watson estimation. [21] used it to investigate nonparametric estimation of regression functions, while [22] used it to estimate the volatility function of diffusion model. [23] developed a modified version of Nadaraya-Watson estimation for the infinitesimal conditional expectation of the second-order jump diffusion model, and [24] considered this re-weighted method for regression mean estimation. Furthermore, [25] utilized this method to estimate the conditional density in right-censored models.

The organization of this paper is outlined as follows. The Methods section establishes the re-weighted estimator for the transition probability density of the second-order diffusion model. The Asymptotic results section lists the necessary conditions and presents asymptotic results of the proposed estimator. The simulation experiment is presented in the Simulation section. The detailed proofs of the main results are given in the Auxiliary results and proofs section. The last section is a conclusion.

Methods

Throughout this paper, we assume that X on a space , . The unknown function is the transition density of given in the model (2), it can be estimated based on joint density p(x, y) and marginal density p(x) (where is assumed positive at x) since we have

The kernel function is a nonnegative and symmetric density function on , satisfies

Let h_n be bandwidth, and . As , we have

(3)

In fact, by Taylor’s expansion, we have

Furthermore, by Taylor’s expansion and Hölder’s inequality, we have

where , and 1 / α + 1 / β + 1 / γ = 1. Selecting and assuming that

by Lemma 1 in the Auxiliary results and proofs section we have

so

Let neglecting the higher order term in (3), which suggests that can be regarded as a regression of − on , so by selecting minimize the weighted sum of squares

we can get the local polynomial estimator of the transition probability density for model (2).

When p = 0, the estimator is known as the Nadaraya-Watson (NW) kernel estimator:

When p = 1, we can get local linear (LL) estimator as follows:

where

According to minimum mean square theory, satisfies

Reference [17] indicates that these moment conditions make the local linear estimator superior to the Nadaraya-Watson estimator in terms of boundary deviation. In order to combine the advantages of Nadaraya-Watson and local linear methods, this paper considers re-weighted methods for transition probability density. This involves making minimal adjustments to the weights of Nadaraya-Watson estimator to ensure the nonnegativity while adhering to the conditions of . The specific form of the re-weighted estimator is detailed as follows:

where the weight function satisfies

subject to

By using Lagrange multipliers method, we can obtain

where λ satisfies

For more details, readers can refer to [8, pp. 1132–1134].

In fact, The kernel function − acts as a smoothing filter that assigns higher weights to historical states which are close to the current state x. Correspondingly, − weights the future states y that are proximate to the observed subsequent state . The re-weighted mechanism further adjusts these kernel weights by incorporating a moment constraint, e.g.,

which effectively corrects for boundary bias. This adjustment proves particularly critical when handling edge cases, such as abrupt yet physically plausible state transitions (e.g., rapid temperature fluctuations), as it prioritizes data sequences that are both statistically influential and dynamically consistent with real-world constraints.

In essence, the re-weighted estimator integrates local data proximity (enforced by the kernels) with structural regularity (enforced by re-weighted method) to produce a physically realistic and statistically efficient estimate of the transition probability from state x to y.

Asymptotic results

The main results of this paper are based on the following conditions.

(i) Let the state space of random process X be . Let z₀ be any point in interval , and be the scale density function. In addition, for , , we assume that

(ii) For , , where is the speed density function;

(iii) X₀ = x has distribution P⁰, where P⁰ is the invariant distribution of process X.

Assumption A1 assures that X is a stationary process ([3]).

Suppose that

Assumption A2 assures that X is ρ-mixing and α-mixing, and under assumptions A1-A2, we can obtain that is a stationary process, readers can refer to [3] for details.

(i) The marginal density function p(x) > 0 is a bounded continuous function and has a continuous first-order derivative;

(ii) The transition probability density function is a bounded continuous function with continuous second-order partial derivatives.

(iii) The joint density p(x,y) is bounded by an independent constant.

Kernel function K(x) is a continuously differentiable, symmetric density function with bounded support and satisfies

The selection of kernel function is a fundamental consideration in nonparametric estimation, and its impact on the proposed re-weighted estimation deserves clear discussion. In practice, various types of kernels can be used, and any density function can serve as a valid kernel, even nonpositive functions have been proven theoretically feasible (see [26]). For the purpose of this study, we restrict our attention to classical positive symmetric kernels (e.g., the Gaussian kernel used in our simulation), due to their computational simplicity and widespread adoption in the literature. It is worth noting that both theoretical analysis and empirical evidence indicate that specific kernel choices have only a weak impact on the performance of kernel based estimators. This insensitivity is due to the asymptotic properties of the estimator, including bias and variance, as described in Theorem 1, which mainly depend on the bandwidth h_n and kernel moments, rather than the exact functional form of the kernel.

Practically, the Gaussian kernel is more popular due to its smoothness and ease of implementation, especially in dealing with boundary effects. For applications that require computational efficiency, alternative kernels (e.g., Epanechnikov) may be substituted without altering the asymptotic properties, as long as the regularity conditions in Assumption A4 are satisfied.

, where m is a positive integer, , .

(i) and have continuous derivatives of order 4 and satisfy and for some .

(ii) , where .

Assumption A6 imposes moment bounds on the drift coefficient , diffusion coefficient and the initial state X₀, which are standard in the literature on nonparametric estimation for diffusion processes (e.g., [3,8]). This assumption is related to the application of Lemma 1 (Auxiliary results and proofs section).

Let denote the σ-field generated by , for all , the mixing coefficient satisfies the following conditions:

(i) For some constant δ, , and , ;

(ii) Assume that there exists a sequence of positive integers s_n such that , , .

Assumption A7 imposes α-mixing condition on {X_t}, a large class of strongly mixing random variables with mixing coefficient satisfies these conditions are included.

(i) , , , as ;

(ii) Assume that , and such that and

(i) The term originates from the modulus continuity of the Brownian motion paths (see [27]), which characterizes the asymptotic magnitude of Brownian motion oscillations at microscopic time scales. It is prevalent in many other studies, serving as a tool to control discretization errors, and it ensures the asymptotic properties of the estimator remain valid as , a result well-established in the literature (see [3, Theorem 3]).

(ii) The bandwidth h_n is a key hyperparameter in nonparametric kernel estimation, it controls the smoothness of the final curve estimation, and its selection directly affects the bias variance trade-off of the proposed re-weighted estimator. The selection of bandwidth requires balancing two conflicting objectives: a smaller bandwidth can reduce bias by closely adapting to local data structures, but it will increase variance due to limited sample information, while a larger bandwidth can smooth noise but may over average the fine-scale features of the data, thereby introducing bias.

Practically, cross-validation (CV) or plug-in methods can be employed to optimize bandwidth selection. As to how to choose the bandwidth the book [17] is recommended.

Under A1-A8, let , , for , as , we have

(i)

(ii) Furthermore, if and , then

where means convergence by distribution.

(i) Under Theorem 1, we know that the expectation and variance of the estimator are as follows:

Compared to [28, Theorem 1, p. 5], the re-weighted estimator shares the variance of the Nadaraya-Watson kernel estimator , the difference in the asymptotic mean squared error depends on the magnitude of the deviation between them. According to [28, p. 5], under the conditions A1-A6 and A8, for , the approximate asymptotic bias of the Nadaraya-Watson estimator is as follows:

Obviously, the re-weighted estimator exhibits superior bias properties in comparison to the Nadaraya-Watson estimator. Firstly, incorporates an additional deviation item, in contrast to . When either or assumes a large value, the deviation of tends to be larger. Consequently, under identical conditions, the asymptotic bias of the re-weighted estimator is smaller than that of the Nadaraya-Watson estimator. Nadaraya-Watson estimator exhibits systematic estimation bias in the boundary region of the data distribution, known as boundary effects, which result in significant deviations between the estimated results and the true values. In a sense, can be regarded as the result of bias reduction of . For a detailed theoretical derivation and in-depth analysis of the bias term, readers can refer to [17], which systematically explains the formation and properties of the bias term in nonparametric estimation.

(ii) Theorem 1 also suggests the criterion for selecting the smoothing bandwidth:

which minimizes the asymptotic mean squared error of the re-weighted estimator. Here C is a constant and can be determined via the cross-validation method. It is noteworthy that, in contrast to the conventional n^−1/5 rate prevalent in univariate density estimation, the optimal bandwidth rate here is n^−1/6, as smoothing is required in both x and y directions.

Simulation

In this section, we perform Monte Carlo experiments to evaluate the estimation performance of the re-weighted estimator. The experiment is based on the following second-order stochastic differential equation:

In the simulation, we select a fixed observation time interval, i.e., , which is subsequently partitioned into 5,000 equal segments, resulting in a corresponding sampling frequency of . To approximate the values of X_t within this interval, we use the Euler-Maruyama method, a widely recognized numerical technique for approximating solutions of stochastic differential equations:

In this section, we consider Gaussian kernel and common bandwidth , where S is the standard deviation of the sample .

Figs 1 and 2 present a comparative analysis of estimation performance at x = 0.1 and , where the exact values are obtained by setting the sampling frequency . As shown in the figures, the re-weighted estimator exhibits a similar symmetry to the true density and other estimators, and indicating a reduced bias compared to the Nadaraya-Watson estimator. This reduction of bias is attributed to the re-weighted estimator’s incorporation of the automatic bias correction mechanism of local linear estimation. Furthermore, the transition probability density attains its maximum value in proximity to the given value of x, and its skewness varies according to x, thus demonstrating the effectiveness of our proposed estimator.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Transition probability density estimators at .

(Nadaraya-Watson estimator (NW), local linear estimator (LL) and re-weighted estimator (RNW)).

https://doi.org/10.1371/journal.pone.0333958.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Transition probability density estimators at .

(Nadaraya-Watson estimator (NW), local linear estimator (LL) and re-weighted estimator (RNW)).

https://doi.org/10.1371/journal.pone.0333958.g002

To compare the performance of re-weighted estimator and local linear estimator in preserving the essential nonnegativity property of density estimates, we simulated the values of different transition probability density estimators at fixed x = 0.2. The results are presented in Table 1, which indicates that the local linear estimator exhibits negative results at certain design points, while the re-weighted estimator consistently provides the nonnegative values for transition probability density.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The estimates of the transition probability density at some design points.

(Nadaraya-Watson estimator , local linear estimator and re-weighted estimator )

https://doi.org/10.1371/journal.pone.0333958.t001

Table 2 compares the mean squared error (MSE) performance of the Nadaraya-Watson estimator (), local linear estimator () and re-weighted estimator () of the transition probability density, indicating superior accuracy of the re-weighted estimator, while all estimators show improved performance with increasing sampling frequency.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The MSE of transition probability density estimators.

(Nadaraya-Watson estimator , local linear estimator and re-weighted estimator )

https://doi.org/10.1371/journal.pone.0333958.t002

Auxiliary results and proofs

[3]. Let Z be a d-dimensional diffusion process determined by the following stochastic integral equation:

where is a d × 1 vector, is a d × d diagonal matrix, and W_t is a d × 1 vector of independent Brownian motions. Suppose and have continuous partial derivatives of order 2s, f(z) is a continuous function with continuous partial derivative of order 2s + 2 defined on and with values in , then

where L is a second-order differential operator determined by equation

and R is a random function of order , and

Here, we consider d = 2. For model (1.2), we have

Based on Lemma 1, we can calculate a variety of mathematical expectations involving , such as (12) and (13) (see the Auxiliary results and proofs section for details).

[8]. Consider

where g is a measurable function defined on × , m is a positive integer. Assume that and A1, A4, A5 hold. If for any , , then

[8]. Let be the density function of the process X_t, , , under the conditions A1-A2, A4-A6, A8, we have

[14]. Consider the model (1), suppose that A1-A4, A6-A8 hold and , , . Then, for , , we have

where

with

and satisfies

Lemma 4 restates Theorem 1 of [14] in the specific context of second-order diffusion processes, establishing the asymptotic properties of the idealized estimator based on the true (unobservable) process {X_t}. This adaptation is rigorously validated under conditions with equivalent effects (mixing conditions A2 and A7, moment bounds A6, kernel smoothing conditions A4) and the infinitesimal moment of the process {X_t}:

A key distinction is that is unobservable in the second-order diffusion processes (2), and the proposed re-weighted estimator is based on , by using Lemma 4 and the difference of and , the proof of Theorem 1 only requires proving the equivalence of the estimators between and .

(i) Let

By Lemma 4, we have

Thus, in order to prove Theorem 1(i)

we only need to prove

According to [8] and [22], we have , . In fact, according to Lemma 2, when , the difference between and has a negligible impact on the kernel function, and by combining the asymptotic properties of the weight function (Lemma 3), it can be concluded that . Therefore, by Dominated Convergence Theorem, we can obtain .

So it is sufficient to prove that

For (4), we have

Thus, we need to prove

From [8], we have

and by Lemma 4,

(6) can be obtained through Lemma 2, and the details can be found in [3, p. 894]. For (5), it is equivalent to proving

(7) can be obtained through Lemma 2, the proofs of (8) and (9) are similar, so the following only proves (8).

where (10) can be obtained through Lemma 2. For (11), by Lemma 1 and Lemma 3, we can get

where the second-to-last equation can be inferred as follows:

In conclusion,

On the other hand,

where . Then

with the same arguments as [3, p. 896], in order to prove , we only need to prove . In fact, by Lemma 1,

so we can get . Since , we have

(ii) Next, we will prove the asymptotic normality of the new estimator. Let

By Lemma 4, we have

From Theorem 1(i), the approximation error satisfies:

and under the condition of Theorem 1(ii) (), we have

Therefore, by Slutsky’s theorem, the sum converges

This completes the proof.

Conclusion

This study considers nonparametric estimation for second-order diffusion processes by developing a novel re-weighted estimator of transition probability density function based on the Nadaraya-Watson kernel estimation and local linear estimation methods. The core of the re-weighted idea is to redistribute weights to follow the prerequisite of local linear estimation without changing the inherent properties of the Nadaraya-Watson estimator. The proposed estimator not only preserves the nonnegativity of density estimation but also incorporates the bias advantage of local linear estimation. In addition, the consistency and asymptotic properties of the estimator under mild conditions are established.

In defining the re-weighted estimator of transition probability density, we used the same bandwidth h_n for both the x-direction and y-direction, the purpose of using a single bandwidth is to simplify the presentation of asymptotic properties (e.g., bias, variance, and asymptotic normality) of the re-weighted estimator, this is consistent with the simplification strategy used in [14] to focus on the essential innovation of the re-weighted method. In the simulation, the state variable of the second-order diffusion process is approximated by the difference of observation , if there are differences in the variability of across different observation intervals (e.g., and ), can be standardized first (with mean 0 and variance 1), and then the single bandwidth (where S is the sample standard deviation after standardization) can be used to ensure the smoothing scale adaptation to the joint distribution characteristics of the two variables. In practical applications, however, it may be necessary to apply different levels of smoothing to each direction. For example, in paleotemperature sequences, if the variability difference between x (current paleotemperature value) and y (subsequent paleotemperature value) is significant, resulting in the need for differentiated smoothing even after standardization. Under such circumstances, the re-weighted estimator in this paper can be redefined into a dual-bandwidth form: introducing the bandwidth h_2,n for the x-direction (corresponding to − ) and the bandwidth h_1,n for the y-direction (corresponding to − ) respectively in the weight calculation. At this time, the nonnegativity and low-bias characteristics of the estimator can still be retained through the original re-weighted mechanism, and only the convergence order conditions of the bandwidth in the asymptotic analysis need to be adjusted (e.g., changing to ).

To enhance the practical applicability of the proposed method in the real world, delving into practical considerations like computational complexity, parameter sensitivity analysis, and performance under more complicated real world data scenarios (e.g., noisy data, irregular sampling intervals) using systematic empirical methods is crucial. This would involve investigating how our method performs compared with existing approaches in handling these practical challenges, and this paper does not conduct such in-depth practical investigations. However, comprehensively tackling these real-world application problems remains a promising direction for future research, being able to more robustly validate the practicality of our method in real-world operating environments.

Theoretical analysis and simulation studies jointly verify the superiority of this estimator. Theoretical analysis shows that the bias of the proposed estimator is indeed smaller than that of the Nadaraya-Watson estimator. And simulation experiments validated these findings, further confirming the advantages of our method.

Acknowledgments

The authors would like to thank the referees for their valuable suggestions, which greatly improved the structure and the presentation of the paper.