2.1 Composite Pareto I models

Let θ be the collection of all parameters and let h(x|η) be a probability density function (pdf) with a collection of parameters η and cumulative distribution function (cdf) H(x|η). We call this the lower data distribution. The composite Pareto I model has the following pdf: (1) where ρ > 0 is the proportion of data coming from Pareto I distribution with pdf f_P1(x|τ, α), τ > 0 is the threshold that separates the lower and upper data, and α > 0 is the tail index for the Pareto type I distribution. Note that In the pdf (1), any observation smaller than the threshold τ is distributed according to the lower data distribution, and any observation bigger than τ is distributed according to the Pareto type I distribution. Note that the pdf (1) may not be continuous and differentiable at τ. To make it continuous, we can specify And to further make it differentiable, we let (2) condition on τh′(τ|η) < −h(τ|η) so that α > 0. This simplifies the condition on ρ to (3)

2.2 Composite Pareto II models

Composite Pareto II models are similar to composite Pareto I models except for the use of Pareto type II distribution for the upper data. As before, let θ be the collection of all parameters, h(x|η) be a pdf with its cdf H(x|η), and η be a collection of parameters associated with h(x|η). Then the composite Pareto II models have the following pdf: (4) where ρ > 0 is the proportion of data coming from Pareto type II distribution with pdf f_P2(x|τ, α, β), τ > 0 is the threshold that separates lower and upper data distributions, whereas α > 0 and β > 0 are the tail index and scale parameters of the Pareto type II distribution, respectively. The pdf of the Pareto type II is: The Pareto type II distribution is also a special case of generalized Pareto distribution with positive scale parameter.

In this case, observations with value less than τ is assumed to follow the lower data distribution while any observation greater than τ follows the Pareto type II distribution. To make pdf (4) continuous at τ, we can specify And for differentiability condition, let (5) condition to h′(τ|η) < 0. This simplifies the condition on ρ to (6)

The twice differentiability property can be satisfied by specifying the value of α. However it is not a focus here due to it being too restrictive [9]. In this paper, focus will be given to the composite models with continuous and differentiable pdf. Going forward, when composite Pareto models are mentioned, we refer to composite Pareto models with smooth pdf, unless specified otherwise.

2.3 Bulk models

The previous subsections deal with the general case of composite Pareto models. In the general model, the parameter ρ is free, or is specified to satisfy the continuity and differentiability conditions. However, in some cases, we may specify: (7) for simplification. This specification is referred to as the “bulk model based tail fraction” by Scarrot [15] and is commonly found in the extreme value literature. The term “bulk model” refers to the lower data distribution that is assumed to covers the bulk of the observations when compared to the Pareto distribution. It was first proposed by Behrens et al. [16] as a way to estimate the threshold value for extreme value data and improvements have been made since then [17–19]. In this paper, we refer composite Pareto models with specification (7) as simply the bulk models.

Historically, the bulk models are developed separately with the general composite Pareto models in the previous subsections. There are two main differences in the development of this two models. First, the general composite Pareto models appear in the actuarial science literature while the bulk models appear in the extreme value literature. Second, the general composite Pareto models typically use frequentist approach to estimate the parameters while the bulk models use Bayesian approach for parameter estimation. Readers interested in the bulk models can refer to [15] which gives an outline on the developments and bibliography related to the bulk models. We include this special case of composite Pareto models since the results shown in this paper still apply to the bulk models.

For bulk models with Pareto I tail, the pdf can be written as: (8) Whereas for bulk models with Pareto II tail, the pdf is: (9)

Note that in the literature covering the bulk models, the continuity and differentiability of the pdf are usually not imposed. This is to allow for more flexibility to model the lower data distribution. In some cases, researchers may even use semi-parametric or nonparametric approach for specifying h(x|η) [17, 20, 21].

However, if desired, the continuity condition for bulk models can be imposed by specifying (10) or (11) for bulk models with Pareto I and Pareto II tail, respectively.

Differentiability condition is more complicated here as it requires specifying parameter in the lower data distribution. For example, the pdf (8) is differentiable when Eq (10) is satisfied and If h(x|η) is a lognormal distribution with log-mean μ and log-variance σ², then the above condition simplifies to which does not have a solution for finite σ. Therefore in this case the differentiability condition cannot be met.

Similarly, the pdf (9) is differentiable when Eq (11) is satisfied and condition to h(τ|η)² + h′(τ|η)[1 − H(τ|η)] < 0. However, this condition may not be met by all lower data distributions. For example, if the lower data distribution is lognormal, then it can be proven that the condition cannot be met unless the log-variance is greater than 1.

3.1 Unique solution of the threshold

Fortunately, the uniqueness of the solutions to Eqs (12) and (13) can be proven for many commonly used lower data distributions including gamma, lognormal, normal, and Weibull distributions. These distributions have the property that h′(x|η) = h(x|η)g(x|η) for some function g(x|η). The following two theorems provide a way to determine for the uniqueness of the solution to (12) and (13) with proofs shown in S1 Appendix.

Theorem 1. Suppose for some function g(x|η). Suppose the solution to exists. If g(τ|η) + τg′(τ|η) < 0 for all τ that satisfies h(τ|η) + τh′(τ|η) < 0, then the solution above must be unique, and the relationship between τ and ρ is one-to-one.

Theorem 2. Suppose for some function g(x|η). Suppose that the solution to exists. If h′(τ|η) + H(τ|η)[g′(τ|η) − g(τ|η)²] < 0 for all τ that satisfies h′(τ|η) < 0, then the solution above must be unique, and the relationship between τ and ρ is one-to-one.

Theorem 1 can be used for the composite Pareto I models while Theorem 2 is useful for composite Pareto II models when checking for the uniqueness of the solution to Eqs (12) and (13). Under Theorem 1, for lower data distribution with h′(x|η) = h(x|η)g(x|η) for some function g(x|η), it is enough to show that g(τ|η) + τg′(τ|η) < 0 to determine the uniqueness of solution to Eq (12). For composite Pareto II models, it is enough to show that h′(τ|η) + H(τ|η)[g′(τ|η) − g(τ|η)²] < 0 to show the uniqueness of the solution to Eq (13). Following Theorem 2, the condition to determine the uniqueness of the solution can be further simplified as shown in the following corollary.

Corollary 1. Suppose h′(x|η) = h(x|η)g(x|η) for some function g(x|η). If g′(τ|η) − g(τ|η)² < 0, then the solution to must be unique

As an example, suppose h(x|η) = f_LN(x|μ, σ²) be the pdf of a lognormal distribution with log-mean μ and log-variance σ². Then

Therefore we can write It follows that . Therefore using Theorem 1, the solution to Eq (12) is unique and there is no need to check for multiple solutions to τ for given values of ρ and other parameters when using lognormal-Pareto I composite model.

Additionally, let z = log(x) − μ. Let φ(x) and Φ(x) be the pdf and cdf of the standard normal distribution, respectively. It can be shown that (14) If z ≥ 0, then the equation above is negative and the condition for Theorem 2 is satisfied. However if z < 0, we would have to use the property of Mill’s ratio to show that the condition is still valid. Gordon [22] has proven that the Mill’s ratio for x > 0. If z < 0, it follows that and This combines with Eq (14) satisfies the condition for Theorem 2. Therefore, there will not be instances of multiple τ solution for (13) and prior information on ρ can be used without problem for lognormal-Pareto II composite model.

3.2 Multiple solutions of the threshold

As mentioned previously, it is possible to have multiple solutions to τ given the values of ρ and other parameters. When such incidence happens, an analyst have the choice to either avoid the problem, or choose one of the solutions to τ. The former choice can be done by either using prior information on τ, use lower data distribution with proven solution uniqueness as shown in the previous subsection, or to use the bulk models.

If the analyst opt to select one of the solutions to Eqs (12) or (13), a naive way to select the value of τ out of the multiple solutions is to choose one at random or the highest value out of the solutions. However, this can lead to bias in estimation and should be avoided. A better method is to let the data to guide the selection of τ by choosing the value of τ out of the possible solutions that maximizes the loglikelihood. Therefore the equations are now (15) (16) for composite Pareto I and composite Pareto II models, respectively, where l(θ|x) is the loglikelihood.

This selection method assumes that the ‘true’ solution of τ should be the one that gives the highest likelihood. The solution is not to be misunderstood as the maximum likelihood estimate as the number of possible solution of τ is limited by the conditions (3) or (6). However, in Bayesian estimation, when the sample size grows, the solution to Eqs (15) and (16) should approach to the maximum likelihood estimates.

3.3 Computation to find the value of threshold parameter

Typically, numerical methods have to be used to compute the solution to Eqs (12) and (13). It is also important to consider the condition for the solution of Eqs (12) and (13) when computing the value of τ. Namely, the solution must satisfy τh′(τ|η) < −h(τ|η) and h′(τ|η) < 0 for composite Pareto I and II models, respectively. Without considering the restriction on τ, the function ψ₁(τ, η) or ψ₂(τ, η, α) may diverge to ±∞, making the numerical computation difficult.

To compute the value of τ given the values of other parameters, we use the following functions. (17) for composite Pareto I model, and (18) for composite Pareto II model. The roots of these two functions give the values of τ that satisfy Eqs (12) and (13), respectively.

The first two lines in Eqs (17) and (18) are added for numerical stability. These lines are useful when h(τ|η) is computationally 0 and H(τ|η) is computationally 0 or 1. The third lines in Eqs (17) and (18) are useful to impose the restrictions on τ as mentioned. The values are set to be ρ − 1 since when τh′(τ|η) + h(τ|η) → 0 from the left, then ψ₁(τ, η) → 1. Similarly, when h′(τ|η) → 0 from the left, then ψ₂(τ, η, α) → 1. Setting the values to be ρ − 1 makes κ₁(τ|ρ, η) and κ₂(τ|ρ, η, α) to be continuous functions.

There may be multiple roots to κ₁(τ|ρ, η) and κ₂(τ|ρ, η, α) which indicates the incidence of multiple τ solutions. This proves to be difficult since most root-finding algorithms output only a single root. To find the values of τ, we use the uniroot.all function from the rootSolve package in R, which divides the domain of τ (for a given lower and upper bounds) into a number of subintervals. The function then detects the changes in the signs of κ₁(τ|ρ, η) and κ₂(τ|ρ, η, α) on the boundaries of the subintervals, and uses bisection method to find the zeroes at which there is a change in the signs of κ₁(τ|ρ, η) and κ₂(τ|ρ, η, α).

4.1 Prior on τ versus prior on ρ

Setting a prior information on τ can be tricky unless there is a good prior information that can be used. Informative prior in the form of normal distribution has been proposed by Behrens et al. [16] but selecting the hyperparameters can be difficult. When no information on τ is available, researchers may opt to use noninformative priors. In this case, a uniform prior can be used for τ as proposed by Behrens et al. [16] and Cabras and Castellanos [18]. But as will be shown in the Section 5, using uniform prior on τ is not recommended. Villa [19] on the other hand has proposed to use data-dependent prior on τ by using Kullback–Leibler divergence to determine the order statistics’ ‘worth’.

In many cases, specifying prior information on ρ is easier as compared to specifying prior information on τ. This is particularly clear because ρ is bounded between 0 and 1 (whereas τ is not bounded above) and that ρ is free of unit. Furthermore, in many applications that uses generalized Pareto distribution to estimate extreme quantiles, researchers typically choose the threshold value based on the quantiles (e.g. 90th, 99th, 99.5th percentiles). Therefore using prior on ρ is more natural than specifying information on some arbitrary threshold value. Informative prior in the form of a beta distribution can be used for ρ when required.

If there are no information on both τ and ρ, we can use uniform prior for either τ or ρ. Note that the distribution of τ affects the distribution of ρ and vice versa. Fig 1 illustrates an example on how the distribution of τ affects the distribution of ρ and vice versa. The figure is based on having ρ = 1 − H(τ|η) as in the case of bulk models with h(x|η) be a lognormal distribution with log-mean 6.5 and log-variance 0.5². In this case, when τ is uniformly distributed, p(ρ) ∝ 1/h(H⁻¹(1 − ρ)|η), and when ρ is uniformly distributed, p(τ) = h(τ|η). It can be seen from the figure that when τ is uniformly distributed, the distribution of ρ is concentrated around 0. On the other hand, when the ρ is uniformly distributed between 0 and 1, the distribution of τ is unimodal and its density decreases as τ increases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The distribution of ρ when τ is uniformly distributed (left), and the distribution of τ when ρ is uniformly distributed (right), when ρ = 1 − H(τ|η) with h(x|η) is lognormal distribution with log-mean 6.5, log-variance 0.5².

https://doi.org/10.1371/journal.pone.0257762.g001

Note that if X is a random variable with pdf f_X(x), Y = v(X) is a function of X, and v(X) is monotonic, then where f_Y(y) is the pdf of Y. In the case of ρ and τ, the above equation translates to (20) where τ = v(ρ) for some function v(⋅), if it exists. Assuming the independence between η, (α, β), and τ, the prior can then be written as (21)

Using Eq (21), the MCMC algorithm can be performed by sampling τ with the prior on ρ specified using p(ρ), which gives equivalent result to the MCMC that samples ρ using the prior written in Eq (19). However, it is important to note that this is only valid when the relationship between ρ and τ is one-to-one, and there is still a need to check for the uniqueness of τ given the values of ρ and other parameters, for example by using Theorem 1 or 2. In the case of multiple τ solutions, the mapping ρ ↦ τ is not even a function since the relation is one-to-many. In this case, the function v(⋅) does not exist and Eq (20) cannot be used.

Not only that Eq (20) is invalid for the cases with multiple τ solution, specifying the prior according to the equation may lead to poor result on the posterior estimation. As an example, consider using a two-component lognormal mixture model for the lower data distribution, i.e. in a composite Pareto I model with r = (0.5, 0.5), μ = (5, 7), and σ = (0.6, 0.3). The relationship between ρ and τ is shown in Fig 2. The shaded area in the figure signifies the area where τh′(τ|η ≥ −h(τ|η) in which the acceptance probability for the Metropolis–Hastings step is zero. Suppose the true value of τ is greater than 1000. Then the MCMC that samples τ using prior specification on ρ and Eq (21) may get stuck in the region between 150 and 500 due to the area with zero acceptance probability, which in turn will cause the parameter point estimates to have huge error. Therefore, it is advised to avoid setting the prior as in Eq (21) for the cases where the relationship between τ and ρ is not proven to be one-to-one.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The relationship between ρ and τ when using a two-component lognormal mixture model for the lower data distribution.

https://doi.org/10.1371/journal.pone.0257762.g002

Following Cabras and Castellanos [18], it is assumed that x_(m+1) ≤ τ ≤ x_(n−2) where x_(i) is the ith order statistic, m is the dimension of η, and n is the sample size. This assumption is required to obtain proper posterior distribution over the parameters η, α, and β.

4.2 Prior distributions for α and β

For smooth composite Pareto I model, and continuous bulk model with Pareto I tail, the value of α is determined by Eqs (2) or (10). Therefore in these cases, prior distribution on α needs not to be specified since α is not a free parameter. However, when the composite Pareto I model is not assumed to be differentiable, or when the pdf of bulk model with Pareto I tail is not assumed to be continuous, the following prior distribution can be used: which is the Jeffreys prior for α for a Pareto I distribution with known threshold or scale parameter.

For composite Pareto II model with smooth pdf and bulk model with Pareto II tail and continuous pdf, the value of β is specified by Eqs (5) or (11). Since in this case α is a free parameter, a prior distribution on α is required. The Jeffreys prior for α for a Pareto II distribution when the parameter β and the threshold or location parameter are fixed or known as written below is used:

However, when the composite Pareto II model is not assumed to be differentiable, or the discontinuous bulk model with Pareto II tail is used, both α and β are free parameters and the prior distributions on α and β are required. Cabras and Castellanos [18] have proposed to use the Jeffreys prior for a Pareto II distribution with fixed or known location threshold as written below: (22) Using this prior leads to proper posterior distribution [23]. This prior distribution is used in our analysis.

4.3 Prior distributions for η

The prior distribution for η depends on the choice of lower data distribution h(x|η). Default priors can be use when no information is available. For example, if h(x|η) is a lognormal distribution with log-mean μ and log-variance σ², then η = (μ, σ²). As recommended by Cabras and Castellanos [18], the following default prior is used:

6.1 Application to income distribution

The composite Pareto I model is applied to the Malaysian Household Income Survey (HIS) to study the income distribution in Malaysia. Twice every five years, the HIS is conducted by Department of Statistics Malaysia (DOSM) to gather information regarding the economic well-being of Malaysian households. Surveys are performed by interviewing randomly selected households and several information including the household income and expenditure are recorded. Five HIS datasets are used: year 2007, 2009, 2012, 2014 and 2016. These data are provided by DOSM and Bank Data UKM, which consist of subsets of the whole survey data. (Unfortunately, access to the data is restricted and cannot be shared publicly. However, these data can be requested online from https://www.dosm.gov.my/v1/index.php.). There are more than 12 000 observations in each dataset, representing the households in Malaysia.

Since two households with the same income may not be in the same economic condition due to the difference in the size of household, the monthly gross income is equivalised by dividing it by the square root of the size of household. This square root equivalence scale is commonly used in studies including Congressional Budget Office [24] and OECD [25]. Although households are given weight depending on their location and strata, for simplicity, the weights are not used in our application. Fig 4 shows the histograms of the five datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Histograms of the equivalised household income in Ringgit Malaysia (RM) for the year 2007, 2009, 2012, 2014 and 2016.

https://doi.org/10.1371/journal.pone.0257762.g004

These data sets have been studied previously by various authors [26–30]. For example, Razak and Shahabuddin [26] have noted that the upper tail of the Malaysian household income fits the Pareto tail behaviour. Safari et al. [27] on the other hand have applied a semi-parametric approach to model the income distribution by combining Pareto type I distribution for the upper tail and a nonparametric approach for the rest of the data. In a recent work, the lognormal-Pareto II composite model was found to be able to describe the income distribution quite well [31].

We apply the smooth composite Pareto I model to these datasets and the two-component lognormal mixture distribution is used for the lower data distribution. In this case, where η = (r, μ₁, μ₂, σ₁, σ₂) and r ∈ (0, 1) is the weight of the first component and f_LN(x|μ, σ²) is the pdf of lognormal distribution with log-mean μ and log-variance σ². This choice of lower data distribution does not guarantee unique solution of τ given ρ and other parameters. However, when implemented to the datasets using the MCMC algorithm, no instances of multiple τ solutions is found.

For the prior distributions, reference prior is used for the parameters of the lognormal distribution such that and uniform prior is used for r. Using this real data, the Bayesian analysis is performed twice by using uniform prior on τ and uniform prior on ρ to compare the two results.

The estimated posterior means of τ and ρ are given in Table 5 along with the p-values of Kolmogorov–Smirnov (KS) goodness-of-fit test using the estimated parameter values. From the table, around 2% to 7% of the observations come from the Pareto tail. It can also be seen that the estimated τ obtained by using uniform prior on ρ are smaller compared to estimated τ obtained by using uniform prior on τ, which is similar to the results obtained in the simulation study. This may indicate that using uniform prior on τ overestimates the threshold value.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Estimates of τ and ρ for Malaysian household income distribution.

Included in the table is the p-value obtained from KS test using the estimated parameters.

https://doi.org/10.1371/journal.pone.0257762.t005

Additionally, the p-values from the KS test are higher when using uniform prior on ρ rather than uniform prior on τ, except for the year 2014 in which the p-value is higher when using uniform prior on τ. The p-value indicates that except for the year 2014, the estimated parameters obtained when using uniform prior on ρ fit the datasets better compared to the estimated parameters obtained when using uniform prior on τ. Therefore in general, using uniform prior on ρ is deemed better when compared to using uniform prior on τ in this application.

6.2 Application to NASDAQ-100

We implement the composite Pareto model to NASDAQ-100, a stock market index made up of 100 largest companies listed in the Nasdaq stock exchange. The daily increments of the index are taken from Yahoo Finance for the period of January 1985 to May 2002 with the number of days n = 4394. The daily increment values are transformed such that where P_t is the adjusted closing price for the index at day t. Fig 5 shows the histogram of the NASDAQ-100 daily increments. From the figure, it is clear that the distribution exhibits heavy tail behaviour, thus using composite Pareto model can be beneficial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Histogram of the NASDAQ-100 daily increments used in application.

https://doi.org/10.1371/journal.pone.0257762.g005

This data has been analyzed previously by Behrens et al. [16] and Villa [19] using bulk model with Pareto type II tail and gamma distribution. For the threshold parameter, Behrens et al. [16] have used uniform prior distribution whereas Villa [19] has used prior distribution based on the Kullback–Leibler divergence. In our implementation, we use both uniform prior on τ and uniform prior on ρ and compare our results with the two literatures.

We apply the same bulk model without the continuity condition to the data. In our application, the lower data distribution is h(x|η) = f_G(x|a, b) where f_G(x|a, b) is the pdf of a gamma distribution with shape parameter a and rate parameter b. Reference prior is used for a and b such that where ψ⁽¹⁾(⋅) is the trigamma function. The prior distribution for α and β is given in Eq (22). Additionally, the uniform prior on τ and uniform prior on ρ are used for comparing the results based on these these two choices of prior, along with the results obtained by Behrens et al. [16] and Villa [19]. Note that both of these literatures have used informative prior for the shape and rate parameters but the values for the hyperparameters are not provided. Therefore, we are unable to duplicate their prior for our application and we opt to use reference prior instead.

Behrens et al. [16] and Villa [19] have used a different parameterization to the model. Let ξ = 1/α and σ = β/α. Table 6 shows the estimated posterior mode for the Pareto II tail along with the estimated parameters from Behrens et al. [16] and Villa [19]. Note that the estimate for ρ is not given in both Behrens et al. [16] and Villa [19] thus left empty in Table 6. From the table, around 30% of the observations come from the Pareto II distribution and the other 70% come from gamma distribution. The parameter estimates using uniform prior on τ is similar to the parameter estimates using uniform prior on ρ, indicating that there are not much advantages of using uniform prior on ρ as opposed to uniform prior on τ as seen in the previous simulations and example.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Parameter estimates for NASDAQ-100 application with the 95% credible interval in bracket.

https://doi.org/10.1371/journal.pone.0257762.t006

Both parameter estimates found using uniform prior on τ and uniform prior on ρ are consistent to those found by Behrens et al. [16] and Villa [19]. Although Behrens et al. [16] also used uniform prior for the threshold parameter, the differences between our results and theirs might be due to the difference prior distribution used for the parameter of the gamma distribution. As mentioned previously, the values of hyperparameters used are not provided by Behrens et al. [16] and thus we cannot duplicate their results. Additionally, the credible intervals found in our analysis is narrower than the credible interval found by Behrens et al. [16], but broader when compared to the results in Villa [19]. It is noted that Villa [19] has used data dependent prior distribution for the threshold parameter.