Financial forecasts accuracy in Brazil’s social security system

Long-term social security statistical forecasts produced and disseminated by the Brazilian government aim to provide accurate results that would serve as background information for optimal policy decisions. These forecasts are being used as support for the government’s proposed pension reform that plans to radically change the Brazilian Constitution insofar as Social Security is concerned. However, the reliability of official results is uncertain since no systematic evaluation of these forecasts has ever been published by the Brazilian government or anyone else. This paper aims to present a study of the accuracy and methodology of the instruments used by the Brazilian government to carry out long-term actuarial forecasts. We base our research on an empirical and probabilistic analysis of the official models. Our empirical analysis shows that the long-term Social Security forecasts are systematically biased in the short term and have significant errors that render them meaningless in the long run. Moreover, the low level of transparency in the methods impaired the replication of results published by the Brazilian Government and the use of outdated data compromises forecast results. In the theoretical analysis, based on a mathematical modeling approach, we discuss the complexity and limitations of the macroeconomic forecast through the computation of confidence intervals. We demonstrate the problems related to error measurement inherent to any forecasting process. We then extend this exercise to the computation of confidence intervals for Social Security forecasts. This mathematical exercise raises questions about the degree of reliability of the Social Security forecasts.


Introduction
The concern with a crisis in public pension systems is not new [1] and policymakers around the world are increasingly concerned with finding alternatives to fund Social Security. In Brazil as well, not differently, the recent debate on the future of the National Social Security system is being driven by the fear that future retirements pose a real financial risk for the solvency of the System as discussed in [2]. a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 The ongoing demographic change in Brazil, marked by an increase in life expectancy at birth, reduced mortality rate, continuous and persistent reduction in the fertility rate and increased life expectancy of people at older ages, will lead to radical changes in the actuarial mechanism of Social Security. These changes are due to both increases in expenses (an increase in the number of inactive elderly people and a longer duration of the benefits received) and to the reduction in the number of taxpayers due to the shrinking economically active population over time. These factors imply additional pressure to the current Social Security system, suggesting the need to evaluate its adequacy to the new demographic reality.
The general Social Security model in Brazil is the so-called pay-as-you-go (PAYGO) system, a mandatory transfer of income from the employed labor force to the elderly portion of the population. In this case, a public agency (Social Security), acts as a buffer between the two parts of the population [3]. In the PAYGO pension plan, the benefits are directly tied to the contributions or taxes paid by individual participants. The right to receive a pension is essentially a political right, the terms of which are guaranteed by the government, although the government might subsequently alter the terms on which pensions are conceded. In this model, the transfer of goods and services from the workforce to the pensioners is very transparent [4].
The most important set of long-term statistical forecasts produced and disseminated by the Brazilian government are population growth forecasts and financial estimates of Brazil's General Social Security System (RGPS). If we consider that these two sets of forecasts are strongly correlated, assumptions in both will have great impacts on expected Social Security results.
Social Security financial forecasts formally aim at providing accurate results that would serve as background information for (socially determined) optimal policy decisions regarding the Social Security System. In fact, the public debate about the probable future of Brazilian Therefore, from the 2002 LDO on, we can find an annex of fiscal targets with forecasts of RGPS result (commonly in Annex IV, but in some LDOs it may be in Annex III). These forecasts are conducted by the Secretary of Social Security Policies (SPS) of the Ministry of Social Security (MPS)-recently absorbed by the Ministry of Finance-and have been used by successive governments for the debate at the National Congress on the financial sustainability of the RGPS.
However, a closer look at Annex IV of the LDO clearly shows, despite the relevance of the topic, a remarkably low level of transparency in the methods used for forecasting Social Security results. Firstly, the model described in Annex IV is not replicable because it is simply incomplete. There is not, on the other hand, any other official document describing it in its original form or a description of the changes that it allegedly suffered during the last eleven years of the existence of Annex IV. Also, year after year, there is no process of (self) evaluation of the efficiency of forecasts. A simple comparison of forecasts against the budget execution would be enough to show the appalling forecast errors. Each LDO presents new forecasts, without making any mention to corresponding forecast results published in the past budget guidelines. Finally, and even more significantly, no executive order at any level of federal government establishes formal parameters, databases and specific methods for forecasts. Official Social Security forecasts simply do not have any regulation.
As we will show in the following sections, the long-term Social Security forecasts are systematically biased in the short term and have significant errors that render them meaningless in the long run. Moreover, forecast models used in LDO have an obvious stochastic character and, despite the various references in the text to the limitations of the model, their results are always presented without even specifying the margin of forecast errors, as if they were simply deterministic results.
Furthermore, as there is no official evaluation of the forecast model's effectiveness, the results published in Annex IV of LDO are informally reviewed, but the nature of such revision and the impact of the decisions guided by such models are unknown to the broader public. We obviously understand that actuarial forecasts can not be totally accurate. However, the longterm forecasts carried out by the Brazilian government serve as input to a set of relevant strategic decisions, and its limits have to be more clearly explained, and the use of more recent-and less pretentious-forecast techniques should be used to a more robust and reliable decisionmaking support system. This paper aims to present a study of the accuracy and methodology of the instruments used by the Brazilian government to carry out the long-term actuarial forecasts of RGPS. We, therefore, base our research on an empirical and probabilistic analysis of the official models. Such a study is of the utmost importance since, as we have mentioned, such forecasts are being used as support for the government's proposed pension reform, which plans to radically change the Brazilian Constitution insofar as Social Security is concerned.
We have divided this study into two parts. The first part performs an empirical analysis based on official data provided by the Brazilian government. We look at the forecast results presented in the LDO, show its forecast errors and demonstrate their low reliability. We also try to replicate the LDO results following the model described in the official document. Finally, we conduct an analysis of the data sets used in the forecasts, showing that the Brazilian government has systematically used outdated data, which has an adverse impact on results.
In the second part, we conduct a theoretical analysis based on a mathematical modeling approach. From a standard exercise with Brazilian Gross Domestic Product (GDP) series, we discuss the complexity and limitations of macroeconomic forecasts through the computation of confidence intervals. We demonstrate the problems related to error measurement inherent in any forecasting process. We then extend this exercise to the computation of confidence intervals for Social Security forecasts as it is well known that pension revenues are closely related to GDP behavior. This mathematical exercise raises questions about the degree of reliability of the LDO forecasts.
The remainder of the paper is organized as follows. In the first section, we present a brief description of the Brazilian General Social Security System. Then, we describe some related works on Social Security financial forecasts. Next, we perform a comparison of Social Security forecasts in each Annex IV of the LDOs from 2002 to 2015 with its corresponding published official statistics. In the following section, we describe the forecast model used by the Brazilian government and present an attempt to replicate the financial results of the 2012 LDO. We then analyze population and labor market data sets utilized in each LDO forecast, showing the impact of outdated statistics on Social Security results. Next, we present the aforementioned theoretical analysis of GPD and Social Security forecasts. In the last Section, we present conclusions and suggestions to improve the Brazilian Social Security forecast system.

Social security financing
As described in the Brazilian Constitution and regulated by Law nº 8212/91, Social Security comprises the right to health, pension, and social assistance. In concrete terms, it involves the payment of pensions with the purpose of guaranteeing its beneficiaries with essential means of maintenance. The right to a pension may be due to incapacity, advanced age, time of service, involuntary unemployment, imprisonment of a family member or death benefits for those who are economically dependent. There are a different set of rules for urban and rural workers as well as men and women.
The largest Social Security system in Brazil is the aforementioned General Social Security System (RGPS), the federal system for employees under the rules established by the so-called Consolidation of Labor Laws (CLT in Portuguese), mostly workers within the private sector but also workers in some state owned enterprises and agencies at the local or national level. Local and state governments also have their own Social Security systems for their public servants as well as federal public servants and military personnel. RGPS is by far the largest pension system in Brazil and monthly pays more than thirty million beneficiaries around the entire country.
RGPS is a pay-as-you-go system of pension and it is funded by a combination of government transfers and payroll contributions by employers and employees. Hence, it is a tripartite system based on an intergenerational simple distribution. Accordingly, article 10 of Law No. 8212/91 establishes that Social Security will be financed by the entire society, directly and indirectly in the form of social security contributions. Table 1 describes the primary sources of Social Security revenues and expenses as described in the Social Security Statistical Yearbook (AEPS) [5]. Also, Fig 1 shows [5].
In the thirteen years until 2015, Brazil underwent a robust process of social inclusion with the lowest unemployment rates in history and the creation of innovative social policies that have allowed millions of Brazilians to access Social Security. On the other hand, this process has made the rate of growth of Social Security expenditures surpass the rate of growth of revenues that in combination with an increasing elder population has raised some concerns about the necessity of a reform in the national public pension's system. As the devil lives in the details, the calibration of any proposal of social security is crucial for social welfare and this is why we believe that efficient-or at least transparent-financial forecasting is crucial for the proper calibration of any proposal of reform.

Related works
Concerns with the soundness and transparency of social security forecasts in the USA are present in the works of [6]. The authors offer an evaluation of the Social Security Administration demographic and financial forecasts used to assess the long-term solvency of the Social Security Trust Funds in the USA. A comparison of Social Security forecasts with the observed data shows that forecasting errors-as evaluated by how accurate the forecasts turned out to bewere approximately unbiased until 2000 and then became systematically biased and increasingly so over time. Also, most of the forecasting errors since 2000 are in the same direction, consistently misleading users of the forecasts to conclude that the Social Security Trust Funds are in better financial shape than turns out to be the case. Lastly, they show that the Social Security Administration's informal uncertainty intervals appear to have become increasingly inaccurate since 2000.
In [7], the same authors show that the US Social Security Administration uses old statistical forecasting methods and makes public insufficient replication information. Their findings come from a large number of interviews with participants at every level of the forecasting and policy processes. A detailed evaluation of the methodology used to calculate mortality rates is described, showing several problems and the impact of the use of these rates on revenues and expenditures. They also suggest some manipulation due to political pressure. Finally, they make several proposals to solve the problems reported.
In [8] is presented a study that seeks to quantify, by using a long-term forecast model, the impact of demographics on the expenses of Social Security as a proportion of GDP for Brazil. Despite the great relevance of the results and the adequacy of the forecast model for the Brazilian case, the model used is quite simplistic. The model is useful only to illustrate the potential effects of aging on Social Security spending. It is inappropriate to forecast policy proposals concerning Social Security. Besides, there is no evaluation of forecast errors and confidence intervals.
In 2007, the Brazilian government promoted the National Forum on Social Security. One of the main products of this event was the document [9]. This document seeks to defend the rationality of parameters and structural elements of the long-term forecast model used by SPS/ MPS while seeking to incorporate some of the suggestions made during the Forum. The document also mentions the use of two forecast models. One for the short term, i.e. for the current year and the subsequent three years, and another for the long term, starting from the fifth year following the current year. The short-term model is used to guide the preparation of the annual budget proposals, the Pluriannual Plans (PPAs) and to track revenues and expenditures in the current year. The long term model is used to simulate the impact of different proposals of structural reforms on Social Security financial balances. The document only describes some aspects of the long-term model without presenting it in detail. The authors refer to Annex III.5 of the 2008 LDO, where the model is explained in greater detail.
It is worth mentioning that for our study, most of the documents, papers, and databases made available by the Brazilian government are, as expected, in the Portuguese language.

Analysis of the social security forecasts
As an initial part of this study, we have made a comparison between Social Security expenditure forecasts as presented in LDOs and the observed data on executed expenditures, available at InfoLogo, the official Social Security database [10]. Each LDO has forecasts of at least 20 years and the period analyzed was from 2002 to 2015. LDOs are produced one year before the corresponding year, i.e., the 2016 LDO was presented in Congress in April 2015. Also, as LDO is written one year before the corresponding year, the data used has a 2 years lag. Thus, the 2016 LDO is produced in 2015 using 2014 data. The first aspect of revenue forecasts is the substantial error, in this case, of underestimation. Note that the error is higher as LDO moves away from the forecast date. An exception was the forecast revenue from the year 2013, explained by the sudden and sharp decline in economic activity observed in Brazil from 2014. In Fig 3, we observe that expenditure forecasts also show a pattern of systematic error, similar to what was observed for revenues, where the error is higher as the forecast point distances from its original year.
The long-term solvency of Social Security is the difference between revenues and expenditures. Thus, we may have a situation of surplus or deficit of pension accounts. A common feature of the three sets of forecasts presented, as one might expect, is the relation between forecast error size and the distance from the base year. In fact, an important  Social security forecasts accuracy in Brazil source of distortion in forecasts comes from model contamination from short-term economic phenomena, which leads to a replication of prevailing economic conditions in the base year into the future. A forecast for a year of higher economic growth tends to replicate this effect for the future series and conversely in years of a stagnant economy. Hence, when we look at the forecasts for the year 2060 as presented in the latest LDOs, we realize they have little statistical validity since we perceive an explosion of the forecast error as we move away from the base year-and yet, no uncertainty intervals are reported by the government. The purpose of uncertainty estimates is to protect oneself from expressing overconfident conclusions from the results, and if estimates are consistently falling outside those uncertainty intervals, then revision and improvement in the forecasting process should follow [7]. One intuitive way to assess forecast errors would be breaking it down into demographic, economic (labor market) and Social Security variables. The difficulty here is the lack of records of the statistical series that were the basis for the LDO forecasts. We have the forecast value of revenue and expense, but we do not have their respective subcomponents.

Results replication
At present, SPS at the Ministry of Finance is responsible for the forecasts depicted in Annex IV of LDO. However, as we have mentioned, SPS does not share sufficient information on forecast procedures and, as a result, no fully independent quantitative evaluation of results presented in LDO has been made possible. The possibility of replication and data sharing is a widely understood and accepted best practice throughout the scientific community [7].
In order to evaluate the results and find the source of errors presented in the previous section, we have extensively tried to reproduce the LDO results. Initially, we examined the Annexes of all LDOs from 2002 to 2017 and identified that the presented equations are precisely the same in all documents. However, from the 2009 LDO on, an updated version of the forecast model is mentioned, which, for example, purportedly adopted a new concept of participation rate and discarded the direct usage of the unemployment rate. Notwithstanding this In an attempt to replicate the forecasts, we have chosen the 2012 LDO as a reference and a set of Social Security data obtained from DATAPREV, the Social Security Information and Technology Company.
Social security forecast model. The mathematical models used in forecasts of Social Security expenditures and revenues are described in Annex III, Fiscal Targets of 2012 LDO [11].
All variables in the equations have four or five indexes. The indexing parameters follow the definitions and domain sets described in Table 2.
To calculate the expenses, firstly, one must calculate the number of current benefits, here referred to as stock. The stock is calculated by the method of flows, where the granting and cessation of benefits are estimated, and then the stock of benefits in a year is determined. The benefits grant flows are calculated as: In b ða; y; g; o; kÞ ¼ Pða; y; g; oÞ Ã P in ða; y; g; o; kÞ; where In b is the inflow in the benefits of type k with age a in year y of gender g, operation zone o, P is the population and P in is the probability of entry into benefit. The values of P in are constant for all years and depend on the number of benefits granted and the population as presented in Eq (2) where S b represents the stock of benefits of type k, S b (a − 1, y − 1, g, o, k) is the stock of the previous year, P surv is the probability of an individual surviving from age a − 1 in year y − 1 to age a in year y and In b are the entries (new beneficiaries). The probability of surviving P surv is Then, the total stock of benefits in a year y is given by the sum of all benefits, for all ages, genders, and operation zones as described in Eq (5). Spending on benefits is calculated from the stock and the average value of the benefits, as shown in Eq (6) where C is the taxpayer stock, P is the population, Part is the labor force participation rate, U is the unemployment rate, and d is the contribution density. Contribution density represents the proportion of months that an employee contributes annually, i.e., a value of 1 means that an employee contributed every month in a year. Once the number of taxpayers is estimated, the value of revenues (R) in a year y is calculated as Eq (8).
R y ¼ P a P g P o Cða; y; g; oÞ Ã ½r 1 Ã MinðC e ; S a ða; y; g; oÞÞ þ r 2 Ã S a ða; y; g; oÞÞ ð8Þ where r 1 is the Social Security contribution rate paid by the employee, r 2 is the Social Security contribution rate paid by the company, C e is the contribution ceiling (the maximum value used to calculate the contribution) and S a is the employee salary. The Min in the equation ensures that the value used to calculate the employee contribution will not be greater than the ceiling (C e ).
Here we have to explain an important point about pension income. As previously described, revenues from Social Security have several sources, but in the model presented, only direct contributions from employees and companies are considered (Eq (8)). Examining revenues from 2011 to 2015, we have observed that the sum of the contribution of employees and companies correspond, on average, to 55.7% of the total Social Security revenue, as described in Table 3. Thus, the model presented includes approximately half of Social Security revenues.
Replication of social security forecasts. As mentioned above, we have chosen the 2012 LDO for forecast replication. The data sets and parameters used in the replication attempt are described in Table 4.
Several variables described in the forecast model do not have their values or equations presented in the LDO, as, for example, the contribution density and probabilities of entry and survival. The probabilities of entry were calculated based on the flow of granted benefits in the recent years, and the probabilities of survival were calculated based on population data provided by Brazilian Institute of Geography and Statistics (IBGE) as previously described in Eqs (2) and (4), respectively.
As previously described, LDO mentions a short-term model that is used to forecast the first five years (2011 to 2015), and then a long-term model that is used for the remaining years. In [9], some differences between the models are described. The short term model has fewer variables and uses the forecast of some macroeconomic parameters provided by yet another branch of government-the Secretary of Economic Policy at the Ministry of Finance. The series for executed Social Security revenues and expenses are provided by the cash flow database of the National Institute of Social Security (INSS). Demographic parameters are not used in this model since, in the short term, demographic variations allegedly have a minimal impact on Social Security. In LDO, there is no description of the short-term model and no other document describing it has been provided by the government. Therefore, all the results presented here are based only on the long-term model. The long-term model is supposedly more complex and considers a much larger set of variables, including population forecasts. By using the demographic forecast, the long-term model incorporates the expected change in the age structure of the population.
Once the model has been described, we have implemented and carried out forecasts of revenues and expenses of Social Security. Fig 5 show the variations (in %) between our results and LDO forecasts. Social security forecasts accuracy in Brazil By following the equations presented in 2012 LDO we have tried to replicate the published series for forecasted expenses and revenues. In what concerns expenses, the variation in the first year is 11% and increases over the years, reaching 39% in 2031. There are several possible reasons for theses differences between our results and the official results. Firstly, as we have mentioned, there is no information about the short-term model. LDO mentions that the short-term results are provided by the Secretary of Economic Policy at the Ministry of Finance, following a so-called parameter grid released on April 8th, 2011. However, a closer examination of the document reveals no particularly useful information that could be used for modeling improvement. Secondly, there is a clear absence of complete data sets used in the forecasts. All necessary data should be described or, at least, referenced, but unfortunately, it is not available for independent exercises.
From the Fig 5, we observed an average variation of 53.7% in revenues, with small shifts. As the LDO does not describe any other source of revenue, and from Table 3, we have seen that the contribution of employees and companies corresponds to approximately half of the income from Social Security contributions, and the variation is almost constant throughout the forecast, we believe that some adjustment is applied to the forecast of each year to account for other incomes.
The failure in replication attempt reinforces the lack of transparency in the methods used by the Brazilian government to forecast revenue and expenses of Social Security. In future financial reports, the government should include a detailed description of methods, data used, a discussion of what forecasting mistakes were made in previous reports, what was learned from the mistakes, and what actions might be needed to improve forecasts. The above results were calculated using the MATLAB software, and all developed code and databases are available in [12].

Analysis of data sets used
To carry out Social Security forecasts, two sets of data are essential: population forecasts and labor market statistics [13,14]. Population forecasts are conducted by IBGE, the institute responsible for official population estimates. Currently, IBGE has four population projections, To ensure reliability in Social Security forecasts, it is crucial that an annual update is performed in the data sets used. Therefore, we expected that the most recent data available from one branch of national government would be applied in the forecasts. However, as described in Table 5, the population (IBGE) and the labor market (PNAD) data used in the forecasts take years to be updated. For example, data from PNAD of 2005 was used for LDOs from 2009 to 2013, but at no time in Annex IV of LDO was the use of lagged data justified.
As an example, Fig 6 shows the relationship between the total number of Social Security taxpayers and the working age population (16 to 64 years). We observe a clear right-upward shift of the curve, demonstrating a significant change in the labor market in a period in which government forecasts consider them constant. Variables such as participation rate and average salary have undergone significant changes but these movements are simply ignored in the forecast models.
A simple way to analyze the impact of labor market data can be made as follows. The number of employees can be calculated as: Empðu; rÞ ¼ f ðpopðu; rÞ; Part; UÞ ð10Þ  From Eq (9), the urban (u) and rural (r) population is calculated from the total population (Pop total ) and urbanization rate (Urb rate ). Then, from Eq (10), the number of employees (urban and rural, men and women) is a function of population, the participation rate in the labor force and the unemployment rate. So, the total amount of revenue is a function of employees, average wages and contribution rate as described in Eq (11).
Revðu; rÞ ¼ f ðEmpðu; rÞ; Wages avg ; Contrib rate Þ ð11Þ As described in Table 5, the last four versions (from 2014) use fixed values of urbanization rates, labor force participation rate and unemployment, calculated from 2009 PNAD. In practice, the set of equations collapses, which makes the Eq (11) a simple function of the total population changes, independent of the labor market dynamics in the referred years. This makes, for example, the correlation between total revenues and the total population very high (0.8) from 2010.
The use of outdated labor market data has great influence on results [14] and, in our opinion, compromises reliability. In another exercise, we revisited the forecasts presented in the previous section by changing the 2005 labor force participation data (used in the 2012 LDO) by 2009 (taken from the 2014 LDO). As shown in Fig 7, there is an average increase of 9% in revenue, 2% in expenditure and 6% in the number of employees.

Dispersion calculation for GDP and social security forecasts
As we have seen, official forecasts are explicitly probabilistic but no reference to error margins are ever made in government documents. Even worse, forecast results are presented with a presumption of certainty with serious social and political consequences as they have embedded confidence intervals that are unknown to the audience. What would be the confidence intervals for the predicted series published in the budget guidelines? A simple exercise can be made starting from the GDP series. We have built a GDP forecast model from the information presented in Table 6, that corresponds to the same period of time for which pension data is available.
The method assumes the behavior of GDP by 2060 will follow that of the calibration period, from 2000 to 2015. In this period, the growth rate of the GDP is modeled by a Normal distribution with mean and variance calculated from the data sample. The forecast of the GDP starts from the last observed point and assumes that its growth follows the laws of a Brownian motion [15] up to the forecast horizon, taking as growth rates of the GDP those obtained from the calibration period. So, if Y 0 is the last observed value of the GDP, a is the mean of the growth rate of the GDP in the calibration period and b 2 its variance, for a level of confidence α, and a time t after the last observation of the GDP, the confidence interval is: where ϕ −1 (.) is the inverse of the cumulative distribution function of the standard normal distribution. An observation on the use of the data, whose sample is rather small: they reflect a recent period of the Brazilian economy for which we have assumed the growth rate of the variable to be stationary. The alternative option, i.e., to use the quarterly GDP values provided by IBGE, would only add seasonal effects without substantially changing our results. The use of timeseries statistical techniques, such as autoregressive moving average (ARMA) or autoregressive integrated moving average (ARIMA) [16] for example, could improve short-term forecasts since past oscillations tend to repeat themselves and short-term decisions do not "divert" the process trajectory. In the long run, however, the results are similar.  The high volatility of the GDP makes its forecast challenging in the short term and practically impossible, with acceptable certainty, in the long run. As seen in both results, 95% and 99% confidence levels generate intervals with a very high margin of error. For comparison purposes, the interval of 50% was drawn to show that only in this case the error margins are at acceptable levels, but with the confidence of a coin toss.

RGPS Forecasts with dispersion indicators
In the same manner, we can extend our exercise to Social Security actuarial results. Table 7 shows the value of Brazilian GDP and the realized RGPS revenue, expenditure and deficit for the period from 2002 to 2015. At the end of Table 7, we present the mean, standard deviation, and correlation with GDP data. The high correlations of revenue and expenditure with GDP motivated the use of the same method used previously for GDP forecasts. However, it cannot be applied to the RGPS deficit for theoretical reasons that reflect the weak correlation between these variables. In this work, as a first approximation, we have chosen-as explained beforeto model the relevant variables (revenue, expenditure, and deficit) directly as a Brownian motion [15].
The proposed forecast exercise consists of extrapolating the series of RGPS revenues, expenditures and deficits-as observed in the years 2000 to 2015-to the target period, assuming that they will strongly influence the future behavior of these variables in the next decades.  show an extrapolation exercise for the year 2025 and 2060. The blue color gradient represents the confidence interval of 50%, 95%, and 99%. The red curve shows the government forecasts for the same variables in the same periods (values taken from the 2017 LDO).
All data and developed software used to implement this mathematical modeling are available on [18].
From our exercise, we have observed that the government overestimates revenues and especially expenses, which causes the deficit to grow exponentially. As such results represent a structural break in the historical behavior of the studied variables, they naturally raise relevant questions: what aspects provided by the government are influencing so drastically the pension Social security forecasts accuracy in Brazil result beyond the aging population problem? What macroeconomic assumptions have been made, directly affecting revenues and expenditures? What is the world scenario and its possible implications for the national economy?
These questions find no answer in the official documents. We insist that the unpredictable behavior of economic variables exposes the fragility of forecasts and that any forecast potentially suffers from this type of shortcoming. Therefore, it is crucial that all assumptions, scenarios and equations involved in the forecasting exercise be made available [13]. More than Social security forecasts accuracy in Brazil fabricating certainty, acknowledging uncertainty may be more useful for decision makers in a context of a Social Security reform.

Conclusions
This paper attempts to make an analysis of the Social Security forecasts presented annually in the budget guidelines of the Brazilian government. Although these forecasts have been reported since 2002, no systematic and comprehensive evaluation of their accuracy has ever been published. We show that the forecasts depicted in the LDO lack reliability in the long term. A self-assessment of the methods used should be performed (and be published) giving a clearer understanding of the limitations of the model. The failure in our replication attempt of the LDO results demonstrates the lack of transparency in official documents, both in the described equations and in the databases purportedly used. The government should follow best practices in academia and make their forecasting procedures public and replicable, making the model and data used fully available or the presented errors are unlikely to be corrected. Moreover, the unjustified delay in updating data sets used for projections greatly compromises forecast results, impairing the establishment of a robust basis for fine-tuned decision-making in Social Security policy.
We have shown that long-term forecasts of variables such as GDP and Social Security solvency carry a large component of inconstancy and uncertainty that needs to be measured through the definition of confidence intervals. As macroeconomic literature points out, both GDP and Social Security financing are affected by the international environment, the evolution of the productive structure, the macro policy options of each elected government, and by political-institutional issues. The burden of unpredictable factors make any conclusions about forecasts quite vulnerable.
The collective efforts of the scientific community could easily be directed to help the Brazilian government in the challenging task of forecasting Social Security financial series, taking advantage of many of the dramatic improvements in statistical modeling over the last years. More transparent forecasting procedures would also enable members of Congress and partisans on both sides of the alley to consider alternative assumptions explicitly when they debate proposals to ensure the solvency of Social Security.
In our future work, we are developing studies of a Brazilian Social Security forecast model with explicit calibration as done in [19], which we consider representing an improvement in relation to current forecasting practices. We are also developing a free and open source software that allows public simulation and analysis of the financial and social impact of possible pension reforms in Brazil.