Evaluating the impact of Hazelwood mine fire event on students’ educational development with Bayesian interrupted time-series hierarchical meta-regression

Background Environmental disasters such as wildfires, floods and droughts can introduce significant interruptions and trauma to impacted communities. Children and young people can be disproportionately affected with additional educational disruptions. However, evaluating the impact of disasters is challenging due to difficulties in establishing studies and recruitment post-disasters. Objectives We aimed to (1) develop a Bayesian model using aggregated school-level data to evaluate the impact of environmental disasters on academic achievement and (2) evaluate the impact of the 2014 Hazelwood mine fire (a six-week fire event in Australia). Methods Bayesian hierarchical meta-regression was developed to evaluate the impact of the mine fire using easily accessible aggregated school-level data from the standardised National Assessment Program-Literacy and Numeracy (NAPLAN) test. NAPLAN results and school characteristics (2008–2018) from 69 primary/secondary schools with different levels of mine fire-related smoke exposure were used to estimate the impact of the event. Using an interrupted time series design, the model estimated immediate effects and post-interruption trend differences with full Bayesian statistical inference. Results Major academic interruptions across NAPLAN domains were evident in high exposure schools in the year post-mine fire (greatest interruption in Writing: 11.09 [95%CI: 3.16–18.93], lowest interruption in Reading: 8.34 [95%CI: 1.07–15.51]). The interruption was comparable to a four to a five-month delay in educational attainment and had not fully recovered after several years. Conclusion Considerable academic delays were found as a result of a mine fire, highlighting the need to provide educational and community-based supports in response to future events. Importantly, this work provides a statistical method using readily available aggregated data to assess the educational impacts in response to other environmental disasters.


Import data
The data as well as the analysis code used in this tutorial can be directly downloaded from Github repository: https://github.com/CarolineXGao/NAPLAN_impact. # use file path of saved NAPLAN data naplan <-read.csv(here::here("Data","naplan _ fake _ data.csv")) The variables in the data set are:

Stan model block
When using Rmarkdown file, stan code can be directly included as a block of code with specification of {stan output.var = "StanModel"} in the code block. In this model we use weakly informative priors, N(10, 5), for the SDs of the random school effects, random cohort effects as well as random error N(10, 5). 10 was chosen because when using two-level mixed-effects models with the mean score differences as the outcome variable, the estimated error terms are close to 10. Evaluating the impact of Hazelwood mine fire event on students' educational development with Bayesian interrupted time-series hierarchical meta-regression y~normal(mu, sigma); } }

Run Stan model
Input data is needed to be saved in a list. ## Inference for Stan model: 832db374c6d4af44fc1e1f951141a4d4.
There was no warning of divergent transitions (using non-centred parameterisation centered parameterisation can help with avoiding divergent transitions), which can be diagnosed using diagnostic plots for the NUTS. All continuous variables are confounding variables (we are not interested in estimating effect sizes from these parameters), hence they were all standardised in the analysis to improve model fitting speed. If any variable of interest is a continuous variable, the original parameters can be easily recovered (see Stan manual) post standardisation.
The mcmc_pairs function is used to visualize the univariate histograms as well as bivariate scatter plots for key parameters. It is useful in identifying multicollinearity (strong correlation) and other non-identifiability issues (banana-like shapes).
Evaluating the impact of Hazelwood mine fire event on students' educational development with Bayesian interrupted time-series hierarchical meta-regression mcmc _ pairs(posterior, pars = c("alpha", thetas, paste0("beta[",1:length(predictors),"]"),"lp __ "), off _ diag _ args = list(size = 1.5)) There is a negative association between sampled coefficients of the intercept term (alpha) and school sector (government, beta[7]). This is possible as school sector (government vs non-government ) is the most important predictor of school-level NAPLAN results. Hence the sampled intercept will be impacted by the sampled coefficient of the school sector.
Evaluating the impact of Hazelwood mine fire event on students' educational development with Bayesian interrupted time-series hierarchical meta-regression

Plot marginal effects
Here we obtain predicted margins using the posterior distribution of coefficients using the following steps: •  Evaluating the impact of Hazelwood mine fire event on students' educational development with Bayesian interrupted time-series hierarchical meta-regression