Study design

All study procedures were approved by the Monash University Human Research Ethics Committee (project number: 5834) and the Victorian Department of Education and Training.

Selection of schools and classification of exposure risks

The school-level exposures were assigned based on the Australian Statistical Geography Standard (ASGS) boundaries [53]. ASGS divides geographical areas, also known as statistical areas (SA), into four levels: SA1, the smallest census unit with a population between 200–800; SA2, comparable to a township; SA3, comparable to a local government area; and SA4, largest sub-State regions.

Due to a lack of air quality monitoring data in the initial mine fire period when pollution levels were highest [54, 55], retrospective modelling of air pollution levels was conducted by CSIRO [42]. Hourly emissions of mine fire-related particulate matter exposure (PM_2.5) were estimated using a prognostic meteorological and pollutant dispersion model at a resolution of 1km x 1km, see details published elsewhere [42]. The estimated spatial-temporal PM_2.5 concentrations were aggregated by SA2 to understand the geographical distribution of cumulative mine fire-related air pollution exposure.

The modelled mine fire-related PM_2.5 concentrations suggested that the Morwell SA2 closest to the mine fire experienced the highest smoke concentrations, which at times greatly exceeded national safety standards [42]. Prevailing winds largely restricted the dispersion of the smoke plume to the Latrobe Valley SA3 region (which includes Morwell), and that area is considered to have suffered a moderate level of exposure [42]. Further to the east, the SA3 area of Wellington Shire, with a socioeconomic profile similar to that of Latrobe Valley, had little to no exposure to the smoke. Therefore, Wellington Shire was chosen as the comparison area for this analysis. There were 69 primary and secondary schools in the region which were classified into three mine fire exposure groups: Morwell (high exposure), the remainder of the Latrobe Valley (moderate exposure) and Wellington (no/low exposure), see Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Map of selected mine fire exposed SA2s in Victoria.

Superimposed dots are the location of schools in the impacted areas.

https://doi.org/10.1371/journal.pone.0281655.g001

NAPLAN and school profile data

In Australia, NAPLAN tests are conducted annually in May for Grade 3, 5, 7, and 9 students, and NAPLAN examines educational domains of reading, writing, numeracy, spelling, and grammar/punctuation [56]. As NAPLAN assesses incremental learning, students’ scores are expected to increase when they are retested at each two-year interval, from an average score of around 420 in Grade 3, to 580 in Grade 9 across individual domains [56].

Aggregated school-level NAPLAN data (mean score and SE for each domain) were requested from the Australian, Curriculum, Assessment and Reporting Authority (ACARA) for all Victorian schools between 2008 and 2018. School profiles, including total enrolments, school sector (government, non-government), school type (primary, secondary), gender proportion, and Index of Community Socio-Educational Advantage (ICSEA), were provided by ACARA for each year. ICSEA is a scale indicating socio-educational advantage of the school based on students’ family background information, which ranges from about 500 (extremely disadvantaged student backgrounds) to about 1300 (extremely advantaged student backgrounds).

Statistical methods

School characteristics (in 2014) and the NAPLAN participation rate (2008–2018) were first compared in the three exposure groups. School-level mean NAPLAN scores pre- (2008–2013) and post-mine fire (2014–2018) were visualised using box plots by exposure group for each educational domain and student grade level.

Centring NAPLAN scores

NAPLAN tests were designed to represent the growth of students’ achievement over time from Grade 3 to Grade 9, with scores expected to increase with increasing grade level. As the test score can be interpreted in terms of academic progression, it was not standardized. However, to make results comparable across students’ grades and domains, all scores were first centred against the matching mean scores in regional Victoria (for the same year, grade and educational domain).

Bayesian interrupted time series hierarchical meta-regression

Hierarchical two-level meta-regression models were carried out in a Bayesian modelling framework to estimate the association between mine fire exposure and centred NAPLAN scores. This approach directly models distributions of students’ performance within schools using mean scores and standard errors, which follows the meta-analysis approach to ensure unbiased estimates that are comparable with the individual-level model without confounder adjustment (assuming there is no ecological bias). Interrupted time series models using only means with linear regression instead of meta-regression will be inefficient (significant loss of statistical power) and biased towards smaller schools (large schools with more students should be given higher weights, e.g., using regression models weighted by the number of students or meta-regressions with inverse variance as weights). Although hierarchical meta-regressions can be carried out in a non-Bayesian framework, the complex model structure can introduce convergence difficulties and problematic confidence intervals [57]. In the Bayesian framework, prior information can also be incorporated.

The modelling framework is illustrated in Fig 2. The first level random effects (random intercepts) were modelled as student cohorts, e.g., the cohort of students progressing from Grade 3 in 2014 to Grade 5 in 2016 at the same school. The nested second level random effects (random intercepts) were modelled at the school-level. The centred mean NAPLAN score at Grade g level for cohort c students in given school s was modelled as follows: (1)

The terms θ_s and θ_s,c are the random effects for the school s and cohort c in that school, respectively. θ_e is the random error term. We assume , and . τ_s,g,c is the standard error of the mean NAPLAN score of the given school, student cohort and grade level (input data). The matrix X_s,g for fixed effects includes potential confounding factors: school characteristics (ICSEA, total enrolments, percentage of girls, school sector), grade level, and the interrupted time series terms to model the mine fire exposure effect variables detailed below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of Bayesian interrupted time series hierarchical meta-regression.

*Means and SE of test results each year for individual students’ grade level at each school. The subgraph (A) represents a theoretic model of the mine fire interruption effect.

https://doi.org/10.1371/journal.pone.0281655.g002

The effect of exposure to smoke during the mine fire was evaluated using an interrupted time series design (Bernal et al., 2017) with a time-specific interruption variable to capture the immediate effect of the mine fire on the NAPLAN results of that year, and a change in trend from pre- to post-mine fire (an interaction between the time interruption variable and year), see Fig 2(A). It was hypothesised that there would be no interruption effect in the no/low exposure group. Mine fire interruption effects and post interruption trend differences were evaluated separately for the high and moderate exposure groups. The interrupted time series terms in X_s,gβ in Eq (1) are summarised as follows: (2)

Here T is the time variable (0 for 2008, 1 for 2009 etc.) and coefficient β_t represents the long-term trend in the study aa. β_m,pre, β_{m, int}, …, β_h,trend represent coefficients related to mine fire exposure. E_m and E_h are indicator variables for moderate and high exposure. I_post is the indicator variable for pre- or post-mine fire (0 for 2008–2013 and 1 for 2014–2018). T_post is the post-mine fire time variable (0 for 2008–2014, 1 for 2015, 2 for 2016 etc.). Therefore, β_m,pre and β_h,pre can be interpreted as the pre-mine fire differences when comparing moderate and high exposure schools with no/low exposure schools. These two coefficients are subsequently referred to as the fixed intercepts. The coefficients β_m,int and β_h,int can be interpreted as the mine fire interruptions effect (relative to the developmental trajectories of students) for moderate and high exposure schools, respectively. The coefficients β_m,trend and β_h,trend are the post-mine fire trend differences compared with the trend before the mine fire. Other than β_m,pre, β_m,int, …, β_h,trend the vector of coefficients, β, also contains coefficients of other confounding variables detailed above.

Estimation and modelling framework

Estimation was conducted using Markov chain Monte Carlo (MCMC) sampling implemented in the Stan programming language [58] via the RStan package [59]. Stan is a platform for high-performance full Bayesian statistical inference using the ‘No-U-Turn Sampler’ (NUTS) [60]. Weakly informative prior distributions were used for standard deviations (SDs) of random effects, namely, σ_s,c~TN(10, 5²), σ_s~TN(10, 5²) and σ_e~TN(10, 5²), where TN is the truncated normal distribution with support over the range. A mean of 0 is normally used for prior distributions of the random effects to take more advantage of the partial pooling effect of the multi-level model [61]. In our case, the mean of 10 was chosen as the high-performance schools were slightly underrepresented in the area and variations across all regional Victorian schools were used (based on a preliminary exploratory evaluation). Weakly informative prior distributions N(0, 50²) were used for all fixed effect parameters as non-informative priors based on uniform distributions can lead to a range of model fitting issues [58, 62].

Separate models were estimated for each testing domain. Results were reported as the posterior mean of estimated coefficients (4 Monte Carlo chains with 2000 iterations each), 95% credible interval (CI) and the probability of estimated coefficients (β) being greater or less than 0. Predicted centred NAPLAN score for schools were calculated and visualised for each testing domain and exposure group using line plots.

Sensitivity analyses

A range of sensitivity analyses were undertaken to test the robustness of results, including using different prior distributions and excluding cohort random effects. In addition, two schools in Morwell were relocated during the mine fire event and remained at their relocation sites for an extended period afterwards; hence sensitivity analyses were conducted excluding the two relocated schools. Code for fitting the models using synthetic data is provided in the Supporting Information I.

School characteristics

The characteristics of schools across exposure groups are presented in Table 1. The ICSEA socio-educational advantage scores were lower for schools in the high exposure group (Morwell) compared to schools in the moderate and no/low exposure groups. Other school characteristics, including the percentage of girls, the number of students, the school sector and the participation rates of the NAPLAN were comparable between exposure groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. School profile (in the year 2014) and NAPLAN participation rate by exposure area.

https://doi.org/10.1371/journal.pone.0281655.t001

NAPLAN score pre-and post-mine fire

Distributions of NAPLAN scores pre- and post-mine fire for each domain and grade level were plotted by exposure group, which are presented in Fig 3. While there was a general trend across all exposure groups for NAPLAN performance to decline post-mine fire, this was greater in higher exposure schools.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Box plot of school mean NAPLAN score for pre-mine fire period and post-mine fire period by exposure group for each domain and grade level.

https://doi.org/10.1371/journal.pone.0281655.g003

Bayesian hierarchical meta-regression models

The results of the Bayesian hierarchical meta-regression models are presented in Table 2 and Fig 4. Compared with the Victorian regional average, there was an estimated downward trend across all domains of testing for the schools in the three exposure groups (see Fig 4 and S1-S5 Tables in S2 File). As shown in Table 2, NAPLAN scores were found to be similar between schools in the moderate and no/low exposure group pre-mine fire after controlling for other confounding factors (i.e., there was no evidence that the fixed intercept coefficients for moderate exposure differed from 0). However, pre-mine fire NAPLAN scores were estimated to be lower in schools in the high exposure group for most domains when compared with no/low exposure schools (fixed intercept for high exposure ranged between -3.68 to -13.41, Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Trends of predicted centred NAPLAN scores (predicted margins when all covariates are fixed at reference values) with error bars by exposure group for each academic domain.

Note: centred NAPLAN scores represent score differences of schools compared with the regional Victorian average scores in the matching year, grade level and domain; the grey line indicates the time of the mine fire.

https://doi.org/10.1371/journal.pone.0281655.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Estimated intercept, mine fire interruption effect and post-mine fire trend difference for moderate and high exposure schools estimated from Bayesian hierarchical meta-regression models.

https://doi.org/10.1371/journal.pone.0281655.t002

Table 2 also shows that there were substantial interruption effects post-mine fire in high exposure schools across all academic domains, with estimated mean score reductions of between 10.31 and 11.09 for grammar and punctuation, numeracy, spelling and writing, and 8.34 for reading. Typically, NAPLAN scores increase by about 27 points per year [56], which means that the delay in educational attainment in high exposure schools was equivalent to four to five months.

After the initial drop in academic performance subsequent to the mine fire, there was evidence that writing and grammar and punctuation scores began to recover (positive slope) in high exposure schools, see Fig 4. However, there was no such improvement post-mine fire for spelling in high exposure schools (estimated slope of -1.49 post-mine fire; 95%CI: -4.20, 1.21). Furthermore, the predicted centred NAPLAN scores in 2018 were found to be lower compared to the pre-mine fire period in all academic domains for high exposure schools (see Fig 4), indicating incomplete recovery five years post-mine fire.

Sensitivity analysis

The choice of prior distributions for the SDs of random effects were found to have little impact on results; however, using a weakly informative prior distribution reduced the time taken to fit the Bayesian models compared with using a non-informative prior distribution. Sensitivity analysis excluding the cohort random effects (only the school level clustering is considered) produced very similar results except for a slightly larger mine fire interruption effect for high exposure schools (see S6 Table in S2 File). Models excluding the two relocated schools showed results that were consistent with all schools, but with slightly smaller interruption effects (see S7 Table in S2 File).

Links between the mine-fire and academic performance

The impact of the mine fire on academic performance may be attributable to a combination of factors. The mine fire led to ongoing distress experienced by students, teachers and parents in the smoke affected community post the mine fire [35, 48, 50, 64]. Hazelwood mine fire-related air pollution has also been linked to a range of adverse physical health effects such as respiratory morbidity [44, 45, 47, 65]. More broadly, the communities’ wellbeing was also negatively affected, with local residents reporting a loss of trust in authorities and feeling abandoned with little support provided [66]. The subsequent closure of the Morwell mine and Hazelwood power station some three years after the fire occurred introduced additional social and economic challenges in an already disadvantaged community [67, 68]. These factors may have impacted students’ learning abilities both directly and indirectly.

The mine fire also introduced disruption to day-to-day operations in nearby schools [48]. Two schools were relocated causing direct educational interruptions as well as threats to students’ sense of safety and security [48]. The models excluding these two schools showed a slightly lower level of interruption in high exposure schools compared with when they were included. This suggests that school relocation might have had an adverse impact on academic performance in addition to the mine fire exposure. Prior research has found that school relocation has a negative effect on students as well as teachers. Students are impacted by the change of routine, additional time travelling to new school locations, and uncertainty about the permanency of school relocation. Teachers experience similar uncertainty and concern about the effects of school relocation on students, and have reported that school relocation increases their workload and decreases time spent in the classroom with children [48].

Fire incidents are common in coal mines [69] and, all too often, mining project impact assessments adopt a narrow range of health and wellbeing domains that omit consideration of epidemiological evidence and potential social consequences [70, 71]. Accordingly, the impacts that events such as the Hazelwood mine fire can have on academic outcomes among school-aged young people warrant consideration within mine project planning and impact assessment, more generally, within disaster risk reduction planning and education department policies.

The potential links between air pollution and cognitive development, educational and behavioural outcomes in children is a rapidly growing area of research. Associations have been found between chronic ambient air pollution and cognitive function, as well as educational attainment [72–74]. A recent study suggested a link between academic underachievement, particularly in spelling, and prenatal airborne polycyclic aromatic hydrocarbons (PAH) exposure [75], a pollutant also found in the Hazelwood mine fire smoke [76]. However, we are not aware of previous research evaluating associations between medium-duration air pollution events and long-term academic outcomes. Although our study has shown a clear impact of the mine fire on academic performance, the model cannot delineate between the potential underlying mechanisms related to the exposure, such as psychological trauma, disruption to schooling, or air pollution exposure. Accordingly, further investigation is needed to understand the relative contributions of different mechanisms by which disaster exposure impacts academic performance, which could inform the development of targeted interventions.

Improvements post-mine fire

Gradual improvements in academic performance were observed across most domains post-mine fire among students from high exposure schools. This may be the result of a slow recovery at both the individual and the community level. Increases in school funding from the Victorian Government to upgrade school facilities and a range of implemented government strategies focused on improving health, wellbeing and access to services in the impacted community, may have assisted the recovery process [77, 78]. Students’ performance in the spelling domain continued to deteriorate post-mine fire, which may be related to idiosyncrasies in the English language orthography [79], or difficulties in teaching spelling [80, 81]. As difficulties in spelling may persist and lead to arrested writing development subsequently [82], further investigation is needed.

Comparison with other studies

Although a theoretical link between disasters and educational outcomes has been well-established [83], most studies have been limited to evaluations of how disasters impacted school attendance or dropout rates [30, 31, 84], and academic delays have rarely been investigated. One study suggested more than 75% of the African American children evacuated in response to Hurricane Katrina (which resulted in the loss of more than 1,300 lives, 800,000 homes and 110 schools) experienced a decline in grades [34]. To our knowledge, only Gibbs and colleagues have previously evaluated the impact of disasters using national standardised academic performance tests. Their study found that exposure to the 2009 Black Saturday bushfire in Australia (loss of 173 lives, 2,000 homes and 3 schools) was associated with delays in academic achievement from Grade 3 to 5 in reading and numeracy but not in writing, spelling, and grammar domains [29]. However, only academic progressions of a single cohort of students (who were in Grade 3 in 2011 and were around 7 years old at the time of the event) were evaluated. Our study, on the other hand, has utilized a time series design to assess academic achievement across all grade-levels completing NAPLAN testing and incorporated more historical data, which sheds further light on academic impacts across age groups.

Longer-term social outcomes after man-made or industrial disasters have the potential to be worse than in cases of natural disasters [85], however, it is at present unclear whether this also applies to academic outcomes. Furthermore, while there is little research available for assessing the relative impacts of different types of disasters on academic outcomes, it is reasonable to anticipate that more severe disasters will have a potentially greater impact on academic outcomes. For instance. if school infrastructure was destroyed, or if students were exposed to the trauma of having lost family or classmates in the event, then academic consequences might be expected to be worse than in the case of the Hazelwood mine fire. Accordingly, seeking to compare how different types of disaster shape academic outcomes would be a useful direction for future research.

Statistical modelling approach

To the best of our knowledge, this is the first study using a Bayesian meta-regression method to evaluate aggregated school-level academic outcomes. Existing studies evaluating the impact of environmental factors on academic outcomes mostly used individual-level data [29, 86, 87]. Where aggregate data was used, it was limited to school-level mean scores or proportions [88–90], which ignores the variations in score distributions.

Our method has several strengths. Firstly, it enables the evaluation of the spatial (geographical regions of no/low exposure to high exposure areas) and temporal profile of the impact, which could be used to inform policy and resource allocation in academic settings after disasters. The inclusion of an appropriate control group of schools from areas with similar socioeconomic circumstances, but not exposed to smoke during the fire, countered the problem of potential time-varying confounders (e.g., changes in tests, education policy or other major interruptions) and enabled the impact of the mine fire on academic progress in the region to be clearly isolated [91].

Secondly, the method can be applied using only readily available aggregate data at the school-level to obtain unbiased estimates without losing statistical power to detect meaningful differences. Although only 69 schools were included in our analysis, the study was effectively powered by over 30,000 students who studied in these schools from 2008 to 2018. Accessing aggregated data is a particularly useful approach to conducting research in this area as it circumvents the challenges of attempting to recruit individuals in communities’ post-disaster who are already burdened and traumatized as a result of their experiences and is a much less resource-intensive approach. A similar approach can also be used without the interrupted time series design to study the associations between students’ education performance and other environmental factors such as air pollution.

Lastly, the interrupted time series design facilitates the understanding of immediate interruptions as well as changes in long-term trends post-event. Compared with simple pre- and post-comparisons, this approach provides information for teachers, schools and education departments not only on the magnitude of the impact but also on the effect of post-disaster interventions to accommodate students’ additional needs.

Similar models can be fitted via a frequentist regime using mixed-effect meta-regression models (e.g., estimated using restricted maximum likelihood), which can yield similar results to Bayesian models [92]. However, fitting models with the Bayesian approach has additional advantages including more stable and generalisable estimates, integration of existing knowledge using prior distributions, flexible compressions using posterior distributions and avoiding convergence and estimation problems of maximum likelihood methods [61, 93]. The choice of prior distributions did not impact our results, however, when there is high model complexity (e.g., higher numbers of potential confounding factors), penalising Bayesian models can be used with shrinkage prior distributions [94, 95].

Limitations and future direction

This paper provides an easily adaptable method to evaluate the impact of different types of community-level traumatic exposures (e.g., disasters and disease outbreaks) on students’ academic progression using only aggregated school-level data. However, the study has recognised limitations. The implemented model assumes that student NAPLAN scores within schools were normally distributed, which could be unrealistic if, for example, distributions were skewed or the number of students is small. Although the number of schools in the high exposure group was modest, most of these schools had over 150 students enrolled, so the mine fire interruption effects identified are likely to be robust. More complicated models such as random slopes, non-linear long-term trends and autocorrelation structure, were not considered due to the unwarranted increase in modelling complexity with limited data. These models can be applied when there are more schools and time points available. More detailed region-specific data on other risk factors, such as service availability, were not available for this analysis, but could in principle be included in the proposed model.

Outcomes across multiple learning domains were compared which can introduce challenges for interpretation. In this study, it is likely that interruptions caused by the mine fire to school routines were distributed similarly across all learning domains; however, the level of impact of this disruption on students’ subsequent progression within each learning domain may differ. Therefore, each domain was evaluated separately to investigate for any differing impacts and trends between domains. Alternatively, if there was an assumption or interest in evaluating a consistent impact across learning domains, all could be included in one multilevel model with an additional nested level [61].

Furthermore, the aggregated nature of the data precluded controlling for individual-level risk factors and, hence, there is potential for ecological bias within the study. While the interrupted time series design was used to allow post-mine fire scores in individual schools to be compared relative to their past, there may be changes in students’ profiles over time, which may impact our findings. For example, some families may have left the region due to the subsequent closure of the mine and power station. Advanced population-level data linkage methods can be used to explore these issues. However, as noted above, accessing population-level data can be challenging and time-consuming.