Study aims

This study explored the gap between statistical theory and practice by focusing on how statistical assumptions are reported and addressed in health-related research. Specifically, it evaluated the reporting practices for linear regression assumptions in 100 randomly selected papers published in PLOS ONE in 2019. Each paper was subjected to a post-publication review by statisticians to assess whether the authors reported checking assumptions, which assumptions were addressed, and how these were reported.

Research questions

• What was the prevalence of publication author teams who reported in their manuscript that they had checked linear regression assumptions?
• Did author teams correctly check assumptions, and were misconceptions about linear regression assumptions evident in their reporting, as inferred from their methods and descriptions?
• What was the agreement of ratings for statistical assumptions made by different statisticians?

Linear regression

This study was not designed to be an in-depth tutorial on linear regression but rather an overview so that the paper is accessible to non-statistical readers, for further reading on linear regression, see [22–24]. Linear regression models explain the relationship between a dependent variable and one or more independent variables by fitting a linear equation. In this equation, each independent variable is multiplied by a corresponding parameter, often referred to as the ‘regression coefficient’. In its simplest form, when considering just one independent variable, average change in (Y) is predicted for each unit increase in (X), assuming that Y depends on X [25]. For example, how much does body weight change (on average) with a one-year increase in age? Linear regression allows the exploration of this relationship’s direction and magnitude. While this paper focuses on linear regression assumptions, the second paper in this series examines reporting practices in linear regression, including key statistics to report and best practices for presentation, see Jones et al. [26].

Statistical inference is used to make conclusions about a population based on sample data. The main goal is to estimate parameters, such as regression coefficients, that describe relationships between variables. Since the entire population is typically not observed, these parameters are unknown and must be estimated using sample data. This paper presents sample-based regression equations with estimated values denoted by hats (e.g., for regression coefficients). In these models, error terms represent the differences between observed and predicted values based on the population regression line. However, since error terms are unknown, we use residuals, sample-based differences to assess the model’s assumptions.

Linear regression models are commonly fit using ordinary least squares (OLS) [22], which minimises the sum of the squared differences between observed values and their predicted values as represented by the regression line. This method identifies the “best-fitting line” by ensuring the total squared deviation between the actual data points and the predictions is as small as possible. Squaring these residuals serves two purposes: it prevents positive and negative deviations from cancelling out and provides a consistent measure of how far data points are from the line. Measuring variability using squared deviations is central to statistical concepts like variance, which is calculated as the average squared deviation between each data point and the mean. The square root of the variance is the standard deviation (SD), which describes the spread of data in the same units as the original values.

Linear regression, as described above, is, strictly speaking, a mathematical optimisation method where the best-fit line is estimated Eq 1. However, we must quantify uncertainty around our estimates to draw statistical inferences, which requires certain assumptions. Assessing whether these assumptions hold is essential for reliable interpretation of results. One way to do this is by analysing the residuals, which are the differences between observed and model-predicted values (fitted values) (Fig 1). The adequacy of the model fit can be assessed by analysing the distribution of the residuals and identifying any patterns or systematic deviations from the regression model’s assumptions.

(1)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Pictorial representation of a linear regression and residual, with the red line representing the line of best fit, the dots are the measured (observed) data.

The residual is the difference between the observed and predicted values.

https://doi.org/10.1371/journal.pone.0299617.g001

In Eq 1, where with N being the total number of observations: is the dependent variable for the ith observation. is the Y-intercept, which is the estimated value of Y when X = 0. is the slope, representing the average change in Y for a one-unit increase in X. The error term is for the ith observation, representing the deviation of the observed value Y_i from its predicted value based on the regression model. This equation can be extended to incorporate multiple X variables (k parameters) as seen in Eq 2.

(2)

To undertake hypothesis tests and create confidence intervals that give realistic approximations of the underlying relationship between variables, it is assumed that residuals are: (i) normally distributed (with a mean of zero), (ii) the relationship between the outcome Y and the predictors is linear, (iii) have constant variance, and (iv) are independent. This can be visualised by plotting the residuals against the predicted values (Fig 2). If residuals are approximately normally distributed, standardising them allows problematic observations to be identified when values exceed , as 99.7% of observations should fall within this range in a standard normal distribution. The terms predicted and fitted values may often be used interchangeably, but fitted refers to the values estimated by the model using the same data that was used to create the model. Whereas predicted is used in a broader sense, it may refer to the fitted values or data in a new dataset that was not used to create the model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example of residuals from a linear regression model.

This example shows no clear violation of linearity, non-normality, or homoscedasticity, with the red line showing a hypothetical mean of zero, where there should be as many points above as below the line.

https://doi.org/10.1371/journal.pone.0299617.g002

Assumptions.

1. Normality: The model residuals are assumed to be normally distributed (with a mean of zero).
2. Linearity: The mean of the dependent variable Y is assumed to change linearly as a function of the independent X variables and their associated parameters.
3. Homoscedasticity: The residuals are assumed to have constant variance across all values of X.
4. Independence: The residuals are assumed to be uncorrelated.

Normality.

Characteristics of a normal distribution include a symmetrical bell-curved shape around a mean, with mean, median, and mode all equal, and 95% of observations falling within approximately two (more precisely 1.96) standard deviations. Linear regression does not require the X or Y variables to be normally distributed, this assumption is only related to the residuals. Violation of normality does not necessarily lead to bias of regression coefficients, which depends on the sample size and degree of the violation. Non-normality of the residuals can lead to inaccurate estimates of p-values and confidence intervals and increased type I errors [27]. Regression models are generally not sensitive to small deviations from normality in the residuals, especially in large samples, but can be sensitive to heavy-tailed distributions or the presence of large outliers and influential points [28, 29]. The best way to check this assumption is through examining the residuals with descriptive statistics, including examining if the mean and median of the distribution of the residuals are similar, exploring if skewness and kurtosis are within reasonable bounds, and visually through plots including histograms and quantile-quantile plots (Q-Q Plot) (Fig 3) [27]. The Q-Q plot is created by plotting quantiles from the data against quantiles generated from the normal distribution [30]. The points should follow an approximately diagonal line if the data are normally distributed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Example of assessing the normality of the model residuals with plots, with points on the Q-Q plot generally following the diagonal line with a small deviation at the end and an approximate bell-shaped histogram, indicating that the residuals in this example are approximately normally distributed.

https://doi.org/10.1371/journal.pone.0299617.g003

Linearity.

The linearity assumption is that the relationship between the predictor terms of the model (which are typically the X variables or powers of them such as X²) and the average change in the Y variable (dependent variable) are linearly related through model parameter/s, i.e. the regression coefficient/s. There is a common misunderstanding about linear regression models that each independent variable must relate linearly, which is taken to be a straight line, to the dependent variable, Y. This stems from the focus on simple linear regression, where there is only one X variable. In multivariable models, independent variables can be represented through multiple parameters, for example, age and age-squared, thus capturing more complex relationships, including splines and polynomials (e.g. quadratic, cubic, etc.). Even though X² is a nonlinear transformation of X (quadratic), the relationship remains linear in terms of the parameters. This is because, in linear regression, the expected value of Y is assumed to be a linear function of the parameters (coefficient/s) [31].

The linearity assumption can be visually checked through scatterplots of residuals against individual variables in the model as well as predicted values [27]. The residual plots should be examined for patterns, including curvature, which may indicate non-linear relationships. Fig 4 shows an example of linearity violation where a strong quadratic relationship is missing from the model. Understanding the underlying relationships in data is essential to resolving any problems. In the example above, the best solution is to identify the variable responsible for the issues and precisely model the polynomial relationship by properly adding a squared term to the model for that variable. Another solution may be to transform the data using the so called “ladder of transformations”. For more detail, see [32].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Example of a non-linear relationship missing from the model that is detectable in the residuals.

The red line indicates a quadratic relationship between the residuals and fitted values.

https://doi.org/10.1371/journal.pone.0299617.g004

Homoscedasticity.

Homoscedasticity, also known as homogeneity of the variance, assumes that the residuals have constant variance and are distributed equally for all independent variables [33]. For example, in a linear regression model, where blood pressure depends on age. If the variance is constant or homoscedastic, then the variance of the errors will remain constant for all values of X. Therefore, the residuals will be equally spread around Y = 0, and the residual scatter should be similar at young and old ages. Suppose the variance is not constant (heteroscedastic). In that case, a plot of the residuals may show that being younger corresponds with a narrow range of low blood pressure, but as people age, blood pressure varies more widely. This may cause a funnelling shape in the residuals as seen in Fig 5 and may indicate that other variables explain blood pressure, such as several chronic conditions or smoking status.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Example of residuals displaying heteroscedasticity, where a funnel shape can be observed in the residuals instead of random scatter around zero (red line).

https://doi.org/10.1371/journal.pone.0299617.g005

Homoscedasticity violations can have serious consequences as they can bias the standard error, causing inaccurate significance values and confidence intervals, leading to increased type I error [34]. Diagnosis of heteroscedasticity is best made by visualising the residuals and predicted values using scatterplots. Still, it can also be assessed statistically with methods such as the Breusch Pagan test [35], although statistical tests are prone to over-detecting minor deviations in large samples and may lack power in small samples. As the cause of heteroscedasticity may not always be easily detected, it is essential to understand the relationships in the data with both clinical understanding and plots. Remedies for heteroscedasticity may include weighted regression [36], robust standard errors [35], data transformation [22], including other variables in the model that improve prediction, or bootstrapping with a heteroscedasticity correction [34].

Independence.

Linear regression assumes that each observation is independent of the other and that their residuals are uncorrelated. A commonly observed violation of independence generally involves repeated measures [27], for example, blood pressure measured over time in the same person. Measurements taken on the same participant (within-participant) are likely to be more similar than observations between participants. When generalising results to a population, treating correlated observations as independent can lead to an underestimation of the variance in the linear regression, making estimates appear more precise than they are in reality. This may lead to increased type I errors, making the accuracy of standard errors and confidence intervals questionable [37].

In addition to repeated measures, health research frequently involves other data structures where a correlation between observations (clustering) may be present [27] such as patients nested within doctors. The experience and ability of individual doctors may influence patient outcomes. Therefore, patients treated by the same doctor may not be independent. The independence of observations should be a part of the study design and sometimes may not require testing but should always be discussed. A lack of independence in the data can be visualised by plotting residuals by the individual (row number) to look for serial correlation (autocorrelation), when there are no violations; points should fall randomly around the zero line, which can be assessed using the Durbin–Watson test [27]. Suppose there is suspected clustering of the data. It can be examined by fitting an appropriate statistical method, such as a random effects model, to adjust for correlated observations. Therefore, the general remedy for independence violations is to use a method such as Linear Mixed Models (LMM) or General Estimating Equations (GEE) to account for the non-independence of data.

Outliers and influential observations.

When undertaking statistical methods such as linear regression, it is important to identify influential observations that, if removed from the model, can substantially change the regression coefficients [33]. While the presence or absence of outliers and influential observations is not an assumption of linear regression, they can potentially change results and may be the cause of assumption violations.

Outliers should not be routinely deleted. To minimise questionable research practices, the study protocol should address the management of outliers, such as data transformation, the use of robust regression, sensitivity analysis, bootstrapping, or variable truncation [38].

Two ways in which a single data point can affect the results of a model are when the observation is an outlier and/or has high leverage [39]. An outlier can be defined as where the response (Y) does not follow the general trend and falls outside the range of the other values. Outliers generally have large residual values with a sizable difference between the observed and predicted data. Leverage measures the distance between an observation’s X value and the average X variable values in the data. Observations with extreme values of X are said to have high leverage [33]. Data points that display high leverage and/or are outliers have the potential to be influential but must be investigated to determine if they substantially change regression coefficients. A way to measure this change is known as Cook’s Distance and is a combination of each observation’s leverage and residual values [23].

In small datasets, large outliers can strongly influence regression coefficients, sometimes even changing the direction of the relationship. For example, consider a study measuring systolic blood pressure (SBP) and age in 10 adults. A good initial step in any linear regression is to visualise the data. In this case, SBP and age have a positive linear relationship, but a large outlier is observed for a 75-year-old with unexpectedly low SBP.

A sensitivity analysis (Fig 6) was performed to assess the impact of this observation. The estimated slope of SBP per unit increase in age nearly doubled, from 0.43 to 0.77, when the outlier was excluded. Visually, it is clear that including the outlier pulls the regression line downward at older ages, resulting in a poor fit to the overall trend. However, the outlier likely represents a real observation, as some individuals naturally have low blood pressure throughout their lives. The large difference in R² and the regression coefficient between the two models suggests that they are capturing different patterns in the data. The model excluding the outlier has a higher-than-expected R², potentially overstating the strength of the relationship between age and SBP. This highlights the need for further investigation and adjustments, such as incorporating relevant clinical factors to improve model fit or applying data transformation or robust regression techniques.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Example of a sensitivity analysis showing linear regression results with outliers included and then excluded, using simulated data.

https://doi.org/10.1371/journal.pone.0299617.g006

Beyond assessing the impact of excluding the observation, we can examine diagnostic measures such as Cook’s distance to determine whether this point has excessive influence on the regression model. One way to visualise this is through an influence plot, which maps standardized residuals against leverage, with Cook’s distance contours overlaying the graph (Fig 7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Influence plot: showing the relationship between leverage, standardized Residuals, and Cook’s distance, with thresholds at 0.5 and 1 indicating potentially influential observations.

https://doi.org/10.1371/journal.pone.0299617.g007

In this plot:

• Leverage (x-axis) measures how much an observation’s X values differ from the rest of the data, meaning points further to the right have more potential to influence the regression line.
• Standardized residuals (y-axis) show how far an observation’s Y’s deviates from the model’s prediction, helping identify extreme values.
• Cook’s distance contours indicate the level of influence; points beyond the threshold values (often 0.5 or 1) suggest that these observations may disproportionately affect the model’s estimates and should be examined further.

Observations that have both high leverage and large standardized residuals, particularly those exceeding the Cook’s distance threshold, should be carefully examined. While some authors suggest a threshold of 4/n for small samples, others recommend a cutoff of >1 [22, 23], particularly in larger datasets. However, whether >1 is a more or less conservative threshold depends on sample size, in small datasets, it may be too strict, potentially overlooking influential points, while in large datasets, it may be too lenient, allowing highly influential points to remain undetected. While not all high Cook’s distance points are problematic, they signal cases where a single observation may significantly alter model estimates. These points should be investigated by checking data accuracy, verifying experimental conditions, and reassessing model assumptions to ensure the results remain meaningful.

The appropriate response is context-dependent, there is no one-size-fits-all approach to handling potential outliers. If an unusually low value is caused by a known experimental error (e.g., a mistake in drug administration, such as only half the intended concentration being added), it would be reasonable to exclude the observation. However, if the low value represents natural variation, then data transformation or robust regression methods may be more appropriate to reduce its influence while preserving the integrity of the dataset. In small sample sizes, it is easy to assume that any data points deviating from the rest are outliers and should be removed. However, in smaller datasets, we might naturally observe less variation simply by chance, making unusual-looking points appear more extreme than they actually are. Removing these points risks eliminating natural variability causing biased estimates.

Regardless of the approach taken, transparent reporting is essential. Any decisions about outlier exclusion, transformation, or alternative modelling approaches should be clearly stated, along with the rationale behind them. Reporting results both with and without outliers (sensitivity analysis) provides a more complete picture of the data and ensures that findings are not driven by selective exclusion. By clearly documenting these steps, researchers can enhance the credibility, reproducibility, and interpretability of their findings.

Multicollinearity.

Multicollinearity occurs when two or more independent variables in a regression model are highly correlated, making it difficult to isolate their individual effects on the dependent variable [22]. This issue primarily affects the interpretability of the model, as the coefficients of correlated variables may become unstable, and their standard errors may be inflated. When multicollinearity is present, the shared variance among predictors reduces the amount of independent variance each variable uniquely explains in the model, making it harder to determine their true effects. Multicollinearity can be assessed using the variance inflation factor (VIF), which quantifies how much the variance of a coefficient is inflated due to correlation with other predictors [40]. Higher VIF values indicate that a variable shares variance with other predictors rather than contributing unique explanatory power to the model.

There has been debate about the threshold of VIF that indicates a problem, with values of 3, 5, and 10 commonly cited in the literature [40]. However, like all statistical cut-offs, VIF should be interpreted in the context of its effect on the model. For example, in a sensitivity analysis where a high-VIF variable is removed, multicollinearity may not be a significant concern if the standard errors and interpretation of variables remain stable across models and align with their univariate results. However, if the standard errors or direction of coefficients change drastically, even VIF values as low as three may indicate a problem.

An example where multicollinearity could be a concern is the relationship between age and years of experience when studying factors influencing doctors’ adherence to medical guidelines. Older doctors generally have more experience, while younger doctors’ experience is inherently limited by their age. Because of this strong correlation, distinguishing the independent effects of each variable may be challenging. If multicollinearity is problematic, age is often dropped in favour of years of experience, as the latter provides a more direct measure of expertise. However, the decision depends on the specific analysis and research question.

Another way to address multicollinearity without dropping variables is to use regularisation methods such as Lasso or Ridge regression. These methods help manage multicollinearity by penalising model complexity, shrinking coefficient estimates, and reducing sensitivity to collinear predictors. Lasso can also perform variable selection by setting some coefficients to zero, while Ridge retains all predictors but shrinks their impact [41].

Sample size

A sample size of 100 papers was found to be adequate to detect a sample proportion of 0.05 (5%) using a two-sided 95% confidence interval with a margin of error of 5%. This sample size was calculated using a test for one proportion with exact Clopper–Pearson confidence intervals, using PASS [43]. For these papers, it was deemed feasible to recruit 40 statisticians (40 statisticians 5 papers = 200 reviews), and from our experience and feedback during the development stage, having each statistician review five papers was manageable. Each paper was rated twice by two independent statisticians to increase the robustness of the results and provide data on the agreement in statisticians when checking assumptions.

Question development

A set of questions was developed to understand current reporting practices for linear regression analysis. The questions were adapted from the Statistical Analyses and Methods in the Published Literature (SAMPL) regression guidelines [44]. A literature review was also used to identify common errors made by researchers when reporting linear regression, and a comprehensive list of 55 questions was developed to assess statistical quality. It was decided by the research team, consisting of three Australian accredited biostatisticians, to reduce the burden on reviewers by substantially reducing the number of questions. We focused on questions important to assessing assumptions and interpreting linear regression. The research team used an iterative approach to improve the wording of the questions by reviewing papers to understand issues reviewers may encounter. When these three statisticians were satisfied that the questions were appropriately worded, five independent experts (four biostatisticians and an epidemiologist) were given a briefing on the aims of this study and the questionnaire. They were asked to assess the questions and provide feedback on readability and length by examining two randomly selected papers. Their feedback was used to further reduce the questions to the current checklist of 30 items, which can be seen in S1 Table.

Randomisation

The randomisation process of selected papers occurred in two steps, as described below.

Measured outcomes

The outcome measures were the prevalence of statistical assumptions reported by authors in the randomly sampled papers, including normality, linearity, homoscedasticity, and independence, as well as reporting on outliers and multicollinearity assessment. Statistical raters first determined whether the authors reported checking assumptions and then selected from a predefined list of how assumptions were assessed and what was assessed. Since all responses were dichotomous, if an assumption was not ticked, it was assumed to be unreported. Statistical reviewers were prompted to check the supplementary by asking, “Was most of the detail for the assumption checks in the supporting information (appendix)?”

Statisticians also assessed whether authors reported checking for outliers and identified the action taken in a question, with response options such as no action taken or sensitivity analysis. Another question asked whether multicollinearity was assessed, with possible responses of yes, no, or not required. For further details on the questions asked, see S1 Table. The primary author, LJ, also gathered data about assumption checks, reviewed all papers and made qualitative notes about how assumptions were checked, misconceptions, and possible reasons for reviewer disagreements.

Data analysis plan

This confirmatory study examined the reporting and quality of linear regression assumptions of published papers in the health and biomedical field. Reporting behaviours were described using frequencies and percentages, with Wilson 95% confidence intervals used to account for prevalence values close to zero.

The agreement of raters was described using observed agreement and Gwet’s statistic [51], which performs well in situations of high agreement. Quadratic weighted Gwet’s agreement was used for ordinal ratings. This weights disagreements according to the square of their distance on the scale and gives greater weight to larger disagreements compared to smaller ones. Assumptions of Gwet’s agreement were considered and found to be acceptable, testing these assumptions is not required, as they are related to the design of the experiment, such as appropriate rating scales. Gwet’s agreement was used for categorical data that was either nominal or ordinal. Gwet’s agreement is less sensitive than Cohen’s kappa to the distribution of ratings across categories (marginal distributions), which may be caused by statisticians rating different papers.

R version 4.4.2 [52] was used for all statistical analysis. This was a descriptive study with no formal hypothesis tests used, with prevalence and 95% confidence intervals reported, while statistical ratings were assessed for reliability. To increase transparency, the STROBE guideline was used for reporting cross-sectional studies [53].

Calculating prevalence and reliability

This study was initially designed so the prevalence of individual assumptions could be estimated using the input of two statistical ratings, with the primary author (LJ) adjudicating disagreements. After the ratings for the first few papers were received, a test was carried out, where the primary author rated the papers and then checked agreement against the two raters; it was realised that due: (i) to the complexity and length of the papers and (ii) sometimes nuanced interpretation, a more comprehensive picture of prevalence was gained through all three ratings. Therefore, the prevalence of reporting behaviours was calculated using all three ratings. The primary author independently rated papers by filling in the survey and making notes on the PDF for the papers. The primary author adjudicated the difference between all three ratings by returning to the paper, making notes, and documenting each disagreement. Authors DV and AB were consulted when decisions were unclear. Reliability was calculated for the two statistical ratings, then the final prevalence rating was used as a gold standard to further assess the agreement of the two ratings separately. Missing data from reviewers was addressed in the reliability analysis by substituting the primary authors’ rating.

Reporting of linear regression assumptions and misconceptions

Of the 95 papers rated, 60 (63%) did not have any reporting of assumptions, 21 (22%) papers reported they checked one assumption, 9 (9%) reported on two assumptions, 5 (5%) checked three assumptions, with no author teams checking all four assumptions. Linearity was not required for 12 papers as they had no continuous independent variables; for these papers, only two checked one assumption (normality), with no other assumptions reported.

The questions initially asked if an assumption was checked, and then statisticians were asked to tick the boxes of how and what was checked. To avoid confusion with percentages, it was decided to keep the interpretation of how and why assumptions were checked at the level of papers rather than individual analyses within papers. Authors commonly reported checking continuous/quantitative data for normality but did not talk specifically about the regression analysis or residuals. Of the 28 (29%) papers that checked normality, five correctly checked the residuals (Table 3). Only three papers displayed some results of these checks, with partial reporting of results, with one paper reporting the optimal transformation result using a Box–Cox transformation and the other two showing box plots. A further six author teams presented box plots but did not mention normality. The same five papers that correctly checked residuals for normality were the only papers with a strategy for checking assumptions for linear regression.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Observed prevalence and 95% confidence intervals for statistical assumptions and outliers.

https://doi.org/10.1371/journal.pone.0299617.t003

Of the 83 papers that required linearity assessment, 15 (18%) directly assessed linearity, with a further 24 papers displaying scatterplots but not discussing them. Six authors who discussed linearity used scatterplots of the raw data to visualise relationships between variables; residuals were discussed in two papers; six papers used a test to assess linearity, with four authors fitting polynomials and two using splines. Homoscedasticity was discussed by 6 (6%) author teams, with most of these authors correctly checking this assumption using the residuals. Independence was addressed in 5 (5%) papers, while another 16 papers mentioned that their studies were cross-sectional but did not directly discuss independence.

Multicollinearity was assessed in 13 (14% CI: 8%, 22%) papers; 60 (63% CI: 53%, 72%) papers did not assess multicollinearity, which was not required for 22 (23% CI: 16%, 33%) papers as they had only univariate models [26]. Correlation was used to assess multicollinearity in 6 papers, in four of these papers pairwise correlation was the only assessment; three of these mentioned moderate to high correlations without checking VIF. Another four authors mentioned multicollinearity checks but did not provide any details. Five authors reported they used VIF to check multicollinearity, with two authors using cut-off <5 another using <2.5; a further two reported in terms of the highest VIF to show there were no issues. No authors reported checking the standard errors or coefficient stability of regression models to determine if variables with high VIF or correlation were actually a problem.

Ten misconceptions regarding linear regression assumptions, outliers, and multicollinearity were identified through a combination of the prevalence of the questions and the main author’s review of the papers (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Common misconceptions for linear regression assumptions and outliers observed and inferred by this study.

https://doi.org/10.1371/journal.pone.0299617.t004

Agreement of statistical raters

The agreement between statistical raters on assumptions and outliers was high (Table 5, for full reporting, see S3 Table), with observed agreement of over 80% for all assumptions except independence, which had a slightly lower agreement of 78%. When considering agreement by chance for independence, the Gwet’s statistic was 0.69; while this is still regarded as good agreement [54], it is an arbitrary threshold and was lower than expected given expert raters. Reviewing the disagreements, a number of authors stated that their studies were cross-sectional without discussing independence and potential clusters within the data. Therefore, simply mentioning a cross-sectional study design was not considered an assumption check. The other main reason for disagreement among raters was that some papers contained plots without any discussion of assumptions, these were counted but not considered an assumption check, as it is possible to show a scatterplot with a non-linear relationship present. When comparing statistical ratings to the final calculated prevalence (gold standard), these were generally higher than between raters, indicating good overall reliability in the study. While there was no missing data in the assumptions and outliers section of the questionnaire, four of the 190 reviews were rated by one of the statisticians to have no linear regression, so they were replaced by the primary authors’ rating for the reliability analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Agreement and reliability of statistical raters.

https://doi.org/10.1371/journal.pone.0299617.t005

Poor agreement between raters was observed for multicollinearity, with a raw agreement of 65% and Gwet 0.43 CI: 0.23, 0.62; the low agreement was likely due to poor reporting [26]. It was often difficult to determine whether papers should be placed in no or not required categories, as it was often unclear whether reported results were univariate or multivariable models [26].

Statisticians were asked to rate the statistical quality of each paper on a Likert scale with one representing very poor and five indicating very good. Gwet’s using quadratic weights showed good agreement (0.74 CI:0.64, 0.83) between raters. After averaging the ratings, the mean ratings for papers were 2.5 SD: 0.8, indicating that statisticians rated the statistical quality between poor and fair.

Peer and statistical review

Although peer review is considered the most trustworthy way of selecting manuscripts for publication and improving the quality of papers in medical journals, Cobo et al. [62] advise that there is very little evidence to support this view. Altman [63] suggested that reviewers are often no more knowledgeable than the authors and recommended that statistical reviewers be used to reduce errors and improve quality. In the only randomised controlled trial in assessing the effectiveness of statistical review [62], papers were allocated into four groups (1) clinical reviewers (control group); (2) clinical reviewers plus a statistical reviewer; (3) clinical reviewers with a checklist; and (4) clinical reviewers plus a statistical reviewer and checklist. This study concluded a measurable improvement in the quality of papers with statistical reviewers, but no improvement in quality was observed in the checklist group. Statistical review results in important changes to manuscripts above and beyond average review about 60% of the time and is essential in improving the quality of published manuscripts [64]. The generally low ratings for papers by statisticians in our study indicate that authors would have benefited from statistical reviews pre-publication and can still benefit from feedback from this post-publication statistical review. We found that the methods sections were often unclear and did not have a detailed account of assumptions checked. While it is encouraging that many researchers are using scatterplots to visualise data, the discussion of assumptions remains sparse.

It is recommended that statistical reviewers should always be part of editorial teams. The method (linear regression) reviewed here is commonly used in the health field, and assumptions are relatively straightforward to interpret. If we extrapolate, the problems are expected to be greater for more complicated methods such as mixed models, structural equation models, etc. It is recognised that the volume of papers going through journals means that a statistician will only view a small proportion of papers going through to publication. Therefore, journals should invest in basic statistical resources for researchers and reviewers. While PLOS ONE already recommends the use of the SAMPL guidelines [44], which include confirming that regression assumptions are met, our research found low adherence of following these guidelines for either reporting assumptions or general reporting [26]. This suggests that reviewers could encourage compliance by providing direct links to the guidelines when poor reporting is identified. A limitation of the SAMPL guidelines is that they are text-based; incorporating visuals and tables could enhance clarity and highlight correct reporting practices.

There is also an opportunity to implement automated tools to search for tests and match appropriate assumptions in documents [65, 66]. While this approach should not replace human reviewers, it can complement them, save reviewers time, and produce automated feedback to researchers, directing them to statistical resources.

Researchers discussing assumptions would be a big improvement on current practice. Detailed assumption checks can be placed in supplementary tables and plots. Journal editorial policies should also be considered. In most journals, page/word limits result in relatively limited space for statistical methods, although some of this can go into supplemental materials. It is possible that some teams did the appropriate checks but chose to avoid reporting on them due to reducing complexity (saving space) or the perception that doing so could make the review process more difficult. PLOS ONE does not have page or space limits, so this may be less of a consideration in this case. However, the author’s normal reporting practice would be expected to affect how statistical sections are reported. It is recommended that journals focus on good scientific practices for statistical sections, which focus on describing what was done in enough detail so that another researcher could reproduce the results.

Strategies for teaching statistics

In teaching statistics to health professionals, it can be tricky to get the balance right between too much theory and too little. Introductory statistical courses may compound problems for health professionals, which are often taught in a cookbook manner, where there is no emphasis placed on investigating the appropriateness of statistical methods [67, 68]. Our results reinforced this view with many papers checking only normality, often with generic statements about continuous variables. Several authors used combinations of univariate non-parametric tests followed by linear regression to do multivariable modelling without commenting on assumptions. These results suggest that importance needs to be placed on underlying statistical theory rather than teaching statistics as an isolated series of tests so that methods can be put in context and better understood by relating them to other methods.

First-year statistical courses often emphasise t-tests, ANOVA and regression individually with well-behaved data. Students are then offered the alternative of the non-parametric test if data is not normally distributed. This basic understanding of assumptions of parametric tests is pervasive. Many researchers are unaware that t-tests, ANOVA and linear regressions can be seen in a general linear model framework [69], where X variables can be either categorical or continuous. This knowledge is vital in selecting the correct choice of statistical tests, and assumptions are related to the residuals rather than the raw X or Y variables. Researchers often feel comfortable testing for normality and using non-parametric tests because it gives a binary answer, and there is comfort in following exact rules. While there is nothing wrong with using non-parametric tests, they often lack power, and the choice of descriptive statistics should fit in with the overall goal of the analysis. If the purpose is multivariable modelling, using non-parametric statistics for the univariate step does not make sense. It is recommended that health professionals be taught more holistically with a bigger picture of ‘everything is a regression’ [70], emphasising statistical thinking where students become more comfortable with uncertainty, and statistical assumptions be taught in the broader overview of modelling rather than a narrow univariate sense.