Reader Comments

Post a new comment on this article

Distinctions between total and indirect effects

Posted by JHilgard on 13 Apr 2016 at 21:43 GMT

I congratulate Drs. Gabbiadini, Riva, Andrighetto, Volpato, and Bushman on their newest publication and salute them for choosing PLOS ONE. I have downloaded the data as provided by the authors at their OSF repository, https://osf.io/3dnbh/ and performed my own set of analyses.

I have successfully reproduced and confirmed the following statistical results as reported by the authors:
There is no main effect of game on the empathy outcome, F(2, 151) = 1.17, p = .313.
Masculine beliefs significantly differed by game, F(2, 152) = 3.34, p = .038. Tukey HSD indicates that this is because the nonviolent game group has lower masculine beliefs than the violent-sexist game group (p = .030). The other two pairwise comparisons are not significant (p = .243, p = .575 for violent-vs-violent-sexist and violent-vs-nonviolent contrasts).
In a structural equation model, there is a significant path from Avatar-Identification × Gender × Game (where game is coded 1, 0, -1 for violent-sexist, violent, and nonviolent) to masculine beliefs, Z = 2.711, p = .007. There is similarly a significant path from masculine beliefs to the empathy outcome, Z = -4.51, p < .001.

I would like to provide the following additional analyses:
The double-moderated mediation model might lead one to believe that a similar GLM would find significant effects on empathy. That is, one would expect to find empathy predicted as a function of a three-way Game × Gender × Avatar-Identification interaction, as in the path model.
However, this is not the case. The three-way interaction is not significant, F(1, 145) = .363, p = .548. Nor are any of the two-way interactions and main effects significant (all ps > .10). This is true also if one treats Game as being coded numerically (1, 0, -1) in order to save a degree of freedom.

I have not attempted to reproduce the estimates of the indirect effects. I have to admit that I do not fully understand the conditional indirect effects as presented in Results, Primary Results, paragraph 6. Is it indeed possible to model the indirect effects of game while conditioning on game? Would it not be better to estimate the bootstrapped indirect effect of the games by conditioning on something else? My inexperience with process modeling is showing, but I hope you understand my confusion.

In short, the authors have correctly identified in their text that there is no significant total effect of game on empathy (Results, Primary Results, paragraph 4). Instead, they have a significant moderated-mediation process model. While I understand that a significant indirect effect does not require a significant total effect, it would seem to suggest a delicate and nuanced interpretation of results is necessary.

The specificity of this double-moderated indirect effect, given that there is no significant total effect, suggests that care might be taken in the discussion. At times the authors are appropriately careful, saying that "This research] address[es ...] who is most likely to be harmed by violent-sexist video games." But at other times, the authors seem to speak too broadly, as though a total effect were found. "The present research shows that violent-sexist video games such as GTA reduce empathy for female violence victims, at least in the short term." The press release also runs the headline "Sexist video games decrease empathy for female violence victims", again suggesting that a total effect was found, despite the null result.

Furthermore, I am concerned as to the possibility of confounding in this study's causal arguments. Is it possible that those who are already low in empathy are more likely to identify with Grand Theft Auto's protagonist? And could a similar process not be responsible for differences in masculine beliefs? It is not clear to me that this can be ruled out. If it can be, I would be glad to learn how.

In summary, I am glad for the ability to access the authors' data and reproduce their analyses. All analyses I attempted were successfully reproduced as in the article. (NB: I did not attempt estimation of the indirect effect via bootstrapping.) However, I think that more should be done to soften the research conclusions given the null total effect. The way the article is currently structured, discussed, and promoted sometimes suggests that there is a total effect of game, or at least a total effect of the game × gender × identification interaction, on the empathy outcome. The authors understand process analysis better than I do -- perhaps they could help the reader to interpret the differences between the total and indirect effects.

The R code for my re-analyses are available at https://github.com/Joe-Hi....

Thank you for your time and attention,
Joe Hilgard

No competing interests declared.

RE: Distinctions between total and indirect effects

agabbiadini replied to JHilgard on 16 Apr 2016 at 19:42 GMT


QUESTION 1
I have not attempted to reproduce the estimates of the indirect effects. I have to admit that I do not fully understand the conditional indirect effects as presented in Results, Primary Results, paragraph 6. Is it indeed possible to model the indirect effects of game while conditioning on game? Would it not be better to estimate the bootstrapped indirect effect of the games by conditioning on something else? My inexperience with process modeling is showing, but I hope you understand my confusion.

ANSWER:
WE do not condition on the game. In a conditional process model, the conditioning accourred when you have some moderators. You can find more in chapter 1, 4 and 6 of Hayes, 2013 - Introduction to Mediation, Moderation, and Conditional Process Analysis.
When using GLM for estimating indirect and direct effects is it possible to have different results than PROCESS. This is clearly explained by Hayes himself. See this FAQ from Hayes website:
"When I estimate a model using PROCESS and compare to what I get just using SPSS or SAS's regression procedure, I get different results.  There must be something wrong with PROCESS.
There is nothing wrong with PROCESS.  When the same model is estimated using the same data with the same output options, the results will be the same as what you get with SPSS or SAS's regression procedures.  There are many sources of discrepancies you may notice when discrepancies exist, and they are all generated by the user, not by PROCESS.  The simplest sources involve requesting options in PROCESS that SPSS or SAS won't do on its own.  A common one is requesting HC3 standard errors in PROCESS, which are different than standard OLS standard errors.  SPSS and SAS won't generate these standard errors, but PROCESS will (but see my HCREG macro) but only if you ask for them.  When you do, standard errors, t-values, p-values, and confidence intervals are different, as they should be. 
Most other sources of discrepancies are due to the user not acknowledging the existence of missing data.   For example, if you mean center or standardize "univariately" (i.e., one variable at a time) prior to conducting an analysis, you will end up with variables in the analysis that are no longer mean centered or standardized after missing data are kicked out by PROCESS or SPSS or SAS's regression routine.  I don't recommend doing centering or standardization computations manually.  If you do, do them to a high degree of precision (three or four decimal places generally is not sufficient) and only after purging the data of cases missing on variables that will end up in the analysis.  In a PROCESS model that includes moderation, PROCESS will center for you if you ask it to, and it will do it correctly (see the documentation.  Also read my debunking of the mean centering myth in Chapter 9 of Introduction to Mediation, Moderation, and Conditional Process Analysis). 
In a mediation analysis, another common mistake I see users make is estimating the effect of X on M and the effect of M on Y controlling for X in separate regressions without acknowledging the existence of missing data.  Suppose, for example, some cases are missing on Y.  In such a situation, your estimation of the effect of X on M will be based on more data than what PROCESS uses, because PROCESS would discard cases missing on Y before it estimates the effect of X on M.  Although we can debate the merits and faults of listwise deletion, it is generally not good practice to piece together a mediation analysis using different subsets of the data for the estimation of different parts of the model. 
Before asking for advice or bringing a "bug" in PROCESS to my attention, please check the residual degrees of freedom for the model in output produced by PROCESS (this shows up as "df2" in the PROCESS model summary section of the output) and compare it to the residual degrees of freedom from SPSS or SAS's regression routine output.  If there is a difference between these, you have missing data you are not properly acknowledging somewhere.  If there is no difference, then the source of the discrepancy is something else you have done differently compared to what PROCESS is doing. "


QUESTION:
The specificity of this double-moderated indirect effect, given that there is no significant total effect, suggests that care might be taken in the discussion. At times the authors are appropriately careful, saying that "This research] address[es ...] who is most likely to be harmed by violent-sexist video games." But at other times, the authors seem to speak too broadly, as though a total effect were found. "The present research shows that violent-sexist video games such as GTA reduce empathy for female violence victims, at least in the short term." The press release also runs the headline "Sexist video games decrease empathy for female violence victims", again suggesting that a total effect was found, despite the null result.

ANSWER:
This is not true. In our article we clearly pointed out why this effect occur and on whom, that is they are significant only for highly identified male players.

QUESTION:
Furthermore, I am concerned as to the possibility of confounding in this study's causal arguments. Is it possible that those who are already low in empathy are more likely to identify with Grand Theft Auto's protagonist? And could a similar process not be responsible for differences in masculine beliefs? It is not clear to me that this can be ruled out. If it can be, I would be glad to learn how.

ANSWER:
In order to disentangle this possibility, we considered three different condition (neutral, violent, sexist games). What you pointed out is interesting, but it seems to me that this not our hypothesis…. I think future study should investigate the moderating effects of empathy-trait on avatar identification…

QUESTION:
In summary, I am glad for the ability to access the authors' data and reproduce their analyses. All analyses I attempted were successfully reproduced as in the article. (NB: I did not attempt estimation of the indirect effect via bootstrapping.) However, I think that more should be done to soften the research conclusions given the null total effect. The way the article is currently structured, discussed, and promoted sometimes suggests that there is a total effect of game, or at least a total effect of the game × gender × identification interaction, on the empathy outcome. The authors understand process analysis better than I do -- perhaps they could help the reader to interpret the differences between the total and indirect effects.

ANSWER:
As I previously said, In our article we clearly pointed out why this effect occur and on whom, that is they are significant only for highly identified male players.

No competing interests declared.

RE: RE: Distinctions between total and indirect effects

JHilgard replied to agabbiadini on 20 Apr 2016 at 16:15 GMT

I owe Drs. Gabbiadini et al. an apology. I did not read the article closely enough and so I implemented the wrong structural equation model. They used PROCESS Model 11, in which the moderators influence only the indirect path. By contrast, I made a model like PROCESS Model 12, in which the moderators influence both the direct and indirect paths.

My concern was that there was a significant indirect effect for males but not a significant total effect for males. In that light, I felt the press release was misleading. However, it appears that the total effect for males is sensitive to the difference between PROCESS Model 11 and PROCESS Model 12, and indeed, the difference between PROCESS Model 11 and GLM.

When I run the authors' model, as I implement it in R's lavaan package, I do indeed find both a significant indirect effect for males as well as a significant total effect for males.

This pattern holds regardless of whether I include age and violence-ratings as covariates on the mediator and outcome. (I cannot include game-frequency as a covariate because it is missing from the available dataset.)

I am not sure it is wise to include violence-ratings as a covariate given the {1, 0, -1} contrast codes for game, but the point seems moot -- there is a significant effect whether one controls for "violent" or not.

I also explored this by coding the game for misogyny {1, 0} for GTA, non-GTA. I felt this was a more appropriate coding than the {1, 0, -1} contrast codes. However, in these models, lavaan indicates significant indirect and total effects for males, whether or not age and violence are included as covariates. (Again, I can't include frequency as a covariate, as it's missing from the dataset.)

In summary, I apologize to the authors for the misunderstanding and thank them for responding to my questions. I feel that there can still be productive debate about finer modeling decisions and the differences between PROCESS and GLM, but I also feel that I have used enough of everyone's time. I was wrong -- there does indeed seem to be a total effect for males in the context of the model reported, as well as in some similar appropriate models.

-Joe Hilgard

No competing interests declared.

RE: RE: RE: Distinctions between total and indirect effects

agabbiadini replied to JHilgard on 20 Apr 2016 at 22:20 GMT

Thank you Joe!

No need to excuse. Science is made by critics, observation and hard work.
As researcher we need to be skeptical of phenomena and of course, results!

Best
Alessandro

No competing interests declared.