Methods

Design.

For each simulated experiment, we first generated a sample of n student scores s from a normal distribution s ∼ N(μ, σ²), truncated between 0 and 100. These scores represented the final grade of each student in the course on a 100 point scale, where μ was the sample mean and σ was the sample standard deviation. Second, students were randomly determined to have access to the textbook at a rate of a, and not have access at a rate of 1 − a. Third, students were randomly assigned to either an OER or Non-OER condition with the constraint that both conditions must have an equal size. Fourth, in order to simulate the effect of access, we decreased the score of the students in the Non-OER condition who were previously determined to not have access to the textbook. The scores of students in the OER condition were unaffected, representing the fact that all of these students now have access to the book. The magnitude of the score decrease was equivalent to dσ. The parameter d represents the effect size [26] of having access to a textbook. Finally, we fit a regression model that predicted student score by condition, and tested the condition coefficient against 0 by using a standard t-test. An overview of the simulation is shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the simulated experiments.

For each experiment, a sample of student scores were generated, and students were determined to have access or not to the textbook. Students were then randomly assigned to either an OER or Non-OER condition. The effect of access was simulated by reducing scores for students determined to not have access, but only in the Non-OER condition. Lastly, a statistical test the OER and Non-OER conditions was performed. See text for more information.

https://doi.org/10.1371/journal.pone.0212508.g001

Results

We examined the proportion of simulated experiments that rejected the null hypothesis at the standard α of.05 (i.e., power). The results are shown on Fig 2. As is the case of all experiments where samples are drawn from normal distributions, the probability of success increases with n [25]. However, we also see that access rate (a) plays a strong influence on the ability to detect the effect of OER. When access is very low, experiments have a much higher likelihood of correctly rejecting the null hypothesis with smaller n. This makes sense, because there are more student in the sample that can be impacted by the intervention. However, as a increases, it pulls the probability of success down. Indeed, when a is large, it requires very large numbers of students to detect a significant effect. To illustrate the strength of this relationship, an OER experiment with 10,000 students will have a 89.3% chance of success when the access rate is 70%. However, the same 10,000 student experiment conducted on a sample with an access of 80% will only have a 56.5% success rate. The situation gets considerably worse when the access rate is 90%. This experiment would only have a 19% success rate. The fact that an experiment with 10,000 students would have such a low chance of correctly rejecting the null hypothesis demonstrates the influential role of access rate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Probability of successfully detecting an effect of OER as a function of access rate and sample size.

A study sample’s access rate to textbooks prior to adopting OER can severely hinder the likelihood of detecting an effect of OER, even at large sample sizes.

https://doi.org/10.1371/journal.pone.0212508.g002

Methods

Selection of research.

From the literature, we were able to find 42 direct comparisons of OER to traditional materials, across 9 publications [9, 12, 14–18, 27, 28]. We did not include a study or comparison if tests of statistical significance were not reported. Further, we only included comparisons that used a continuous performance metric as their dependent variable (i.e., grades on a 0-4 scale or test scores). Comparisons that used non-performance based dependent variables (e.g., drop or withdrawal) were not included, as they are not suitable for use as measures of learning. Some studies (e.g., [15]) examined both grades and pass rates separately, which is a dichotomous version of grade (i.e., C- or better.). As an aside, it is not clear to us why both measures are sometimes used, as the measures are likely highly correlated. In cases where both measures were used, we only included comparisons on course grade. We did not examine studies that only examined pass rates, because these studies use non-parametric statistics which are not applicable to the power analysis we conducted. Also, several studies conducted both an analysis which collapsed across different courses or semesters, and then conducted separate analyses for each of these levels [16, 27]. In these cases, we only included each separate analysis, but not the overall analyses. In the case of [18], they collapsed across multiple courses without conducting separate analysis for those courses. In this case, we only included the overall analysis. The complete list of comparisons is shown on Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Estimated power of prior efficacy studies across different access rates (a).

https://doi.org/10.1371/journal.pone.0212508.t001

Results

The estimated power for each comparison, for access rates of 40%, 60%, and 80%, are shown on the far right columns of Table 1. We can see that for most comparisons, even under the most optimistic of scenarios (i.e., 40% access), the expected likelihood that the comparison would yield a positive significant effect of OER is very small. Only the comparisons which had very large sample sizes had substantial power at the 40% access level [16, 18], though even some of these comparisons had low power at 80% access rates. Note that for many studies, power is so low at the 80% access level that the probability of correctly rejecting the null hypothesis is just as likely as falsely rejecting a true null hypothesis (α = .05)! Thus, if there was an 80% access rate, these experiments were just as likely to detect a real effect of OER as they were to detect a false effect of OER. Interestingly, one comparison [15] had low power even with a sample size over 4000 students. This was due to the extreme imbalance of students in the Non-OER and OER conditions.

Given the results of this power analysis, we can determine the expected number of comparisons that should have correctly rejected the null hypothesis by summing the power values for each level of access. Across the 42 reported comparisons, we would only expect to observe significant effects of OER 18, 11.5, or 5.2 times, for the 40%, 60%, and 80% access rates, respectively. Note that only 9 of the 42 comparisons on Table 1 found positive effects of OER on learning outcomes. Even though this number seems very low, the results of the simulation power analysis demonstrate that this is well aligned with what should be expected, even if there is a real effect of OER. Of course, our power estimates assume perfect conditions. These real world studies have many confounding factors to contend with, so it is likely that the real power of these studies was even lower than what we estimated. In this case, it is possible that the number of significant effects found so far could even be higher than what would be expected.

Implications for OER research

These results have several implications for future research in OER. First, we recommend that researchers attempt to measure textbook access rates in their student population prior to implementation of OER. If access rates are very high, it is important to consider that the likelihood of detecting an effect on learning outcomes should be very low. The effect of access rates should be considered when interpreting null results.

Second, it is critical that future research works towards determining the true effect size of textbook access on learning. Determining the true effect size will afford far more reliable power calculations, and more importantly, enable more meaningful interpretation of research studies. For instance, a high powered study that produces a null result is more meaningful than a low powered study that produces a null result, because the null result is unexpected in the case of a high powered study. Unfortunately, accurate measures of power require a reliable measure of effect size, and the vast majority of studies on OER efficacy do not report enough statistics in their analyses for computation of effect size estimates. It is critical that researchers report all relevant test statistics, p-values, sample sizes, means, and measures of dispersion. We encourage reviewers and editors of future research insist that authors report these measures.

Finally, it is common for OER researchers to conduct comparisons without making an explicit hypothesis or prediction. Hypotheses and predictions are critical, because they help guide research designs and interpretation of results. In the case of the access hypothesis, having an explicit mechanism makes it clear that the intervention should only affect some students. We cannot help but wonder if so many low power null effects would have been published had the access hypothesis been formally proposed earlier.

Potential theoretical mechanisms for OER efficacy

In this paper, we have discussed access as being the primary mechanism for why OER might improve learning. It is certainly possible that adoption of OER could affect learning outcomes in other ways. One idea is that the open nature of OER affords the ability to teach in ways that are not possible given the constraints of closed source materials [7]. Another idea is that OER may provide better or worse quality than the commercial counterpart (e.g., [17]). However, as mentioned previously, these ideas are rarely expressed as a formal hypothesis, and the mechanisms are rarely tested as part of the research. One exception is the work of [17], which compared learning outcomes from an OER and commercial textbook, but also examined the perceived quality and readability of the books. While differences in perceived quality and readability were observed, these differences did not translate into strong benefits to learning [17]. It should be noted that other mechanisms would not be subject to the same power constraints as access, as these mechanisms would presumably affect all students in the study. Thus, detecting quality difference effects, for example, should require far fewer students than access effects.

With regards to the access hypothesis, we made the assumption throughout this paper that students who have access to the textbook would use the textbook in effective ways. Of course, access is not a guarantee for learning. A student with access to a textbook could easily choose to ignore it or engage in ineffective learning strategies. These students are no better off with a textbook than they were without one. This fact creates a general boundary condition on the ability for access alone to affect learning. Practically speaking, the effect of access on learning depends critically on usage after access to the materials is supplied. If students engage with the book in ineffective ways, then access will be an irrelevant factor. To this end, we simply caution readers that while access is an important step towards improving learning, it not sufficient.

Limitations

It is important to point out that our simulated experiments provide only a proxy measure of statistical power. In particular, these simulations estimate power under an unrealistically optimistic experimental scenario. The situation only gets more difficult in real world studies, which have instructor effects, student effects, and other confounds to control for. These variables only add noise to the data, and reduce this probability of success even further. Thus, a researcher hoping to estimate their statistical power with a real-world data should understand that their actual power will be lower than those shown in Fig 2.

Another limitation of our analysis of past research studies is that we assumed an effect size d of 0.25, rather than computing the observed effect sizes post hoc. Unfortunately, as previously mentioned, the vast majority of research we reviewed did not report sufficient statistics to conduct such analysis. If the true effect size of OER adoption is larger, then these studies may have had considerably more statistical power than what we estimated. To this end, we conducted a supplemental analysis which estimates the minimum effect size required in order for an OER study of varying sample sizes and access rates to achieve an acceptable level of power. This analysis is explained in detail in S1 Appendix, and the results shown on S1 Fig. Should additional research become available that suggests the effect size is different than the one we used, S1 Fig can be used to determine whether power of these past studies was adequate. Also, we remind readers that the source code of our analysis is available such that anyone rerun our analysis with varying levels of d.

Relevance to educational research

While it is tempting to think that the research failings discussed in this paper are unique to OER, the reality is that these failings are the result of a common mistake in educational research (and even social science research more broadly). Specifically, that mistake is overgeneralizing the influence of an experimental variable without critically considering the context in which that variable is manipulated. The importance of contextual factors was articulated decades ago by [33, 34], who noted the fragile nature of many of the most landmark findings in memory research (e.g., levels of processing [35]). In particular, it was observed that minor changes to an experimental design could completely change the outcome of a manipulation. The critical insight of [33] was that variables not manipulated by the experimenter are just as important as the ones that were manipulated. The materials used, the final assessment, the types of students, and the interactions among all these factors were all critically important. Thus, if one wants to understand whether an intervention affects learning, they need to be aware of the context in which that intervention is taking place.

Of course, researchers and practitioners are naturally compelled to focus only on variables of interest in isolation. To illustrate, one of the most influential studies in education is a meta-analysis of over 800 factors that affect student learning [36]. While compiling such an extensive list of factors is quite the achievement, in our view, it presents an unrealistic view about the nature of learning. It leads one to a misplaced belief that certain techniques are better than others. However, even the strongest factors listed by [36] could quite easily be rendered ineffective by applying them to certain topics, certain populations of students, or certain outcome measures. For example, it is well known that effectiveness of an educational strategy or intervention can depend on the prior aptitude of individual’s in a study [37]. The very existence of such interactions prevents us as a field from ever discovering “laws” of human learning [38] or making broad sweeping claims about any intervention. In sum, the effectiveness of any educational intervention will almost always depend on the context in which it is implemented.

Failing to consider the importance of context can lead to poor study design and misleading conclusions. In this paper, we discussed the importance of student access in moderating the effectiveness of OER. Past researchers assumed OER would have a general effect on learning and failed to context influences, which lead to a dearth of under powered and ill designed studies. A similar analogue comes from the oft maligned enterprise of media comparison studies [39–42]. Media comparison studies typically evaluate student learning from a standard instructional strategy delivered on different types of “media” (e.g., computer vs. paper). Like OER, most of these studies have produced null results, and have been vehemently criticized for decades as being without merit [39]. Indeed, [39] took a strong stance that media is only a vehicle for delivering instructional strategies, and that media itself will never influence learning. While this is often true, others [41, 42] have argued that many media comparison studies employed standardized research designs that were not well suited to measure the unique mechanisms afforded by media evaluated in the study. [43] reviews a wide variety of media studies which reveal the nuances of when media can have a meaningful influence on student outcomes. Thus, by carefully considering the context in which an intervention occurs and is evaluated and devising appropriate hypothesis to test, one can design studies that effectively and appropriately measure the unique merits of the intervention.