Fig 1.
Number of participants needed for 80% power to detect posttest improvement, as a function of number of test questions and size of posttest improvement.
The data reflect the results of simulated experiments in which participants completed a pretest and a posttest. We varied the number of questions appearing on each test (x-axis) and the mean improvement in accuracy from pretest to posttest (legend). For each combination of these variables, 10,000 simulated experiments were run for each of a range of sample sizes. The y-axis shows the smallest number of participants for which improvement from pretest to posttest was statistically significant (as determined by a paired t-test) in at least 80% of simulations. The results show the trade-off between the number of questions each individual participant responds to and the number of participants required to achieve adequate power.
Table 1.
Demographic information for the student population of Introductory Psychology at Indiana University Bloomington.
Table 2.
Items used during pre- and posttest.
Fig 2.
Schematic representation of the procedure for this study.
Students were instructed on measures of central tendency by their instructor in the classroom during regular classroom activities. The online homework was then assigned. This homework started with 4 pretest questions about measures of central tendency without feedback and then proceeded with the tutorial. Four posttest question on measures of central tendency were included in each class’s regular mid-term exam.
Fig 3.
Tutorial interface for one of the trials during study.
This example shows a problem and response feedback. The buttons at the bottom include all possible choices for the next problem. Left panel: Interface for the Self-Regulated group. Right panel: Interface for the Yoked group.
Table 3.
Summary statistics for study behavior for the Self-Regulated and Yoked groups and t-tests statistics comparing differences between the two groups.
Table 4.
Results of Regression Analysis of Posttest Accuracy (N = 2061).
Fig 4.
Average Posttest score by concept repetition rate for the Self-Regulated (left panel) and Yoked (right panel) groups.
Students were divided into bins by their adjusted rate of repetition and average posttest scores (Logit transformed) within each bin were plotted. Concept repetition rate was adjusted by subtracting the average rate of repetition for the entire group from the rate of repetition for the bin—a value of 0 in the x-axis indicates mean concept repetition rate (represented by the vertical dashed line) and increasing values indicate increasing difference from average. The values in the y-axis represent Logit transformed posttest scores. Each point in the graph lies at the center of a 20%-wide interval of concept repetition rates, and represents the average posttest score among students whose concept repetition rates fell in that interval. The number of students in each bin is represented by the area of the circles surrounding the data points. The regression lines represent best fitting lines of the regression analyses assuming average values for all predictors other than concept repetition rate.