Writing week-journals.

Students’ motivation and engagement in writing are likely to grow in learning environments providing many opportunities and encouragements for students to express themselves through writing [11–14]. Journal writing is a practice that can be easily implemented in classrooms without much effort, time, or resources (e.g., [5,15]). Journals are a type of free writing that is informal and personal [16–17] and have gained popularity among the activities aiming at promoting writing [17] and students’ confidence in writing [18]. The nature of this educational tool allows students to write freely without strict directions, restrictions or assessment purposes [16]. While writing journals, students will choose their writing topic [19], engage deeply in their writing activities [17] and improve their writing skills and creativity [16]. Furthermore, writing journals allow students to enhance their reflection skills, critical thinking, self-expression, self-regulated skills, and knowledge [17].

Notwithstanding the potential positive influence of writing journals on students’ motivation and writing performance (e.g., [16,18]), findings from the extant research are not consistent. Prior research (e.g., [4,16,20–21]) found no statistical evidence on the effectiveness of free writing on students’ writing quality. But, a recent study with fourth graders concluded that students who wrote weekly journals for twelve weeks showed a higher improvement on the quality of their compositions, than that achieved by students in the comparison group [15]. Despite these encouraging findings, students in the experimental group reached a plateau after the first three weeks of writing journals, which might indicate that this type of intervention may not be sufficient to foster progress on writing quality.

Writing and self-regulation.

Considerable progress has been made in the last thirty-five years to understand the role of self-regulation in writing. Not surprisingly, research found that skilled writers master self-regulated learning competencies (e.g., self-set goals, self-reinforcement) [22], and also that many students struggle with writing [23]. This may happen because effective writing requires: (i) high levels of self-regulation and attentional control to manage the writing environment; (ii) knowledge of the writing topic, genre, processes and skills involved in writing [22]; (iii) strategies for planning, text production [24–25] and monitor the writing activity [26] to meet specific self-set goals [27].

Three decades ago, Karen Harris and Steve Graham built the Self-Regulation Strategy Development model (i.e., SRSD model; [28])—an instructional program designed to enhance writing and self-regulation strategies. SRSD was designed to attain the three major goals, as follows [29]: (i) to help students develop the knowledge and skills needed to manage the writing strategies involved in the writing processes (i.e., planning, writing, revising and editing); (ii) to support students using the strategies and self-regulatory skills (e.g., goal-setting, self-instruction, self-assessment, self-reinforcement) while monitoring and managing their own writing (e.g., [30–32]); and finally (iii) to help students develop positive attitudes and beliefs about themselves as writers [31,33–34]. In fact, when students perceive themselves as self-efficacious in writing, they are likely to exhibit good writing quality and invest effort while carrying out a writing task [34–36].

The meta-analysis by Graham et al. [5] analyzed the impact of the SRSD model on students’ writing and found that adding self-regulation instruction (e.g., goal setting and self-assessment) to strategy instruction can improve the overall writing quality of typical developing writers and, in most cases, of struggling writers. The benefits of participating in SRSD programs are well established in literature (e.g., [23]), but further research is needed to explore complementary forms infused in regular curriculum that may increase the teaching of writing strategies [3,5,15]. Recently Rosário and colleagues (e.g., [15,37–41]) discussed the use of story-tools in class as a successful strategy to foster students’ motivation, and promote self-regulated learning (SRL). Based on the extant evidence which supports the role of stories to promote SRL, current authors believe that infusing story-tools in the regular curriculum combined with writing instruction (i.e., SRSD) may be beneficial for increasing the levels of writing quality.

Story-tools to promote SRL.

Stories, traditional tales and fables are well-known ways of delivering knowledge [40], to promote children’s development [42–45], imagination [46], and self-reflection about their own behaviors [40]. Bearing this in mind, researchers in Iberian Peninsula, created SRL story-tools programs that focus on promoting SRL through different types of narratives. The Yellow trials and tribulations [45] is a story–tool developed to promote SRL at elementary school, and was used in the present study. This narrative tells the story of the disappearance of the color Yellow from the Rainbow and describes the adventures experienced by Yellows’ friends, the other colors of the rainbow, whilst searching for Yellow. Along this quest in search for Yellow, who should not be left alone, the other colors of the rainbow met new friends and learned various useful SRL strategies to overcome the obstacles found along the way.

This story-tool was designed to promote students’ SRL strategies (e.g., goal-setting, self-reflection, strategic planning, and organizational strategies), to increase motivation and academic achievement [47]. This tool is grounded on the social cognitive framework [48], and assumes that contextual variables and learning settings play important roles in students’ motivation and self-regulation [47]. The stories in each chapter of the story-tool address the PLEE cyclical model: Planning, Execution and Evaluation (see [40] for a more detailed explanation), which is rooted in the SRL model by Zimmerman [49–50]. Students are expected to regulate their school behaviors in three cyclical phases: forethought (i.e., processes prior to learning), performance control (i.e., processes while learning), and self-reflection (i.e., processes after learning). The former model presents a recursive structure, through two paths of logic. The process is derived from Planning through Execution to Evaluation, but the same cyclical nature is also reset in each phase, thus reinforcing the self-regulation logic of the process. These two structuring loops, throughout and within the phases, reinforce the SRL synergy strengthening the process [38,40,51].

Modeling and teaching the learning strategies (e.g., goal-setting, strategic planning, organizational strategies), embedded in the story-tool underlies on three types of knowledge [38]: (i) the declarative knowledge—learning the meaning of a learning strategy (e.g. know what taking notes is); (ii) the procedural knowledge, that is related to learn how to implement these learning strategies (e.g., know how to take notes in class); and, finally, (iii) the conditional knowledge that demands students to know when it is more appropriate to use a specific learning strategy in a particular learning context (e.g., when it is more useful to take notes) [52]. For example, in chapter 6 of the story-tool [45], the Ant General, one of the characters, explained the planning phase to his troops (i.e., declarative knowledge): “in order to plan, we have to decide what we need to know and what we need to do for everything to run smoothly. Afterwards, to avoid any problems, we allocate time for each task” (p. 27).

Each chapter provides students with the opportunity to acquire, practice and reflect on the use of the SRL strategies embedded in each phase of the PLEE model. This tool allows the analysis of the characters’ behavior which are similar to those of children in real life situations (e.g., the Bird-Teacher told the little birds a story about a lazy deer who did not listened to the teacher advice’s friends and hurt himself while competing with a grasshopper), hence helping students to reflect on what they may learn with the characters’ behaviors. This experiential closeness fosters children’s engagement in learning [40]. For example, it is expected for students to transfer the content learned throughout the story to the process of writing compositions.

Design.

The present study was conducted with fourth grade students, the final grade level in Portuguese elementary school. The Portuguese Ministry of Education approved the study by giving their written consent (n. 036000004). This study was reviewed and approved by the ethics committee of the Universidade do Minho. The study followed a longitudinal cluster-randomized controlled design for twelve weeks, in 18 public schools in the north of Portugal. The participating teachers and their fourth-grade students were randomly assigned to the four conditions, with five classes participating in each condition (i.e., Groups A, B, C and D; see Fig 1). This methodology is useful to access the comparative effectiveness of experimental conditions that vary in their practices. Moreover, this tool helps avoid “contamination” between those participants receiving the intervention and those who are not, preventing that the treatment effect would be compromised [54]. During the twelve weeks of the study, students on the comparison condition (Group A) did not participate in any type of program focused on writing instruction. Teachers were instructed to follow the regular Portuguese writing curriculum to meet fourth grade level teaching requirements. According to the Directorate-General for Education and the Minister of Education and Science [55] this included teaching students about grammar, vocabulary, spelling, sentence construction, punctuation, handwriting, organization and revision of different types of text (i.e., narrative, informative, descriptive, letters, invitations, and texts using direct speech). In group B students wrote a journal on a weekly basis for 12 weeks. Students in group C and D were given writing instructions following the SRSD model; in group D the story-tool “Yellow Trials and Tribulations” [45] were added to the treatment received by the group C (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic of each treatment condition and procedures.

https://doi.org/10.1371/journal.pone.0218099.g001

Participating students and their teachers.

The participants were 370 (183 girls) fourth graders nested in 20 classes from 18 public elementary schools in the north of Portugal. All the participants had Portuguese as their home language, aged between 9 and 10 (M = 9.45, SD = .51). The fourth-grade classes were randomly assigned to four groups: A (N = 92); B (N = 90); C (N = 98); and D (N = 90). Students with special education needs (i.e., specific learning disorder and learning disabilities) were excluded from the data analyses.

All the 20 teachers, 17 were female, aged between 34–56 years (M = 42.4, SD = 6.59) had an undergraduate degree and experience in teaching ranging between 12 and 34 years (M = 21.5, SD = 6.16). Class sizes ranged between 10 and 23 (M = 20.38, SD = 4.75). None of the teachers enrolled in the study reported having received specific writing instruction in their professional development.

After receiving the consent from the Portuguese Ministry of Education, an email explaining the overall study objectives was sent to 26 public schools located in northern part of Portugal. Eighteen schools (a response rate of 69.2%) and 20 teachers agreed to participate in our research. In these schools, the families were lower-middle classes, as noted by the high percentage of students (40%) receiving free or reduced-price lunches. These demographics were collected from the offices of the participating schools. A letter informing about the study was sent out to ask permission for the children participate the study. Participants’ confidentiality was assured (e.g., eliminating the names and researchers’ personal notes that could link the participants to their teachers or schools). All students returned the signed parental consent forms. Finally, the 20 teachers (classes) who agreed to participate were randomly assigned to the four treatment conditions (i.e., comparison group and three experimental groups). Teachers were blind to the purpose of the study and all agreed to follow the fourth grade Portuguese curriculum (e.g., variety of text genres, grammar and punctuation) throughout the study.

Comparison group (group A) and Week-Journals (group B).

During the twelve weeks of the study, students on the comparison condition and weekly-journals did not participate in any type of writing instruction, besides the writing of the weekly compositions proposed for this research. Teachers were instructed just to follow the regular writing curriculum [55] to meet fourth grade level expectations.

Additionally, for twelve weeks, students in the week-journals condition (i.e., group B) wrote a journal in 25 minutes each Friday morning under the supervision of a research assistant. While students were writing their journals they did not receive any instructions, nor feedback afterwards. Prior to the beginning of the study, participants’ confidentiality was assured, by telling students that the journals would only be used for research purposes (i.e., teachers did not read the journals). Each student received a notebook “journal” to write their weekly entries (i.e., approximately ten lines) about their week’s events at school or at home. Journals were kept in the classroom in a closed box and were the responsibility of a research assistant.

General instructional procedures (intervention conditions C and D).

SRSD writing instruction, as well as the topics for condition D, were delivered along eleven sessions on a weekly basis, by one of the authors, during regular Portuguese language lessons. The length of the sessions for students in group C and D was 45 minutes. Both intervention conditions are briefly described in S1 Appendix. An extended description of the lessons and materials suggested for instruction is provided elsewhere [53].

SRSD instruction (intervention condition–group C).

The writing instruction followed the six stages of the SRSD model [25,28] as follows: (i) development of background knowledge; (ii) discussion and description of the strategies to be learned; (iii) modeling the use of those strategies; (iv) memorization of those strategies; (v) supporting of the strategies; and, finally, (vi) independent performance. In the present study, instruction started at the first stage and continued into the following stages (see S1 Appendix). Despite acknowledging the sequence of the content, we followed Harris and Graham [28] and asked students to memorize the mnemonics taught (strategy from stage four) since session 1. Thus, this stage was recalled at the beginning of every session to analyze if students had memorized the mnemonics [60]. A number of self-regulation procedures were also taught to students, including self-monitoring while planning their stories, self-reinforcement and self-assessment [60]. The materials for teaching writing narratives using the SRSD model were translated to Portuguese and used by fourth graders and teachers in class.

Writing strategies.

In the first sessions, students learned a general strategy to apply while writing their compositions. This strategy included three steps, represented by the mnemonic POW: Pick my ideas (i.e., decide what to write about), Organize my notes (i.e., organize writing ideas into a writing plan), Write and say more (i.e., continue to modify, upgrading the plan while writing). For example, on the second step of POW (i.e., organize my notes) students were taught a genre-specific strategy for writing notes for each part of the story: the mnemonic S-A-C [principal steps of a story: Setting (S), action (A) and conclusion (C)] (see [53]). To help students become familiar with the S-A-C mnemonic, students were taught to ask themselves the following six questions, aligned with the three S-A-C steps: Where does the story take place? When does the story take place? Who are the main characters (describe them)? What do the main characters do or want to do (sort them in the right way)? How does the story end? How do the main characters and the others feel? For writing notes, students were presented with a graphic organizer (see [53]).

Strategy instruction.

The strategy instruction followed the SRSD model [28], however the time spent on each stage was adjusted to the design of the current study. As shown in S1 Appendix, lesson one and two aimed to develop students’ prior knowledge on composition and to discuss and explore the characteristics of a good story. General writing strategies (i.e., POW) were presented and discussed with students. Students’ negative beliefs about writing performance were also discussed, and students were encouraged to transform negative thoughts into positive beliefs (e.g., "I can do it, if I use the right strategy”). In lesson three and four, students revisited the general writing strategies (i.e., POW) and discussed the SRL strategies (i.e., self-instructions, goal setting, self-assessment and self-reinforcement) they will use during and after writing a story. In lesson five, six and seven the planning, writing and assessing of compositions using general (i.e., POW) and SRL strategies (i.e., self-instructions, goal setting, self-assessment and self-reinforcement) were modeled collaboratively in class. Modeling the use of strategies helped students to learn to apply these strategies and to develop competencies, attitudes and beliefs, while writing independently. Lesson eight, nine and ten focused on strengthening students’ abilities for independent planning, writing and assessing of stories by using general (i.e., POW) and SRL strategies (i.e., self-instructions, goal setting, self-assessment and self-reinforcement). The work on these lessons aimed to wean students off the graphic organizer [60]. Finally, in lesson eleven students wrote, without support, a composition, using the strategies learned. Still, as suggested by authors [61], if any story elements were not included, the previous stages were recalled.

SRSD instruction plus the story-tool (intervention condition–group D).

In the current study, the Yellow Trials and Tribulations story-tool [45] was used to help students learn a set of learning strategies and apply them into the story-tool learning context while reflecting upon their own writing activities (i.e., on how and when to implement the general and SRL strategies). Sessions for the group D were preceded by the reading out loud of one or two chapters of the book in class. During the reading, small breaks were made and students were invited to discuss and analyze what was happening in the story plot (see [40,53]). During the session students did the same writing tasks as students in group C. The Appendix aligns the stages from SRSD (i.e., group C) with the chapters of the story-tool.

Self-regulated learning strategies inventory (SR_W).

The SRL Strategies Inventory [38] assesses nine SRL strategies concerning the three phases of the SRL process (i.e., planning, execution and evaluation). In the preset study, this scale was adapted with the aim of assessing the SRL strategies used while writing: Planning (i.e., ‘‘I make a plan before I begin writing. I think about what I want to say and how I need to write it”), Execution (i.e., “While I write my composition I follow my plan”, and Evaluation (i.e., ‘‘I compare the grades I received with the goals I set for that subject.”). The 9-items were scored on a 5-point Likert scale, ranging from 1 (never) to 5 (always). Cronbach’s alpha in this study was .80. Data from the confirmatory factorial analysis run support the construct validity of this measure. The model fits well data [χ² (25) = 53.639; p < .01; AGFI = .907; TLI = .900; CFI = .927; SRMR = .058; RMSEA = .076 (.048-.104)]. The factor weights of the nine items ranged from .507 to .703 (all statistically significant at p < .001). After fit the model, none of the modification indexes was greater than 5.00.

Attitude towards writing (AT_W).

Each of the nine items from the writing attitude survey [34] asked students to indicate how they felt when they engaged in writing activities at school or at home (e.g., How do you feel when you think you have to write instead of being able to play?). Students were asked to mark one of the four images of Garfield the Cat on a 4-point Likert scale (1 = very unhappy; 4 = very happy). This scale was, in the present study, translated and adapted to the Portuguese population. Cronbach’s alpha in this study was .86. The construct validity analysis yielded data supporting a unifactorial model [χ²(25) = 34.086; p > .05; AGFI = .933; TLI = .976; CFI = .983; SRMR = .034; RMSEA = .043 (.000-.076]. The factorial weights of the nine items ranged from .660 to .750 (all statistically significant at p < .001). After fit the model, none of the modification indices was greater than 6.00. All data suggest construct validity.

Compositions.

In order to assess the writing quality of students’ compositions, a holistic rating scale was used based on the criteria defined in the Educational Progress Test (i.e., a standardized exam) in Portuguese language for fourth graders [62]. The rating scale assesses topics such as (i) title; (ii) organization (introduction, main body paragraph, ending), (iii) grammatical correctness of sentences (e.g., active verbs, use of direct speech, descriptive adjectives, punctuation, morphology) (iv) coherence; (v) originality; (vi) sentence structure, (vii) word choice; (viii) spelling errors. Prior to scoring, all narratives were typed into a word document and the number of words were counted. Students’ personal information was removed and punctuation, spelling and capitalization were corrected to minimize bias that might influence the scoring process as suggested by the literature (e.g., [34]). Teachers were encouraged to read the composition to obtain a general impression of overall writing quality. Compositions were then scored on fourteen 5-point Likert scales (1 = low quality; 5 = high quality), ranging from 0 to 65 points. All compositions from the same class were scored independently by a dyad (teacher-research assistant) using the mentioned rating scale. Each dyad met every week to find a consensus about the grades for each composition as previously stated (see procedures subsection). Cohen’s Kappa coefficient showed an inter-rater agreement that ranged among the 20 dyads between .82 and .90 (M = .86, SD = .023) which can be labeled as “almost perfect” according to authors [63]. The compositions rated for each topic were assessed and the final score were delivered before students write the following composition.

Journals.

Feedback on the week-journals was not provided to students. In the end of the study four new research assistants who were unfamiliar with the design of the study, assessed all journals quality using the same holistic rating scale. Two research assistants assessed each journal independently, following procedures similar to those used to assess the compositions. The Kappa value obtained was .84, considered as very good according to Landis and Koch [56].

Prior achievement.

Prior achievement in Portuguese language was obtained from students’ writing quality scores on three compositions written between April and June from the previous school year (third grade). Two independent research assistants scored the compositions by following the same procedures as described above. Compositions were scored on fourteen 5-point Likert scales (1 = low quality, 5 = highly quality), ranging from 0 to 65 points (M = 50.46, SD = 8.63). Cohen’s Kappa coefficient showed an inter-rater agreement of .87, which can be labeled as “almost perfect” according to authors [63].

Selecting the best model.

To address the goals of the present study (i.e. compare the performance of subjects receiving training in writing skills with the performance of subjects with no training, verify whether all treatments have the same effectiveness, and determine which of two treatments (C or D) was more effective); first the best linear mixed model to the CS use data was selected. Tables 2 and 3 present the results of fitting eight growth curve models to the CS data using full ML in SAS PROC MIXED. Table 2 summarize the results for five multilevel models applied to CS data as follows: the unconditional two-level growth model (A) examined the standard linear change, the unconditional two-level growth model (B) and three-level growth model (C) examined the quadratic change, the conditional three-level growth model (D) examined the effects of the time-varying predictors and their interactions through time, and the conditional three-level growth model (E) examined the process of adding time-invariant predictors to models. Table 3 presents the models that incorporate the effects of treatment conditions, both with and without the heterogeneous variance specifications at level 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results of fitting alternative multilevel models for change to the composition skills data.

https://doi.org/10.1371/journal.pone.0218099.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of fitting alternative homogeneous and heterogeneous level-1 variance models for change to the composition skills data.

https://doi.org/10.1371/journal.pone.0218099.t003

To facilitate the selection of the best model, results (not shown in the table due to space) corresponding to the unconditional means model (i.e., a no-change trajectory model) were described. The estimated outcome grand mean across all occasions and students was 54.29 (p < .001), which suggests that between the first and the twelfth week, the average CS is non-zero. Examining the variance components, we found statistically significant variability both within-students (31.55, p < .001) and between-students (39.37, p < .001). Findings allowed to conclude that CS outcome varies from week to week, and also that students differ from each other.

To determine whether the unconditional means model was preferable to Model A, the compound null hypothesis was tested on a set of differences between models (e.g., regarding the linear growth rate, its associated variance components and covariance between slope and intercept—this last term is not shown in the table due to space). The difference in deviance statistics, (31830.5–30516.5) = 1314, far exceeds 16.27, the 0.001 critical value of a χ² distribution on 3 degrees freedom (df), allowing to reject the null hypothesis (H₀) at the p < .001 level stating that all the three parameters are simultaneously 0. Hence, the unconditional two-level growth model provides a better fit than the unconditional means model. It is possible to conclude that Model A is the best fit model? Comparison of Models B and A suggest a positive response. Comparing deviance statistics for pair of nested models yields a difference of 189.8. As this exceeds the .001 critical value of a χ² distribution on 4 df (18.46), the H₀ is rejected, and we may conclude that there is potentially predictable variation in the acceleration rate across students. For Model B, despite the variance for quadratic component of change (r_2i) being statistically significant (p < .001), its associated fixed effect (TIME²) is not. The tests associated with the random acceleration parameter indicate that there is substantial variation in the quadratic rates across students. The test for the fixed effect indicates that the average value of these rates is indistinguishable from 0. Thus, the trend across time is essentially linear at the population level but curvilinear at the individual level.

Then the unconditional quadratic three-level Model C was compared to the unconditional quadratic two-level Model B. Since students are nested within classes, and may vary considerably among themselves, a three-level model of level-1 occasions nested within level-2 students nested within level-3 classes was also used to analyze this clustered longitudinal design. As there are only 20 classes, CS dataset is not ideal for building a three-level growth model, but it can still be useful for descriptive purposes. As indicated in Table 2, the deviance statistics and number of estimated parameters for the unconditional Model C were 30011.4 and 16, respectively. The likelihood ratio test comparing the Model C to Model B yields a deviance difference statistically significant at any alpha level we might reasonably select (30326.7–30011.4 = 315.3, with 6 df, p < .001). Findings indicate that the more complex model provides the better fit. Each information criterion is consistent with that judgment.

Because we are interested in finding a level-1 individual growth model that describes the fundamental structure of these data, additional time-varying predictors and interactions among level-1 predictors and TIME (i.e., SE_W, SR_W, AT_W, SE_W × TIME, SR_W × TIME, and AT_W × TIME), but also the required additional variance and covariance components (see Model D) were included. Although not shown in the Table 2, the covariance components were not constrained to be 0. When comparing the Model D with the Model C, there is significant evidence that the model incorporating the main effects of time-dependent covariates and interactions fits better (i.e. the difference in deviances was (30011.4–29960.6) = 50.8; df = 6, p < 0.001). Having identified an appropriate level-1 model, the additional effects of time-invariant predictors were included in the level-2 model (i.e., AGE, GEN and P_ACHIEV). For Model E (i.e., model that incorporates time-varying predictors, time-invariant predictors, and the level-1 interactions), the deviance statistic was 29441.1 with 25 df, and 29960.6 with 22 df for Model D (i.e. model that only incorporates time-varying predictors and the level-1 interactions). As a result, the likelihood ratio test statistic was 518.5 with 3 df (p < .001), which provides strong evidence for Model E. Although the Model E provides a more realistic representation of the pattern of change in CS scores than Model D, the Model E contain terms that are not necessarily required. In this paper an even more parsimonious model will be assessed (i.e. Model F). Model F is a simplification of Model E in which the main effect of AGE and non-significant level-1 interaction terms are removed. Comparing the last two models each other, we find a trivial difference in deviance of 0.7 on 3 df, showing that the elimination of AGE, SE_W by TIME and AT_W by TIME has hardly changed the goodness of fit.

After running the appropriate model selection at level-2 for the CS use data, we examined the performance of subjects receiving training in writing skills with the performance of subjects who did not receive such training, and the performance of participants receiving treatment in different modalities. Model G in Table 3 presents the results of fitting this model to data. The final conditional model (Model G) included three class-level variables (i.e., the aforementioned set of Helmert contrasts for group), two student-level variables (GEN and P_ACHIEV) and five within level-1 repeated observations (TIME, TIME², SE_W, SR_W and AT_W). The cross-product between SR_W and TIME and cross-level interaction term H₁ by linear TIME (i.e., difference between the control and treatment groups across time) were also included in the Model G. Data in Table 3 and indicated that adding the three group-related Helmert contrasts (i.e., H₁, H₂ and H₃) cross-level interaction between H₁ and TIME to the model which decreased the deviance from 29441.8 to 29407.5, a decrease of 34.3. This change in deviance is tested at 4 df using the χ² statistic and was found to be significant.

It might appear, that Model G is preferable. But before proceeding to examine Model G in depth, we considered the possibility that the residual variances at level 1 may depend on treatment groups (see, [56]). Returning to Fig 1, we note that participants display considerable heterogeneity across the groups. Thus, we might hypothesize that residual variance at level 1 in these data is different for the four conditions. Table 3 presents estimates for homogeneous variances (Model G) and for heterogeneous variances that occurs at level-1 (Model H). The likelihood ratio test comparing Model G to Model H, shows that the deviance declined 132.1 (29407.5–29275.4), which far exceeds the .05 critical value of a χ² distribution on 3 df. We therefore may reject the null hypothesis stating that all four variances are equal and conclude that the heterogeneous model fits this data better than the simple homogeneous level-1 specification. For this reason, Model H was adopted as our “final model” (see Table 3). The AIC (BIC) weight of this model (> 0.97) implies that there is a high probability that this is the best model among all of the examined models.

Analysis of the selected multilevel model.

Once a suitable final model was selected, the results for the fixed effects corresponding to Model H were analyzed (see Table 3). The comparison of the regression coefficients allows to conclude that the constant ( = 55.408; p < .001) and linear trend terms ( = .552; p < .001) are significant. The intercept being significant is not particularly meaningful (i.e. indicates that CS scores are different than zero at midpoint of time). However, because the trend is essentially linear at the population level, we may conclude that participants are improving across time. On the contrary, the quadratic term is nonsignificant at the individual level ( = .022; p < .208). Similar inspection of the other parameter estimates in Model H shows that CS score was positively associated with prior achievement ( = 3.139; p < .001), SE_W ( = .390; p < .0027), SR_W ( = .660; p < .001) and AT_W ( = .604; p < .002). On the other hand, the CS score was negatively associated with the cross-product between the self-regulation and linear time ( = -.137; p < .001). The relationship between the self-efficacy, attitude toward writing and the CS score were constant across time (i.e., no time interactions with these time-varying covariates, see Model D in Table 2). We also found that the gender effect was significant ( = .847; p < .0012), suggesting that performance in CS was higher for girls than for boys.

At class level, the estimates of γ₀₀₁, γ₀₀₂, γ₀₀₃, and γ₁₀₁, and their estimated standard errors are of primary interest in Model H. Table 3 indicates that the difference between the control and treatment groups in the mean response at the midpoint of time was significantly different from zero ( = 5.165; p < .0001). This indicates that the intervention had a statistically significant effect on CS score. In addition, due to the marginally significant effect of cross-level interaction between H₁ and linear TIME ( = .272; p < .068), it seems that this benefit increases over time. The regression coefficients of the H₂ contrast were inspected to determine whether a differential performance in the mean response CS had occurred in the intervention WJ group in comparison with groups C and D. In Model H the effect of γ₀₀₂ was estimated to be 1.738 while its corresponding standard error was estimated to be 0.579. The ratio was 3.01 and the p-value was approximately 0.008, which indicates a significant benefit for participants who received treatments C and D (i.e., SRSD or SRSD + SRL) in relation to participants of WJ group at the midpoint, and further suggests that this effect does not vary significantly across time (i.e., no time interactions with the second and third Helmert contrasts). Finally, regarding H₃, although performance is higher for group SRSD + SRL, no evidence found differences in CS scores among SRSD and SRSD + SRL intervention conditions ( = .709; p < .179).

Finally, following the procedure of Vallejo and colleagues [65] in examining statistical power to detect a significant group-by-time-interaction (i.e., H1 × TIME), a power below the often-mentioned benchmark of 0.80 was obtained; specifically, the post hoc power was found to be approximately 0.44.

Before describing the structure of the random-effects model matrix, we included two intraclass correlations (ICCs) for this three-level hierarchical model (see Table 3, bottom panel). The first is the level-3 ICC at the class level, the correlation among quality of compositions from different second level students nested on the same class. The second is the level-2 ICC at the student-within-class level, the correlation among quality of compositions measured at different time points in the same student and class. We found that the quality of compositions is strongly correlated within the same student and class, but only slightly correlated within the same class, while this ICC is non-negligible. Table 3 (bottom right panel) also displays the design effect (DEFT) indices at levels 2 and 3 in Table 3. DEFT is used to determine how much larger the standard errors estimates will be (considering clustering compared to the analysis that ignore clustering). For example, for the ICC in level two (class) (see Table 3) the unconditional DEFT is expected to be 1.73; meaning that standard errors would only capture a little more than one-half of the true sampling variability if the third-level was ignored.

Analyzing the variances estimates, data shows that at the student-level the estimate constant variance is much larger than the estimate linear trend component , which is much larger than the estimated quadratic trend component . In terms of relative percentages, these three represent 98.5, 1.4, and 0.1, respectively, of the sum of the estimated individual variance terms. A similar result was observed at class level (, and ), although heterogeneity in trends across time becomes smaller as the order of trend terms increased. Note also that final estimation of level-1 and level-2 variance components has been affected very little by model respecification (Model F vs. Model G/H). However, the final estimation of level-3 variance components has been substantially diminished when compared with the parameters estimates for Model G/H.