Learning analytics and measuring of student learning behavior

Learning Analytics (LA) has emerged as a powerful tool for examining and optimizing learning processes by analyzing student interactions within digital platforms [15,16]. Predictive modelling of academic success in blended learning, in particular, is considered a promising area, especially focusing on instructional conditions [17]. Commonly used LA indicators include frequency of logins, time spent on learning tasks, interaction with study materials, and participation in discussions [18]. The field of LA has grown significantly; however, its applicability has often been limited due to issues related to the generalizability and transferability of findings [11,13,19]. Given the variability in course design, it is essential to contextualize LA indicators to understand their implications for student learning behavior better [20].

Previous research indicates that differences in course structures contribute significantly to students’ engagement levels, learning persistence, and overall performance [21]. Consequently, an effective framework that captures the instructional design features influencing SRL is necessary for deriving actionable insights from LA data. Prior research has highlighted that instructional designs in OLEs exhibit significant variability, affecting the extent to which students engage in SRL [22]. Factors such as the presence of explicit goal-setting tasks, self-assessment activities, and reflective prompts contribute to the degree of SRL support within a given course [23].

Instructional support in online learning environments

Instructional support in OLEs comprises various elements, including structured learning materials, scaffolding mechanisms, timely feedback, and interactive activities designed to enhance student engagement [24,25]. Research suggests that well-designed instructional support can improve students’ self-regulation, enhancing their ability to effectively manage learning tasks [26]. Studies indicate that different instructional support structures, such as guided prompts, formative assessments, and collaborative learning opportunities, influence the extent to which students engage with course content and regulate their learning behaviors [27].

Building on the previous section, it is necessary to distinguish instructional support from instructional design. Instructional design involves the systematic planning, organization, and structuring of course content, activities, and resources to optimize learning outcomes [28]. It focuses on the pre-planned layout and sequencing of the course rather than real-time interactions. In contrast, instructional support involves the guidance, scaffolding, feedback, and other assistance given to students as they engage with the learning environment [29]. By clarifying this distinction, we differentiate between the structural design of the course and the ongoing support students receive while learning.

While perceived teacher support is often considered a critical factor in student engagement, research suggests its impact on learning behavior may be indirect [30]. Instead, SRL support embedded in course design serves as a more direct determinant of students’ self-regulation and engagement levels [31], although the specific mechanisms of SRL still provide mixed evidence (see, e.g., [32,33]). To systematically measure the degree to which course designs support SRL, it is necessary to consider the creation of a specific research framework engaging coding scheme, aiming to assess the variability in SRL support and link these differences to students’ learning behavior, including regularity and intensity of engagement [34,35].

SRL framework guiding the study

To anchor our analysis, this study adopts Zimmerman’s cyclical model of self-regulated learning [36–38], which conceptualizes SRL as a dynamic process consisting of three interrelated phases: (1) the preparatory phase, where learners set goals and plan strategies; (2) the performance phase, where learners engage in task execution, monitoring, and control; and (3) the reflective phase, where learners evaluate their progress and outcomes. This cyclical structure has been widely applied in SRL research because it captures both the proactive and reactive aspects of self-regulation and it encompasses metacognitive, cognitive, motivational, and behavioral processes that enable learners to set goals, choose strategies, and adapt their learning to achieve success [39]. However, it also engages time management, monitoring, self-evaluation, and self-reflection [40]. This model also aligns well with the affordances of online learning environments, in which students often must self-direct their progress over extended periods of time. In OLEs, where students often engage with content asynchronously and autonomously, SRL plays a critical role in determining learning effectiveness [41–45]. Given the flexible yet demanding nature of OLEs, SRL-supportive instructional designs become essential to fostering meaningful student engagement and learning outcomes [46]

Building on Zimmerman’s framework, we also incorporate the cognitive–affective distinction in SRL processes [25,47]. The cognitive dimension refers to mental strategies such as planning, monitoring, and applying learning techniques, while the affective dimension refers to motivation, self-efficacy, interest, and emotional engagement. This dual framing is especially relevant in OLEs, where both the management of learning strategies and the regulation of motivation/emotion determine the persistence and regularity of engagement [1]. In SRL, cognitive and affective dimensions are considered predictors of learning outcomes [47]. Cognitive support targets thinking and strategy use [38], whereas affective support focuses on emotional engagement [25]. Distinguishing these helps understand their influence on students’ online learning behavior. At the same time, the emotional dimension of learning in OLE is especially considered understudied [1] and should be further studied.

This integrated model provides the theoretical foundation for the methodological design of our study in several ways. First, the coding scheme for analyzing instructional design features was structured around the three SRL phases and further differentiated by whether a feature primarily supported cognitive or affective processes. Second, the selection and interpretation of learning analytics indicators were guided by this model. Finally, our research questions build directly on this framework by examining not only overall SRL support in OLEs but also the distinct contributions of specific phases and dimensions to students’ learning behavior.

By grounding the study in this combined cyclical-phase and cognitive–affective model of SRL, we provide a coherent theoretical lens that links instructional design, students’ perceptions of instructional support, and observable learning behaviors within online learning environments.

Present study

Despite substantial progress in understanding self-regulated learning (SRL) in online learning environments (OLEs), at least four persistent research gaps remain. First, most empirical studies deal with individual courses or a limited number of courses. This results in a narrowly defined context that significantly limits the generalizability of findings to diverse online courses and environments [10,12]. Second, research often focuses on student-level variables while inadequately exploring how differences in course instructional design shape learning behavior. This is despite the fact that course-level differences have been proven to play a crucial role in shaping students learning activity and their learning outcomes in OLEs [11,13,22]. Third, SRL is often examined as a unitary construct, with little attention given to the distinct roles of the cognitive and affective dimensions or the preparatory, performative, and reflective phases that constitute its cyclical nature [1,38,42]. Finally, little attention has been given to the mechanisms linking the instructional design of SRL support at the course level to students’ perceptions of instructional support and how these two factors together affect the intensity and regularity of students’ online learning behavior in courses.

This study directly addresses the aforementioned gaps by collecting and analyzing a multi-source dataset from 76 different courses. This dataset combines instructional design data aimed at supporting SRL, which was collected through a qualitative content analysis of the courses; student self-reports of perceived instructional support, which were collected through a questionnaire; and behavioral traces of students’ learning behavior, which were collected from an authentic online learning environment. Thus, this study aims to contribute new knowledge to SRL research in OLE contexts by investigating (1) the extent to which SRL-supportive design features of courses predict student online learning engagement, (2) how these features interact with students’ perception of instructional support, and (3) how cognitive vs. affective dimensions and SRL phases differentially shape learning behavior.

Using qualitative content analysis of selected courses, we developed a coding scheme that enabled us to measure the extent to which individual courses’ design supports particular SRL processes. Three independent researchers critically reviewed the coding scheme and then applied it to the analysis of 76 different courses. To gain insight into how students perceived the courses, participants of analyzed courses were asked to fill in the online questionnaire mapping (among other things) the instructional support from the teacher within the course. To capture students’ learning behavior, several indicators of learning behavior in the individual courses were extracted from OLE at the end of the semester.

Based on this multi-source dataset, our study aims to answer the following research questions:

• RQ1: To what extent do the perceived instructional support from the teacher and the self-regulated learning support in OLE predict the intensity and regularity of students’ online learning behavior in courses?
• RQ2: To what extent does the self-regulated learning support in OLE predict perceived instructional support from the teacher?
• RQ3: Does the perceived instructional support from the teacher serve as a mediator for the relationship between the self-regulated learning support in OLE and the intensity and regularity of students’ online learning behavior in courses?
• RQ4: To what extent does the support of individual SRL dimensions (cognitive vs. affective) predict the intensity and regularity of students’ online learning behavior in courses?
• RQ5: To what extent does the support of individual SRL phases (preparatory, performative, reflective) predict the intensity and regularity of students’ online learning behavior in courses?

The figure below (Fig 1) illustrates the models considered in relation to the research questions above. The conceptual model shown in the upper left part of the figure deals with research question 1 (RQ1). The upper right part relates to research questions 2 (RQ2) and 3 (RQ3). The lower left part shows the model addressing research question 4 (RQ4), and the lower right part addresses research question 5 (RQ5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Conceptual models of the relationships considered in relation to the research questions.

https://doi.org/10.1371/journal.pone.0332296.g001

Sample and procedure

A total of 882 students enrolled in 76 different courses held at the Faculty of Arts, Masaryk University, Brno, Czech Republic, were part of the analyses in this study. Only those students who completed the questionnaire and agreed that their OLE behavior data could be analyzed for this research were included in the sample from these courses. The study was conducted in accordance with relevant guidelines and regulations following the principles of Declaration of Helsinki, including approval from the ethics committee and written informed consent. Participants were briefed on the study’s purpose and asked to provide informed consent by selecting “I agree” in the online system before accessing the questionnaire. They were informed that they could refuse or withdraw from the study at any time without facing any consequences. The research project received approval from the Research Ethics Committee of Masaryk University (project identification number: EKV-2020–037). At the same time, only courses with responses from at least five different students were included in the sample (due to the planned multilevel analyses).

The mean age of the respondents was 21.98 years (Med = 21, SD = 4.25). As for gender, 76.87% of the respondents were female, 21.77% were male, and 1.02% of the students chose the option ‘other.’ This distribution is fairly consistent with the overall gender distribution of students at the Faculty of Arts. Both bachelor (77.55%) and non-follow-up master (21.77%) students were approached to participate in this research and complete the questionnaire. Similarly, full-time and part-time students were invited to participate in the research. However, most students in our sample are full-time (94.56%). In terms of courses studied, 43.95% of students studied compulsory courses, 22.56% of students studied so-called compulsory electives (i.e., courses that are part of a compulsory block, but students choose only a certain number of courses from the available range), and 33.49% of students studied in elective courses (i.e., courses that are optional in relation to a specific course of study). In 27.80% of the cases, the course was in the form of seminars; in 40.09% of the cases, the course was mainly in the form of lectures, and in the remaining 32.10% of the cases, it was a combination of lectures and seminars.

Regarding the procedure, data collection took place over three semesters—namely, the autumn semester of 2021, the spring semester of 2022, and the autumn semester of 2022—from November 22, 2021, to December 23, 2022. First, in each of the semesters, teachers who taught their courses in an online learning environment (OLE) at the faculty were contacted and asked to include their course in the research. Each semester, we aimed to recruit at least 30 courses for which their teachers agreed to participate in the research. At the same time, these were always unique courses in each of the semesters (i.e., data were not collected from the same course in two different semesters). Then, in each of the three semesters, all students in the selected courses were approached to complete the questionnaire, which focused on various dimensions of self-regulated learning and other relevant factors. Subsequently, after the end of the semester, relevant data on students’ behavior in the online learning environment were extracted for those students who agreed to participate in the research. The entire research, i.e., the distribution of the questionnaire and the extraction of data from the selected courses in the online learning environment, was carried out in collaboration with the teachers of the selected courses and the Centre for Information Technologies, which manages the faculty’s online learning environment. The online learning environment used at the faculty was Moodle.

Measures

Self-regulated learning support.

To capture how SRL is supported in OLE, we conducted a directed content analysis [49] of 11 courses purposefully selected to cover the widest possible variety of a course design within one faculty. In these courses, we aimed to code all units (tools, structures, and discourse) that might serve as SRL support. In this step, we have indicated about 60 codes that were attributed to the three phases of SRL (preparatory, performance, reflective) and, at the same time, to the type of SRL process (metacognitive or cognitive and affective or motivational). Codes were joined into categories capturing all identified ways of SRL support. Five additional courses were analyzed to ensure data saturation at this step, which led to further elaboration of categories. Critical codes in each category were selected and reformulated to formative indicators, ensuring that all previously indicated ways of SRL support were included in the scheme. The developed tool SRL-S (Self-Regulated Learning Support) was used in a quantitative content analysis of 16 courses done by three coders (one ICT researcher, one university teaching researcher, and one educational ICT specialist) [50]. After four rounds of clarification and cyclical reformulation of indicators, researchers achieved agreement on the final formulations of indicators and on coding particular features.

SRL-S consists of 55 items indicating the presence or absence of a particular SRL support process. For each item, coders decide between 1 (presence) and 0 (absence) values. Items are divided into three phases: preparatory, performance, and reflective. In each of these phases, support of two groups of processes is measured: cognitive and metacognitive processes and affective and motivational processes. Support for each process within these groups is indicated by five items. Thus, each SRL process—such as goal setting, time management, or help-seeking—is scored on a 0–5 scale indicating the degree of support provided, with higher values reflecting stronger support. The tool also produces aggregated scores across learning phases (preparatory, performance, reflective) and self-regulated learning (SRL) dimensions (cognitive/metacognitive vs. affective/motivational), calculated as the mean of the relevant process scores (thereby maintaining the original 0–5 scale). An overall SRL support score is computed as the mean of all process scores. The table below (Table 1) presents a complete version of the SLR-S coding scheme.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of the coding scheme.

https://doi.org/10.1371/journal.pone.0332296.t001

As mentioned above, each of the 76 courses was independently coded by three different raters. To assess the reliability of the coding procedure, we calculated the inter-rater reliability (IRR) using intraclass correlation coefficient (ICC). For the overall SRL support score, the ICC indicated 0.936. For the individual phases of SRL, the ICC indicated values 0.911 (Preparatory phase), 0.912 (Performance phase), and 0.887 (Reflective phase). For the two SRL dimensions, the ICC indicated values 0.915 (Cognitive and metacognitive dimension) and 0.917 (Affective and motivational dimension). Values above 0.75 are generally considered to be a good match, values above 0.9 are an excellent match [51], thus the results of the IRR analysis confirms that the coding procedure provides consistent ratings across different coders.

Research question 1

The first research question aimed to determine whether and to what extent support during learning is related to the learning behavior of students in courses in an online learning environment (OLE) in terms of its intensity and regularity. Specifically, we questioned to what extent the perceived instructional support and self-regulated learning support in OLE predict the number of course visits, the total time spent in a course, and the irregularity of course visits. To address this research question, three separate models were estimated in which the three aforementioned indicators of students’ online learning behavior were used as dependent variables, and two types of support served as independent variables. The estimated models are presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Effects of perceived instructional support and self-regulated learning support on indicators of students’ online learning behavior.

https://doi.org/10.1371/journal.pone.0332296.t003

It can be seen from Table 3 that SRL support in courses in an OLE proves to be a significant predictor of all three types of students’ online learning behavior in courses. Thus, it can be said that the design of each course, in terms of how it reflects the teacher’s efforts to support students’ self-regulation, has a significant impact on how students behave and learn in the course, precisely how often they visit the course, how regularly they visit the course, and how much time they spend in total in the course during the semester. A higher level of self-regulated learning support in the course leads to a higher number of visits, less irregularity of visits (i.e., more regular visits), and a higher total amount of time spent in the course by a student. On the other hand, perceived instructional support from the teacher affects students’ online learning behavior only in terms of their total time spent on the course. So, the more students feel that the teacher supports them in the course (e.g., gives clear instructions, gives feedback, answers follow-up questions, etc.), the more time they will spend on the course overall.

Research question 2

The second and third research questions aimed to explore in more detail the relationship between self-regulated learning support in OLE courses and the teacher’s perceived instructional support. Specifically, we were interested in whether or to what extent the level of perceived instructional support from the teacher is determined by the extent to which the course design focuses on supporting students’ self-regulated learning (RQ2). In other words, the extent to which the course design, in terms of its support for SRL, is reflected in students’ perceptions of their teacher and his/her ability to provide instructional support. In addition, we want to find out whether self-regulated learning support in courses affects students’ online learning behavior in OLE directly and indirectly through the teacher’s perceived instructional support (RQ3). Thus, the design of a course in OLE, in terms of its support for SRL, will affect students’ online learning behavior directly but also indirectly through its effect on students’ general perception of instructional support from the teacher.

The answer to the second research question is provided by Table 4, which indicates that there is indeed a statistically significant positive relationship between self-regulated learning support in courses and perceived instructional support from the teacher. That is, student perception of their teachers’ ability to provide instructional support is significantly informed by the design of their courses in OLE and the extent to which a course in OLE is designed with self-regulated learning support in mind.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Effect of self-regulated learning support on perceived instructional support.

https://doi.org/10.1371/journal.pone.0332296.t004

Research question 3

To answer the third research question, we conducted a mediation analysis focusing on whether self-regulated learning support in courses indirectly affects students’ online learning behavior in OLE through perceived instructional support from the teacher. Similarly to the first research question, we carried out the media analysis three times for each indicator of students’ online learning behavior (number of course visits, total time spent in a course, and irregularity of course visits) as dependent variables (see Fig 2). The mediation analysis results show that a statistically significant indirect effect can only be detected in the case of total time spent on the course (0.021, p = 0.012). On the other hand, no significant indirect effects were found for the number (0.006, p = 0.360) and irregularity (0.001, p = 0.836) of student visits to the course. Moreover, even in the case of time spent on the course, the detected statistically significant effect can be considered rather small.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Statistical diagram of the mediation model.

Here, c represents the total effect, c’ the direct effect).

https://doi.org/10.1371/journal.pone.0332296.g002

Research question 4

Concerning the fourth and fifth research questions, we aimed to provide a more nuanced view of the effect of self-regulated learning support in OLE on students’ online learning behavior in courses by differentiating between different dimensions (RQ4) and phases (RQ5) of SRL that the support implemented in course design is aimed. Table 5 shows the resulting models in which, instead of the overall level of SRL support in the courses, a distinction was made between levels of support directed at the cognitive and metacognitive dimensions of SRL and at the affective and motivational dimensions of SRL. It is clear from the data in Table 5 that for all three indicators of students’ online learning behavior, only the effect of support focusing on the cognitive and metacognitive dimensions of SRL appears to be statistically significant, while the effect of support focusing on the affective and motivational dimensions of SRL is insignificant. Thus, support that focuses on SRL’s affective and motivational components is not directly reflected in how students behave and learn in courses in OLE.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Effects of perceived instructional support and different dimensions of self-regulated learning support on indicators of students’ online learning behavior.

https://doi.org/10.1371/journal.pone.0332296.t005

Research question 5

The last table (Table 6) then shows the models in which the individual phases of SRL (i.e., preparatory, performative, and reflective) were distinguished instead of using the overall score of SRL support. The results presented in the table suggest that it is mainly the support focused on the performative phase of SRL that significantly affects students’ online learning behavior in courses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Effects of perceived instructional support and self-regulated learning support on indicators of students’ learning behavior.

https://doi.org/10.1371/journal.pone.0332296.t006

Limitations and future research

While the present study provides valuable insights into factors influencing engagement in Online Learning Environments (OLEs), several limitations should be acknowledged. These limitations may affect the generalizability and interpretability of the findings and should be considered in future research endeavors. One of the primary limitations of this study concerns the composition of the research sample. The study did not have complete control over who enrolled in the courses analyzed, leading to potential sampling biases. Participants may have self-selected into courses based on prior interest, experience, or motivation, which could influence engagement levels.

Additionally, variations in course difficulty, instructor effectiveness, and institutional policies were not controlled. Future studies could employ randomized controlled trials or stratified sampling techniques to ensure a more representative and comparable participant pool. Another limitation of this study is that it examined existing courses retrospectively, meaning that instructional elements were coded after course completion rather than being systematically manipulated or designed a priori. As a result, causal inferences cannot be drawn regarding the specific instructional features that enhance engagement. Furthermore, variations in course design, content delivery, and instructor involvement were not standardized across all courses analyzed. Lastly, a small portion of the data (less than 5%) was missing, primarily for the measure of irregularity of visits. It was due to low activity by some students. We addressed this issue by using pairwise deletion, which preserved most of the available data for each analysis. Due to the minimal amount of missing data, we consider this limitation to be minor and unlikely to substantially affect the results.

Future research could adopt experimental or quasi-experimental designs, allowing controlled manipulation of instructional variables to establish more apparent causal relationships, which can be used for studying unstructured data such as graphics and videos or emotional aspects of learning in OLEs [1]. Also, it should further explore the interplay between SRL support, perceived instructional support, and learning behaviors across diverse OLE settings. Given the variability in course designs and student demographics, longitudinal studies are needed to assess the long-term impact of SRL-supportive instructional strategies. Moreover, extending the current research framework by incorporating additional LA indicators—such as discussion forum interactions or assessment performance—could provide a more comprehensive understanding of how instructional designs shape online learning experiences [7]. By addressing these limitations, future studies can build a more comprehensive understanding of engagement in OLEs and contribute to developing evidence-based instructional strategies.

Practical implications

Based on the above results, SRL-S can measure features of OLE that influence students’ behavior. Such a tool enables us to measure SRL support, contextualize data received from OLE, and thus enhance the findings of learning analytics or interventional studies. The finding that improving the level of SRL support enhances perceived teacher support and students’ learning behavior should be used to guide focused teacher support. We can quickly diagnose a particular course and scaffold teachers in supporting SRL processes that are weakly encouraged within the course. Such targeted support might be more effective than focusing on the overall development of technological competencies. SRL-S might be an easy-to-use tool for teachers’ self-evaluation or guiding teachers in designing OLE. Furthermore, a tool like SRL-S could inspire the development of a data-based framework for assessing the quality of OLE at the institutional level.