Step 1 – Initial development of OPTIS based on exiting collection of teaching principles, the constructivist learning theory and existing observation protocols.

When developing an observation protocol, there are different developmental approaches in research [26]. Either, researchers came up with a first catalog of observation items gathered from literature [71–76], they built their protocols on existing observation protocols [11,59,61,77,78] or based their protocols on a mixture of both approaches [4,48]. These approaches were followed either by a group of experts reviewing and discussing the initial selection of observation items [70–72,75,77], by piloting the protocol directly in the field [59,74] or by adjusting the initial items based on the observation of video-recorded teaching [4]. Each of these approaches has its strengths. Therefore, all the approaches mentioned above were integrated in the development of OPTIS at different times.

Firstly, OPTIS was based on literature-derived teaching principles. In doing so, following the recommendations of Praetorius et al. [26,41], an inventory of commonly mentioned teaching principles from the literature was compiled. These evidence-based principles were added with quality criteria (“teaching principles”) needed to reflect the constructivist understanding of learning (theoretical foundation) that OPTIS follows (Table in S1 Table). This approach resulted in a first list of six dimensions of teaching and 27 teaching principles as a first version of OPTIS. After compiling this initial list of pertinent teaching principles, it was necessary to describe them with observable categories (observation categories). Providing well-described observation categories for each teaching principle was deemed necessary, as the close interrelation among different teaching principles could otherwise make it difficult to distinguish between them [40]. Therefore, existing, available, and validated observation protocols were reviewed, and three protocols were selected for specifying the observation categories: ELL [66], UBB [48], and an unpublished protocol from the MOVEOUT project [79]. The previously described protocol TEACH [4] was not used, as its intended purpose and origin differed too much from the aims of the present study. ELL [66] aligns with the intended purpose and therefore served as an inspiration for the observation categories in OPTIS. Furthermore, observation categories from the observation protocol UBB [48] helped to describe the literature-based teaching principles with observable categories. With the help of these two protocols, 16 of the 27 teaching principles could be described with observation categories. For the remaining eleven teaching principles, the research team formulated observable descriptions, based on the theoretical understanding of each teaching principle. After establishing the content of OPTIS, insights from an unpublished observation protocol from the project MOVEOUT [79] inspired the make-up of OPTIS. Aspects, such as summarizing all information on a single page and including additional details – such as “time,” “subject,” and “teacher” – were incorporated based on this protocol.

Subsequently, a user manual with examples for each observation category was developed. Content, design, and approach from Helmke [80] inspired the user manual of OPTIS. Each observation category was described with examples, which should help the observer to rate the extent to which the respective teaching principles can be observed (see the user manual in the appendix S2, which, in its current state, is only available in German).

In addition, the developmental progression included a determination of observation (time) intervals, which will structure observations using OPTIS. For observation intervals, different observation periods are commonly used during observational practice. The influence and importance of the length of observation intervals are discussed by Mashburn et al. [81]. Different observation intervals can vary from short snapshots of a few seconds [82–85] to intervals of one hour [75]. Helmke et al. [86] propose to use episodes, structured by the change of a specific didactic function or the applied social form. To target the goals of OPTIS to analyze teaching and gather as much information as possible, it appears appropriate to utilize applied social forms to define observation intervals, as different social forms promote the use of different teaching principles. Therefore, observations with OPTIS are structured with the help of five social forms (individual work, partner work, group work, plenum, and frontal teaching). Thus, each time the social forms changes, a new observation interval begins.

Regarding the application of rating scales in observing instructional quality, common practice varies from binary scales (e.g., [75]) to seven-point scales (e.g., [4,87]), with four-point (e.g., [88]) or five-point scales (e.g., [66]) being the most frequently used. Increasing the number of rating options allows for finer differentiation between observed behaviors; however, in real-time observation, more options make it more challenging to assign ratings quickly. Additionally, odd-numbered scales tend to encourage raters to select the middle option, while the even-numbered scale encourages raters to make more decisive judgments. Helmke adds another option, by including a fifth rating option, “not observable,” to his four-point ELL observation protocol. During the test phase, OPTIS was applied using a five-point rating scale, following Helmke’s recommendation to include the “not observable” option. In the final piloting of OPTIS, a six-point scale, also frequently used in the literature (e.g., [48]), was additionally applied to compare different approaches. The results of this comparison are reported in the Results section.

Step 2 – Testing and adjusting OPTIS based on expert feedback and real-time teaching observation.

At several points during the development process, experts were engaged in reviewing the observation protocol and provided ongoing verbal and written feedback to clarify the initial list of teaching principles and their observation categories. The first version of OPTIS was sent to a multi-disciplinary group of experts, consisting of teachers (n = 3), experts from education development practice (n = 2), educational research (n = 3) and observation specialists (n = 1). This diverse composition ensured that both practical and theoretical perspectives were considered, allowing the instrument to reflect the needs of practical teaching aspects while maintaining a strong scientific foundation. In July 2023, OPTIS went into its first test phase. During these field tests, OPTIS was tested by two independent observers in four different elementary schools in Bavaria, Germany. Refining the protocol’s categories and testing the usability during real-time observation were the main objectives of these first school observations. Furthermore, the user manual was supplemented with additional examples for each observation category drawn from those observations. The extension of the user manual was necessary to create the observer training later on. After completing the test phase, learnings from using OPTIS also led to rephrasing and modifying some observation categories.

Based on the experiences made during the test phase, the minimum length of observation intervals was restricted to five minutes, as Helmke [89] suggested. If social forms are changed within less than five minutes, the observation protocol remains unchanged, and the observation interval stays the same. However, the alteration in social form should be documented in the comment section, which is placed below the rating scheme. Additionally, in this comment section, the content of the teaching and the methods used by the teacher should be written down. This section was added due to recommendations from Praetorius et al. [41] that indicated that teaching principles varied in dependence on objectives, contents, and methods of teaching.

Within the second step, the observer training was developed. In previous observational practice, the duration of observer trainings varies widely, ranging from just a few hours to two weeks [21,70]. Throughout the development process, various training methods were evaluated (e.g., individual engagement with OPTIS and discussions during observation training with video material), leading to the design of a comprehensive three-step observer training program spanning approximately eight hours. This observer training starts with an individual confrontation with the observation protocol and associated user manual. A joint discussion followed once observers were confident in understanding each observation category and their differences. These first steps of creating a common theoretical understanding are essential for observer training [21,90]. During the test phase of the development of OPTIS, the observer training took five hours, only consisting of step one and three of the final observation training (it did not involve observations of video-recorded teaching). Reliability was computed after these training steps during the test phase. Reliability values were unsatisfying. Therefore, after establishing a common understanding of the observation items (first step of the training), watching video-recorded teaching was added to the observer training. The website https://www.fdz-bildung.de/videoportale provides a huge amount of freely accessible video-recorded classroom teaching for different subjects, teachers and grades. At least one video (45 minutes) should be observed during the observation training for using OPTIS (step two). Afterwards, observation ratings should be compared with other observers and discussed (step three). If necessary, more video-recorded teaching should be rated by using OPTIS, until the understanding of each teaching principle with its observation category is clear to all observers.

In October 2023, the test phase was followed by discussing and presenting OPTIS during a workshop in Switzerland, where experts from early childhood education and outdoor education met for discussing current practice and visions for the future. Especially the last teaching principles of OPTIS (learning environment, use of learning environment) as well as the criteria of instructional quality in tradition of the constructivism (e.g., action orientation, experience orientation) were discussed with experts (n = 9) from outdoor teaching and educational research. Besides this, many observation categories were revised and supplemented. After implementing the experts’ feedback and the learnings from the test phase, OPTIS consisted of 27 teaching principles with 39 observation categories. The amount of the teaching principles OPTIS consists of at this stage is equal to the test-phase, but two teaching principles were renamed. These principles and observation categories are displayed in Table 1. At this developmental stage, OPTIS went into the final piloting.

Step 3 – Piloting OPTIS with two independent and trained observers.

For the final piloting, two independent and trained observers rated video-recorded and real-time teaching from November 2023 to January 2024. During piloting, the two different rating scales were tested. Additionally, feasibility was continuously discussed and reliability of these two scales was calculated. First, ratings were realized using a five-point rating scale (like used by [66]). The five-point rating scale provides the rating options “does not apply”, “rather does not apply”, “rather does apply”, “does apply” and the option “not observable”. Further on in the piloting, observations with a six-point rating scale were piloted. The six-point rating scale includes the options +++ (“does apply completely”), ++ (“does apply”), + (“rather does apply”), – (“rather does not apply”), - - (“does not apply”), - - - (“does not apply at all”).

In this final stage, minor adaptations to OPTIS were made, mainly concerning the make-up of OPTIS. For example, the observation date was added to the heading of OPTIS or the articulation of observation categories were adjusted. The final version of OPTIS can be found in the appendix (S2 Table).

Procedure.

During the test phase, real-time observations in four different elementary schools in Bavaria, Germany, were carried out. Two observers visited each school for one day and observed in total 13 school lessons by using the five-point rating scale. Because OPTIS was meant to be applicable subject-independently, data collections were realized across different subjects. For the test phase, lessons in mathematics, German, science, art, physical education and English were observed. The first author acted as one of the observers, the other observer was a university student.

The next phase, the piloting, started with observing video-recorded lessons. To be able to evaluate the intra-observer reliability, two observers rated 13 video-recorded school lessons twice. In this phase, the first author acted as one of the observers and a university student, different from the one during the test phase, acted as the second observer. The observed lessons were from different subjects (mathematics, German, physical education, French and science), took place in different grades (second to ninth grade) and were conducted by nine different teachers, across different cultures (Germany, Switzerland, and the UK). The average time between the first and the second rating amounted to two days. In total, both observers rated nine lessons by using the five-point rating scale and four lessons using the six-point rating scale. Two lessons were observed by both raters, using the five-point rating scale.

During this piloting, ten video-recorded school lessons and 13 real-time lessons were observed by the two observers from the intra-observer validation. The video-recorded lessons spanned from second to ninth grade, encompassing one lesson from second grade, one from third grade, two lessons from fourth grade, and one from fifth, sixth, seventh and ninth grade, respectively. The video-based piloting was implemented in the subjects of mathematics, German, science, physical education and French. During this piloting, the two observers rated six lessons with the five-point rating scale and four lessons using the six-point rating scale. For the real-time piloting, teaching was observed in a fourth grade in an elementary school in Bavaria, Germany. Lessons in mathematics, German and ethics were observed during these school visits. For the real-time piloting, the two observers rated three lessons using the five-point rating scale and ten lessons using the six-point rating scale.

Content validity.

To ascertain whether OPTIS really assesses instructional quality within the framework of a constructivist approach of learning, and to enhance and deliberate upon the content validity of OPTIS, 18 experts reviewed the observation protocol at different time points during the developmental process. The experts’ feedback was received in five consecutive rounds. The first round consisted of written feedback from three experts in the education section (one schoolteacher, one university teacher and one school seminar leader). They provided feedback on the general applicability of OPTIS and were asked to eliminate or add further teaching principles to the initial list. The second round, involving three experts from school and university teaching, was dedicated to feedback concerning the connection of teaching principles with their observation categories. Further, the experts were asked to identify duplicates among the observation categories. This expert round started with an independent analysis of each expert of the observation protocol and ended with an open online discussion guided by the main author. In the third round, the resulting version of OPTIS was reviewed by one teacher, with specific regard to the applicability in school. This third round was then followed by the test phase of the first version of OPTIS. This test phase was followed by the fourth expert round with two experts from educational and school observational research. They provided written and oral feedback about the extent to which the observation categories can measure the respective teaching principle they describe. Additionally, the rating scales were discussed. The resulting version of OPTIS was then presented during a workshop with nine experts from educational research, which represented the fifth round of evaluating the content validity. During the workshop, especially the constructivist understanding of teaching and the associated teaching principles were in the focus of discussion. Finally, the observability of each observation category was checked during the final video and real-time piloting.

Data analysis.

In accordance with classical test theory, an observation protocol should be examined with regard to the traditional criteria of objectivity, reliability, and validity to establish the foundation for defining the quality of an observation [48,91]. In data assessments using observations, the objectivity of the ratings is highly connected with the reliability of the observation. The intra-observer reliability refers to the consistency of observations made by a single observer across multiple time points. Inter-observer reliability refers to the agreement between two or more raters and measures the extent to which different raters provide similar judgements or scores for the same observations [91]. When analyzing classroom practice by means of observation, observer bias is considered as a crucial aspect [21,92]. To discuss this bias, inter- and intra-observer reliability was calculated by using Gwet’s Agreement Coefficient 2 [93,94]. In comparison to other commonly used reliability coefficients (e.g. Cohen’s Kappa or Krippendorff’s Alpha), Gwet’s AC2 is less affected by prevalence or marginal probability problems [95,96]. Contrarily, evaluating the inter- and intra-observer reliability with Gwet’s AC2 shows very stable reliability coefficients for observer agreement and may therefore serve to avoid these problems [93]. AC2 is based on the same underlying principles as AC1, but it differs in the calculation of the expected agreement. The formula for AC2 involves computing observed agreement and expected agreement adjusted for chance, similar to the calculation of AC1. However, AC1 only supports categorical data, whereas AC2 can be used when dealing with ordinal data [93].

Formula for calculation of Gwet’s AC2:

P_{o =} Observed agreement

P_{e =} Expected agreement adjusted for chance

The values of P_o and P_e are calculated based on the observed and expected frequencies of agreement, considering the overall distribution of ratings across all categories. This adjustment accounts for chance agreement and provides a reliable estimate of inter- and intra-observer reliability. Observation ratings were saved and preprocessed in Excel. Later, they were analyzed with Python. AC2 was computed using Fergadis [97] Python port of Gwet’s [98] irrCAC package for calculating chance-corrected agreement coefficients. Gwet’s AC2 scores can be interpreted using commonly utilized benchmarks, as for example Altman [99] proposed. Gwet’s AC2 has a minimum value of −1 (systematic disagreement among raters) and a maximum value of 1 (perfect agreement among raters). Perfect agreement means that there is complete consistency among raters with no variability due to chance. A value of 0 indicates no agreement beyond what would be expected by chance. In other words, the observed agreement is no better than what would be achieved randomly. Every value between 0 and 1 indicates agreement beyond chance, but less than perfect agreement. The closer the value is to 1, the higher the agreement among raters. Altman [99] suggests the following interpretation: values < 0.2 represent a slight agreement. If the value is between 0.20 ≤ AC2 < 0.4, there is fair agreement. For values of 0.40 ≤ AC2 < 0.6, there is moderate agreement; for 0.60 ≤ AC2 < 0.8, there is good agreement and AC2 values ≥ 0.80 indicate very good agreement. However, to interpret the AC2 coefficient the standard error was considered, as proposed by Gwet [100]. All data and results can be viewed in a Streamlit Data App (https://optis-irr.streamlit.app/).

Intra-observer reliability.

When using the five-point rating scale, the overall coefficient for Gwet’s AC2 across both observers was 0.898. With regard to the different observation categories, the AC2 coefficient varied between 0.514 and 1. For observations using the six-point rating scale, the overall AC2 coefficient across both observers was 0.912. For the different observation categories, the AC2 coefficient varied between 0.744 and 1. For the two lessons both observers rated twice, the AC2 coefficient for the first observer was 0.879 and for the second observer 0.909, using the five-point rating scale.

Inter-observer reliability.

Test phase: The inter-observer reliability was computed at several times during the developmental process of OPTIS. During the test phase, the AC2 coefficient across the 36 observation categories varied between 0.105 and 0.974, with an average of 0.595. During this test phase, the teaching principle cooperative learning achieved poor agreement (0.105) beyond what would be expected by chance alone. The teaching principles differentiation (0.456/0.410) and independence (0.445) as well as the observation category the teaching process is based on recognizable rules (0.407) achieved moderate agreement between the raters. All the other observation categories achieved AC2 coefficients above 0.5 during the test phase, which indicated moderate to good agreement.

Piloting: During the video-based piloting, the observation protocol consisted of 39 observation categories, which represented the final version of OPTIS (Table in S2 Protocol). For the five-point rating scale, the AC2 ranged between 0.772 and 1, with an overall AC2 coefficient across all categories of 0.937. For the six-point rating scale, the AC2 values were between 0.791 and 1 with an overall AC2 coefficient of 0.929. For the piloting based on real-time observations, the overall coefficient for the five-point rating scale was 0.903 (0.511–1) and for the six-point rating scale 0.932 (0.801–1) (see Table 2). The values for the different teaching principles with their observation categories are documented in Table 3.

Content validity: To guarantee content validity, OPTIS was developed based on commonly mentioned teaching principles, added with teaching principles to reflect the constructivist learning theory (see Table in S1 Table/Fig 1). Furthermore, OPTIS was continuously discussed with experts during the whole development process. The chosen teaching principles and their observation categories underwent repeated evaluation, drawing upon theoretical foundations, empirical evidence, and the expertise of the involved experts. Due to these discussions, the content validity of OPTIS could be improved continuously. Final adjustments were made due to the observations during the field tests and video-recorded teaching. Finally, the content validity of OPTIS is considered given based on the experts’ feedback, as well as the evaluation and the adjustments made throughout the process (e.g., [77,82] 7).