https://orcid.org/0000-0002-2526-8682
Figures
Abstract
Background
Generalizability theory is widely used in psychological and educational measurement. Budget and cost are the problems that cannot be neglected in the measurement. When there is a budget constraint, the generalizability theory needs to consider how to design a measurement program with relatively high reliability and feasibility, which requires the optimal sample size to be estimated by some means. Lagrange multiplier method is a commonly used method for estimating the optimal sample size under budget constraints in generalizability theory. Unfortunately, to date, many formulas of estimating the optimal sample size for some more complex generalizability designs such as those with four facets or more facets have not been derived using the Lagrange multiplier method.
Purpose
The purpose of this article is to provide a detailed step-by-step derivation of the formula of estimating the optimal sample size for three typical complex generalizability designs using the Lagrange multiplier method under budget constraints in generalizability theory, which can demonstrate the new roles of the Lagrange multiplier method.
Method
This article derived the optimal sample size for teaching ability evaluation of college teachers with budget constraints in three generalizability designs such as the (s:t)×i, (s:t)× (i:v) and (s:t) × (i:v) ×o and explored their practical applications. By estimating the optimal sample size, the optimal generalizability design, which is more applicable in practice, can be compared to obtain.
Findings
(1) Using the Lagrange multiplier method, the optimal sample size for students and items under budget constraints in different generalizability design can be derived. (2) For an example, based on teaching ability evaluation of college teachers in China, these designs of (s:t) ×i, (s:t) × (i:v) and (s:t) × (i:v) ×o were used to obtain the optimal sample size, which indicates the Lagrange multiplier method can been used in practice. (3) Under budget constraints, the (s:t) × (i:v) is the optimal generalizability design. The optimal sample size of students is 17 for each teacher and the optimal sample size of items is 4 for each dimension.
Citation: Li G (2024) New roles of Lagrange multiplier method in generalizability theory: Inference of estimating the optimal sample size for teaching ability evaluation of college teachers. PLoS ONE 19(10): e0307710. https://doi.org/10.1371/journal.pone.0307710
Editor: Jianchao Bai, Northwestern Polytechnical University, CHINA
Received: September 29, 2023; Accepted: July 9, 2024; Published: October 17, 2024
Copyright: © 2024 Guangming Li. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: This research was funded by the Philosophy and Social Sciences Planning Project of Guangdong Province in China (GD24XXL03) and Natural Science Foundation of Guangdong Province in China (2021A1515012516) and was funded by Striving for the First-Class, Improving Weak Links and Highlighting Features (SIH) Key Discipline for Psychology in South China Normal University.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Generalizability Theory is a modern measurement theory widely used in psychological and educational practice [1, 2]. A larger generalizability coefficient often indicates higher reliability of the test. Generalizability coefficients can be improved by increasing sample size [3]. However, the sample size is subject to budget constraints. For example, the sample size may be limited by budget factors such as manpower, material resources, and financial resources, and cannot be infinitely increased [4]. Given limited conditions, increasing large level number of facets of measurement may only result in a slight change for generalizability coefficient. Therefore, how to find the balance between budget and reliability, as well as how to obtain the optimal generalizability coefficient under overall budget constraints have become an important problem that researchers need to consider. Fortunately, this problem can be transformed into the problem of estimating optimal sample size [5]. Therefore, it is important to examine how to effectively determine the sample size considering the budget constraints.
For example, generally the more students a teacher is evaluated by, the higher reliability the evaluation has [6]. However, in the process of evaluating the teaching ability of college teachers, budget and cost need to be considered and cannot be ignored when conducting measurement research [7]. In fact, the more students evaluate teachers, the higher their costs will be. Therefore, in the evaluation process, it is necessary to consider how to design a relatively feasible and reliable measurement program under budget constraints [8]. However, the current teaching abilities evaluation of college teachers often overlook budget constraints, typically involving all most students in the process and potentially leading to unnecessary costs. The sample size of the subject population is an undeniable factor that affects the reliability of evaluation results [9]. Due to practical operational limitations, it is difficult to conduct comprehensive evaluations throughout the population. Most evaluations are conducted using sampling methods, and the sample size becomes an important factor affecting the effectiveness of evaluations.
The Lagrange multiplier method is commonly used to solve the optimization problem about estimating the optimal sample size with equality constraints by introducing the Lagrange multipliers (new scalar unknowns) [10]. Let us give the function of two variables z = f(a, b) and additional condition φ(a, b) = 0. In order to find the extreme point of z = f(a, b) under additional conditions, the Lagrange function F(a, b, λ) = f(a, b) − λφ(a, b) should be made. Let F(a, b, λ) for a, b and λ in the first partial derivation be equal to zero, that is:
By solving the above equation, a, b and λ can be got. Thus, the obtained (a, b) is just a possible extreme point of the function z = ƒ(a, b) under additional condition φ(a, b) = 0. If there is only one such extreme point, it can be directly determined that the obtained (a, b) is the extreme point from the actual problem.
Its geometric significance is to assume that the given objective function is f(a, b) with the constraint condition φ(a, b) = 0. The curve L is the family of contour lines whose constraint condition is φ(a, b) = 0 and objective function is f(a, b) = C. Under the condition that the partial derivative of f(a, b) and φ(a, b) is continuous, the possible extreme point of the objective function f(a, b) under constraint conditions φ(a, b) = 0 is M(a0, b0).
Because two curves must have a common normal line at the tangent point, the contour line of objective function at point M(a0, b0) has a normal vector , and the curve of constraint condition at point M(a0, b0) has a normal vector
. The
and
are parallel, i.e.
. That is to say, there are real number λ which makes the following equation hold true:
.
It should be noted that, the tangent point between the contour line of objective function and the curve of constraint condition may not necessarily be extreme point of the objective function f(a, b) under constraint condition φ(a, b) = 0.
According to the Lagrange multiplier method, the unified formula of Lagrange function can be formulated as follows [11]:
(1)
For example, according to Eq (1), if s is used as evaluation student, i is used as the evaluation items, v is used as the dimensions of evaluation items, and o is used as the evaluation occasions (or number of times), the Lagrange function can be expressed as [12]:
(2)
Where ns represents the number of evaluation students, ni is the number of evaluation items, nv is the number of the dimensions of evaluation items, and no is the number of evaluation occasions. According to Eq (2), the optimal sample size of all designs can be estimated.
The Lagrange multiplier method proposed by Marcoulide and Goldstein [13] provides a new perspective for further solving the problem of estimating the optimal sample size in multidimensional design under budget constraints. Later, some scholars extended the Lagrange multiplier method to the more complex situation [7, 14, 15]. However, the research of these scholars is to divide and conquer the Lagrange multiplier method based on different generalizability designs. The focus of research is mostly on simple crossed design instead of expanding the research focus to more complex mixed design (mixed design includes both crossed design and nested design). However, in fact there are a number of generalizability designs that belong to multi-faceted mixed design. This has led to some shortcomings in current research when discussing how to estimate the optimal sample size under budget constraints. Based on existing literature, estimating the optimal sample size under budget constraints in generalizability theory can be improved from the following two aspects.
On the one hand, so far, the unified formula of Lagrange multiplier method has not yet been derived. Estimating the optimal sample size under budget constraints in generalizability theory is affected by factors such as the rounding of the total budget, the negative variance components, the fixed facet, the unbalanced design, and the optimal generalizability design [12]. In some generalizability designs, the estimated variance component may be negative, which may be due to issues with the data itself or may be some problems with parameter estimation methods, such as maximum likelihood estimation [16]. Generally, generalizability design is a design with random facets. However, in some cases, if only a certain number of facet levels are required, or if certain facet levels are immutable, this design becomes a design with fixed facets. At present, most studies have not further systematically summarized the basic principles of Lagrange multiplier method, but still based on the "divide and conquer" of different generalizability designs, nor has a relatively unified formula of Lagrange multiplier method been proposed to overcome the problems caused by these factors [7, 14, 15, 17–19]. Therefore, it is necessary to build the unified formula of Lagrange multiplier method and derive the equations if the optimal sample size for different generalizability designs, which can be remedied by subsequent research.
On the other hand, the Lagrange multiplier method is not applied to the more complex designs of generalizability theory. On the basis of previous studies, Meyer et al. [7] applied the Lagrange multiplier method previously proposed by Marcoulides and Goldstein [13, 20, 21] to estimate the optimal sample size for some generalizability designs under budget constraints, which is of great significance and shows that the Lagrange multiplier method has wide applicability. However, there are two issues for the generalizability design of Meyer et al. [7]: (1) there is no detailed derivation process for estimating the optimal sample size in different generalizability designs; (2) estimating the optimal sample size for the generalizability design of four facets or more facets has never been discussed. Comparing with previous similar studies, Meyer et al. [7] did not apply the Lagrange multiplier method to the more complex generalizability designs. Specifically, there has been little discussion on the some complex generalizability design with four facets or more facets [22–24], which may become one of the hot topics that can be further explored in depth in the future.
The purpose of this paper is to derive the equations of estimating optimal sample size under budget constraints for some more complex generalizability designs by using the unified formula of Lagrange multiplier method in generalizability theory, which can provide reference for subsequent practical applications.
2 Lagrange multiplier method
2.1 The Lagrange multiplier method of the (s:t) ×i design
Often, college students are required to evaluate the teaching ability of teachers on the same items, but the students taught by each teacher may be different. Therefore, this design is (s:t) ×i. Among them, t represents the teachers, i represents the items, s represents the students. The mean error variance and relative error variance of the (s:t) ×i design are [4]:
(3)
(4)
In Eq (3), represents the mean error variance;
,
,
,
,
are the variance components of the teachers; the variance component of the items; the variance component of the students nested in the teachers; the variance component of the teacher-item interaction; and the variance component of the students crossover the items, but nested in the teachers with unmeasured and random sources of variation respectively. In this paper, the symbol e was used to denote unmeasured and random sources of variation [25]. The nt, ni and ns represent the sample size of the teachers, the sample size of the items, and the sample size of the students respectively. In Eq (4), σ2(δ) represents the relative error variance, and the meanings of other representation symbols are the same as in Eq (3).
The optimal number of students and items can be obtained by conducting min F(ns, ni, λ) = σ2(δ) − λ(cnsni − B) using the unified formula of Lagrange multiplier method of Eq (2) with Eqs (3) and (4). The specific process is as follows:
(5)
(6)
(7)
Let ,
,
are equal to 0:
Let (5) = 0, and get:
(8)
Let (6) = 0, and get:
(9)
Let (7) = 0, and get:
(10)
Let (8) = (9), eliminate λ, and get:
(11)
Jointly established by (10) and (11):
In Eq (12), ns denotes the optimal number of students for (s:t) × i design. In Eq (13), ni represents the optimal number of items for (s:t) ×i design.
2.2 The Lagrange multiplier method of the (s:t) × (i:v) design
Compared with the (s:t) × i design, the (s:t) × (i:v) design adds some dimensions of evaluation items such as teaching methods, teaching content, teaching attitudes, teaching organizations, and teaching effects, etc. The items in each dimension are different. The v represents the dimensions of evaluation items. The mean error variance and relative error variance of the (s:t) × (i:v) design [4] are:
(14)
(15)
In Eq (14), represents the mean error variance;
,
,
,
,
,
,
,
are the variance components of the teachers; the variance component of the dimensions; the variance component of the items nested in the dimensions; the variance component of the students nested in the teachers; the variance component of the students crossover the dimensions, but nested in the teachers; the variance component of the teacher-dimension interaction; the variance component of the teachers crossover the items, but nested in the dimensions; and the variance component of the students crossover the items, but nested in the teachers and dimensions with unmeasured and random sources of variation respectively. The nt, ni, ns and nv represent the sample size of the teachers, the sample size of the items, the sample size of the students, and the sample size of the dimensions respectively. In Eq (15), σ2(δ) represents the relative error variance, and the meanings of other representation symbols are the same as in Eq (14).
The optimal number of students and items of the (s:t) × (i:v) design can be obtained by conducting min F(ns, ni, nv, λ) = σ2(δ) − λ(cnsninv − B) using the unified formula of Lagrange multiplier method of Eq (2) with Eqs (14) and (15).
Let ,
,
are equal to 0:
Let (16) = 0:
and get:
(19)
Let (17) = 0:
and get:
(20)
Let (18) = 0, and get:
(21)
Let (19) = (20), eliminate λ, and get:
(22)
Jointly established by (21) and (22):
In Eq (23), ns denotes the optimal number of students for (s:t) × (i:v) design. In Eq (24), ni represents the optimal number of items for (s:t) × (i:v) design.
2.3 The Lagrange multiplier method of the (s:t) × (i:v) ×o design
Compared with the (s:t) × (i:v) design, the (s:t) × (i:v) ×o design indicates an increase in the number of evaluations, such as twice a year, so the occasions are two, o represents the occasions. The mean error variance and relative error variance of the (s:t) × (i:v) ×o design [4] are:
(25)
(26)
In Eq (25), means the mean error variance;
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
represent the variance component of the occasions; the variance component of the teachers; the variance component of the dimensions; the variance component of the students nested in the teachers; the variance component of the items nested in the dimensions; the variance component of the interaction between the teachers and occasions; the variance component of the interaction between the occasions and students but nested in the teachers; the variance component of the interaction between the occasions and dimensions; the variance component of the interaction between the occasions and items, but nested in the dimensions; the variance component of the interaction between the teachers and dimensions; the variance component of the interaction between the teachers and items, but nested in the dimensions; the variance component of the students crossover the dimensions, but nested in the teachers; the variance component of the interaction between the students and the items, but nested in the teachers crossover the dimensions; the variance component of the interaction of occasion-teacher-dimension; the variance component of the interaction of occasion-teacher-item, but nested in the dimensions; the variance component of the interaction of occasion-student-dimension, but nested in the teachers crossover the dimensions with unmeasured and random sources of variation respectively. The nt, ni, ns, nv and no are the sample size of teachers, the sample size of items, the sample size of students, the sample size of dimensions, and the sample size of occasions respectively. In Eq (26), σ2(δ) represents the relative error variance, and the meanings of other representation symbols are the same as in Eq (25).
The optimal number of students and items can be obtained by conducting min F(ns, ni, nv, no, λ) = σ2(δ) − λ(cnsninvno − B) using the unified formula of Lagrange multiplier method of Eq (2) with Eqs (25) and (26).
Let ,
,
are equal to 0:
Let (27) = 0, and get:
(30)
Let (28) = 0, and get:
(31)
Let (29) = 0, and get:
(32)
Let (30) = (31), eliminate λ, and get:
(33)
Jointly established by (32) and (33):
In Eq (34), ns denotes the optimal number of students for (s:t) × (i:v) ×o design. In Eq (35), ni represents the optimal number of items for (s:t) × (i:v) ×o design.
2.4 Operation steps of estimating the optimal sample size
The operation steps of estimating the optimal sample size for the Lagrange multiplier method in this article are as follows:
- Step 1: In order to calculate the optimal sample size for each design, these optimal sample equations (ns, ni) need be derived according to different generalizability designs using the Lagrange multiplier method as above.
- Step 2: After deriving the optimal sample Equations (ns, ni) for each design, the corresponding variance components needed be calculated by urGENOVA software [26].
- Step 3: After obtaining the calculated variance component and providing other statistics such as n, c and B, according to step 1 and step 2 the optimal sample of ns and ni can be got.
- Step 4: However, some results are non-integer values with decimals, which is unreasonable. Therefore, those results with decimals need to be rounded.
3 Empirical study
3.1 Data source
The analysis data used came from teaching ability evaluation of college teachers. The data was obtained through a questionnaire survey, which was conducted in two semesters. The first test was conducted at the end of the first semester (December in 2022) and the second test was conducted at the beginning of the second semester (March in 2023). The participants in this test were a total of 530 students from three colleges in Guangdong Province in China. The students who participated in the test evaluated their teachers (a total of 19 teachers). The Teachers’ Teaching Level Evaluation Scale for Colleges (TTLES-C) [27] was used to evaluate the teaching ability of college teachers. The TTLES-C consists of 25 items, divided into 5 dimensions, with each 5 items being a dimension. These dimensions include teaching methods, teaching content, teaching attitudes, teaching organizations, and teaching effects.
Before the formal investigation, with the written consent of the students, anonymity was also processed, which meet ethical requirements. Training was provided to the students on how to answer questions before formal surveys. Informed consent for behavioral experiment was given every student and teacher. The participants face to face were told that each of them needed to verify their approval of the questionnaire survey. Informed consent was obtained from all students and teachers. There were no obstacles to their participation because all the students and teachers are willing to participate. This study was approved by the South China Normal University (SCNU) research ethics board (Institutional Review Board) who approved the experiments, including any relevant details and confirmed that all experiments were performed in accordance with relevant guidelines and regulations. Research staffs were trained before they administered the survey. Data were collected during the class in 45 minutes using a survey administered to all students.
Some generalizability designs were considered such as the (s:t) ×i design, (s:t) × (i:v) design and (s:t) × (i:v) ×o design. Among them, t is the object of measurement, and i, s, v, and o are the facets of measurement. As mentioned above, in this article t represents the teachers, i represents the items, s represents the students, v represents the dimensions of the items, and o represents the occasions. The main purpose of this study is to present the application of Lagrange multiplier method to solve the optimal sample size of different generalizability designs under budget constraints. According to the operation steps of estimating the optimal sample size for the Lagrange multiplier method, the optimal sample size for the (s:t) ×i, (s:t) × (i:v), and (s:t) × (i:v) ×o design can be estimated respectively.
3.2 Estimating the optimal sample size for the (s:t) ×i design
In the design of teaching ability evaluation of college teachers, the number of students being evaluated need be considered for generalizability design, but the students (s) are nested in the teachers (t) and crossed with the items (i), which is the (s:t) × i design. The estimated variance components for the (s:t) ×i design are shown in Table 1.
According to Eqs (12) and (13), it can be calculated as ns = 262.97048, ni = 23.76693. In practical operation, the number of students ns and the number of items ni should be integers, and the calculated results of ns and ni should be rounded to the nearest 263 and 24. In the (s:t) ×i design, when ns = 263 and ni = 24, the relative error value of the design is σ2(δ) = 0.00151, and the optimal generalizability coefficient is Eρ2 = 0.98650. In the (s:t) ×i design, 263 students are selected, meaning that each teacher is evaluated by approximately 14 students. The item is set to 24 items. The optimal generalizability coefficient can be obtained, which means that the relative error variance of the measurement is the smallest and the reliability is the highest.
3.3 Estimating the optimal sample size for the (s:t) × (i:v) design
When designing tests, if the impact of item dimensions on test results is considered, dimensional analysis of tests can make the analysis results more specific and comprehensive [28]. In the process of teaching ability evaluation, the student (s) is nested within the teacher (t), and the item (i) is nested within the dimension (v), which is the (s:t) × (i:v) design. The estimated variance components for the (s:t) × (i:v) design are shown in Table 2.
According to Eqs (23) and (24), assuming nv = 5, it can be calculated as ns = 329.83942, ni = 3.78972. In the process of teaching ability evaluation, the number of students ns and the number of items ni should be integers, so the calculated results of ns and ni should be rounded to 330 and 4. In the (s:t) × (i:v) design, when ns = 330 and ni = 4, the relative error variation of the design is σ2(δ) = 0.00325, and the optimal generalizability coefficient is Eρ2 = 0.97293. In the (s:t) × (i:v) design, when the dimension of the items is set to 5, the number of students is 330, which means that each teacher is evaluated by approximately 17 students. The item is set to 20 items, which means that each dimension of the evaluation items is set to approximately 4 items.
3.4 Estimating the optimal sample size for the (s:t) × (i:v) ×o design
In the classical test theory, retest reliability is one of the important indicators for the consistency of test results. As the name suggests, retesting refers to the application of the same test method to test the same group of subjects twice, and the reliability of retesting reflects the stability and consistency of the test over time [16, 25, 29]. When using generalizability theory for measurement research, the number of tests is also an important factor to consider. In the teaching ability evaluation of college teachers, in addition to considering the size of the test population, the number of evaluation items, and the dimensions of evaluation items, whether to use the same test items for a second test is also one of the issues that researchers need to weigh. To some extent, the presentation of retest results can make the test results more convincing. However, an increase in the number of tests means an increase in test costs [30]. Taking student evaluation as an example, in the evaluation process, evaluation at multiple time points or in multiple occasions can be included in the generalizability design, which means retesting. The evaluation student (s) is nested in the teacher (t), and the item i is nested in the dimension (v), the two are crossed with the occasion (o), which is the (s:t) × (i:v) ×o) design. The estimated variance components for the (s:t) × (i:v) design are shown in Table 3.
According to Eqs (34) and (35), assuming nv = 5, no = 2, it can be calculated as ns = 183.35696, ni = 3.40865. In practical operation, the number of students ns and the number of items ni should be integers, and the results of ns and ni must be rounded to 183 and 3. In the (s:t) × (i:v) ×o design, when ns = 183 and ni = 3, the relative error variance of the design is σ2(δ) = 0.01640, and the optimal generalizability coefficient is Eρ2 = 0.83501. In the (s:t) × (i:v) ×o design, when the dimension of the evaluation items is set to nv = 5 and no = 2, the number of students is 183, which means that each teacher is evaluated by approximately 10 students. The item is set to 15 items, which means that each dimension of the evaluation items is set to approximately 3 items.
3.5 The optimal generalizability design
In this paper, The Lagrange multiplier method is applied to relatively more complex mixed generalizability design that includes both crossed design and nested design. The relatively more complex mixed generalizability design with four facets such as the (s:t) × (i:v) ×o design extend these researches of Marcoulides [14, 18] and Meyer et al. [7]. The research results of three generalizability designs that presented in this article are shown in Table 4.
From Table 4, the optimal generalizability coefficient for the (s:t) × i design is 0.98650; the optimal generalizability coefficient for the (s:t) × (i:v) design is 0.97293; the optimal generalizability coefficient for the (s:t) × (i:v) ×o design is 0.83501. Although the (s:t) × (i:v) ×o involves the other occasion (o) facet, but the optimal generalizability coefficient is the lowest among the three designs. On the premise of having a high generalizability coefficient, the (s:t) × i design and (s:t) × (i:v) design both exceed 0.90 which is often considered as the cut-off value for a reliability indicator. But the optimal generalizability coefficient of (s:t) × (i:v) ×o design is lower than 0.90. Therefore, it can be seen from Table 4 that the results of (s:t) ×i and (s:t) × (i:v) seem more reasonable.
In Table 4, represents the number of items included in each dimension and
represents the average number of students evaluating each teacher. The
and
in the (s:t) × i design are 5 and 14; The
and
in the (s:t) × (i:v) design are 4 and 17; The
and
in the (s:t) × (i:v) ×o design are 3 and 10. If the actual situation is considered, the more items and students there are, the more reliable the estimated reliability may be. Therefore, undoubtedly the (s:t) × i design and (s:t) × (i:v) design are more reasonable.
Comparing with the (s:t) × i design, the (s:t) × (i:v) design examined more factors. For the (s:t) × (i:v) design, more consideration was given to the factor of dimension v. The (s:t) × (i:v) design not only considers evaluation students and evaluation items, but also considers the dimensions of the evaluation items, allowing for a more comprehensive analysis of the results.
In this paper, actual cost is cnsni, c = 0.4. According to Table 4, the cost required for the (s:t) × i design is 2524.8 yuan (0.4×263×24), while the cost required for the (s:t) × (i:v) design is 2640 yuan (0.4×330×20), and the costs of the two designs are relatively similar and the budget cost is relatively reasonable. If the design has a high generalizability coefficient and can consider more facets, the (s:t) × (i:v) design seems to be more persuasive. From the perspective of generalizability design, this article believes that the (s:t) × (i:v) design belongs to the optimal generalizability design with budget constraints.
3.6 The originality and limitations of this study
The originality of this study is as follows: (1) This study firstly discussed that how is the unified formula of Lagrange multiplier method derived very meticulously and step-by-step, and how is the Lagrange multiplier method applied to estimate the optimal sample size under budget constraints for a relatively complex design with four facets in generalizability theory. (2) By conducting generalizability research and estimating the optimal sample size under budget constraints, the optimal generalizability design can be inferred effectively. Under budget constraints, taking into account the actual situation, the (s:t) × (i:v) design belongs to the optimal generalizability design.
For this paper, there are some following shortages: (1) Only three designs were selected to derive the formulas, and there are still many designs that can be discussed in practice. (2) As an example, only teaching ability evaluation of college teachers was selected. In fact, there are many examples that can also be used in the derived formulas. However, unfortunately those other examples were not discussed in this article. People can continue to study these issues in the future. (3) The Lagrange multiplier method has a long history and has many applications in obtaining numerical solutions for nonsmooth constrained optimization problems. This-type of method also has certain relationships with other first-order methods such as augmented Lagrangian method, proximal point method, alternating direction method of multipliers [31, 32]. However, the author only recalled the traditional constructing method, while the advanced constructing methods are neglected. (4) Although the title mentions the Lagrange multiplier method, the paper only discusses the traditional Lagrange function. It is recommended that future work explore methods like the augmented Lagrange multiplier method and the alternating direction method of multipliers for the problem addressed in this paper. In future, it’s better to make a comparison of the results with other methods and further analysis.
4 Conclusions and implications
By using the Lagrange multiplier method and the estimated variance components of different generalizability designs, the optimal sample size can be derived to minimize the relative error variation and maximize the generalizability coefficient under budget constraints. The Lagrange multiplier method can be effectively applied to the different generalizability designs under budget constraints, which indicates that the Lagrange multiplier method has new roles. According to empirical study, for some examples through three designs in teaching ability evaluation of college teachers, the equation for calculating the optimal sample size is derived using the Lagrange multiplier method.
By conducting generalizability research and estimating the optimal sample size under budget constraints, the optimal design generalizability under budget constraints can be inferred. As an example, in the teaching ability evaluation of college teachers in China, comparing with the (s:t) ×i design and (s:t) × (i:v) ×o design, the (s:t) × (i:v) design belongs to the optimal generalizability design under budget constraints if taking into account the actual situation. As for the (s:t) × (i:v) design, the optimal sample size of students is 17 for each teacher and the optimal sample size of items is 4 for each dimension by using the derived formula of the optimal sample size.
Supporting information
S1 File. This file includes (s: t)×i.doc, (s: t)×i_output.doc, (s: t)×(i: v).doc, (s: t)×(i: v) _output.doc, (s: t) ×(i: v)×o.doc, (s: t)×(i: v)×o_output.doc.
For (s:t) ×i design, the (s: t)×i.doc is the program file in which contains data and the (s: t)×i_output.doc is result file. For (s:t) × (i:v) design, the (s: t)×(i: v).doc is the program file in which contains data and the (s: t) ×(i: v) _output.doc is result file. For (s:t) × (i:v) ×o design, the (s: t) ×(i: v)×o.doc is the program file in which contains data and the (s: t)×(i: v)×o_output.doc is result file.
https://doi.org/10.1371/journal.pone.0307710.s001
(ZIP)
References
- 1. Truong Q. C., Choo C., Numbers K., Merkin A. G., Brodaty H., Kochan N. A., et al. (2021). Applying generalizability theory to examine assessments of subjective cognitive complaints: Whose reports should we rely on-participant versus informant? International Psychogeriatrics, 4, 1–11. pmid:33847260
- 2. Wu M. Y., Steinkrauss R., & Lowie W. (2023). The reliability of single task assessment in longitudinal L2 writing research. Journal of Second Language Writing, 59: 100950.
- 3. Brennan R. L., Kim S. Y., & Lee W. C. (2022). Extended multivariate generalizability theory with complex design structures. Educational and Psychological Measurement, 82(4) 617–642.
- 4. Li G. (2023). How many students and items are optimal for teaching level evaluation of college teachers? Evidence from generalizability theory and Lagrange multiplier. Sustainability, 15: 2.
- 5.
Suh Y.S., Hwang D., Quan M., & Lee G. (2016). Optimizing the costs and GT based reliabilities of large-scale performance assessments. In: van der Ark L., Bolt D., Wang WC., Douglas J., Wiberg M. (eds) Quantitative Psychology Research,173–785. Springer Proceedings in Mathematics & Statistics, vol 167. Springer, Cham.
- 6. Lakes K. D. (2013). Restricted sample variance reduces generalizability. Psychological Assessment, 25(2), 643–650.
- 7. Meyer P. J., Liu X., & Mashburn A. J. (2014). A practical solution to optimizing the reliability of teaching observation measures under budget constraints. Educational and Psychological Measurement, 74(2), 280–291.
- 8. Goldstein Z., & Marcoulides G. A. (1991). Maximizing the coefficient of generalizability in decision studies. Educational and Psychological Measurement, 51(1), 79–88.
- 9. Hill H. C., Charalambous C. Y., & Kraft M. A. (2012). When rater reliability is not enough: Teaching observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64.
- 10.
Wang Z. M., & Wu K. H. (1999). Higher Mathematics (Third Editor). Nanchang, China: Jiangxi Higher School Publishing House.
- 11. Zheng F., & Gao X. (2018). A sufficient condition for conditional extreme value in using Lagrange multiplier method. Studies in College Mathematics, 21(2), 41–43.
- 12. Li G., & Qin Y. (2022). A new method for estimating the optimal sample size in generalizability theory based on evolutionary algorithm: Comparisons with three traditional methods. Acta Psychologica Sinica, 54(10), 1262–1276.
- 13. Macrolides G. A., & Goldstein Z. (1990). The optimization of generalizability studies with resource constraints. Educational and Psychological Measurement, 50(4), 761–768.
- 14. Macrolides G. A. (1993). Maximizing power in generalizability studies under budget constraints. Journal of Educational Statistics, 18(2), 197–206.
- 15. Macrolides G. A. (1997). Optimizing measurement designs with budget constraints: The variable cost case. Educational and Psychological Measurement, 57(5), 808–812.
- 16.
Brennan R. L. (2001). Generalizability theory. New York, NY: Springer.
- 17. Macrolides G. A., & Goldstein Z. (1991). Selecting the number of observations in multivariate measurement studies under budget constraints. Educational and Psychological Measurement, 51(3), 574–584.
- 18. Marcoulides G. A. (1994). Selecting weighting schemes in multivariate generalizability studies. Educational and Psychological Measurement, 54(1), 3–7.
- 19. Marcoulides G. A. (1995). Designing Measurement studies under budget constraints controlling error of measurement and power. Educational and Psychological Measurement, 55(3), 423–428.
- 20. Macrolides G. A., & Goldstein Z. (1991). Selecting weighting schemes in multivariate generalizability studies under budget constraints. Educational and Psychological Measurement, 51(3), 573–584.
- 21. Marcoulides G. A., & Goldstein Z. (1992). The optimization of multivariate generalizability studies with budget constraints. Educational and Psychological Measurement, 52(2), 301–308.
- 22. Gitomer D. H., Bell C. A., Qi Y., McCaffrey D. F., Hammer B. K., & Piñata R. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6), 1–32.
- 23. Spooren P., Mortelmans D., & Christiaens W. (2014). Assessing the validity and reliability of a quick scan for student’s evaluation of teaching. Results from confirmatory factor analysis and G Theory. Studies in Educational Evaluation, 43(11), 88–94.
- 24. Chen J. F., Hsieh H. N., & Do Q. H. (2015). Evaluating teaching performance based on fuzzy AHP and comprehensive evaluation approach. Applied Soft Computing, 28(C), 100–108.
- 25.
Shavelson R. J., & Webb N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
- 26.
Brennan, R. L. (2001). Manual for urGENOVA; Iowa Testing Programs, University of Iowa: Iowa City, IA, USA.
- 27. Li G., Hou G., Wang X., Yang D., Jian H., & Wang W. (2018). A multivariate generalizability theory approach to college students’ evaluation of teaching. Frontiers Psychology, 9, 1065. pmid:29997553
- 28. Clayson P. E., Carbine K. A., Baldwin S. A., Olsen J. A., & Larson M. J. (2021). Using generalizability theory and the erp reliability analysis (era) toolbox for assessing test-retest reliability of erp scores part 1: Algorithms, framework, and implementation. International Journal of Psychophysiology, 166, 174–187. pmid:33465427
- 29. Vispoel W. P., Xu G., & Kilinc M. (2020). Expanding G- Theory models to incorporate congeneric relationships: Illustrations using the big five inventory. Journal of Personality Assessment, 103(1), 429–442. pmid:32926640
- 30. Vispoel W. P., Lee H., Chen T., & Hong H. (2023). Extending applications of generalizability theory-based bifactor model designs. Psych, 5, 545–575.
- 31. Liu X., Dai Y., Huang Y., & Sun J. (2023). A novel augmented Lagrangian method of multipliers for optimization with general inequality constraints. Mathematics of Computation, 92, 1301–1330.
- 32. Bai J., Jia L., & Peng Z. (2024). A new insight on augmented Lagrangian method with applications in machine learning. Journal of Scientific Computing, 99: 53.