https://orcid.org/0000-0001-8900-2106
Figures
Abstract
Recent developments in the field of artificial intelligence and machine learning allow the wide application of large language models for the evaluation of written text and other non-numerical data. When applied in the context of psychological and educational assessments, such models can be used for assigning scores to essays and other types of responses. In contrast to classical tests, essays do not consist of test items, which leads to specific challenges in the evaluation of testing standards for scores obtained from AI models that differ from those observed for classical ability tests and personality questionnaires. To address these challenges, we discuss the evaluation of validity, fairness, and reliability for scores obtained from models of artificial intelligence in the context of automated essay scoring. We review existing methods, propose new methods, and further illustrate the reviewed methods with an empirical example based on the Hewlett Foundation data set on automated essay scoring. By applying the proposed framework to an evaluation based on a DistilBERT model, we find the model to be robust with sufficiently high internal consistency (Spearman-Brown coefficients in the range from .77 to .92). We further found empirical evidence for the validity of the evaluation model, but also indications for violations of fairness when comparing the human and AI scores across different topics. This study provides a standardized, replicable toolkit for researchers and practitioners to evaluate the psychometric quality of AI-based assessments.
Citation: Debelak R, Ziegler M (2026) Testing standards for AI-based scores in automated essay scoring. PLoS One 21(7): e0354680. https://doi.org/10.1371/journal.pone.0354680
Editor: Iftikhar Ahmed Khan, University of Lahore - Raiwind Road Campus: The University of Lahore, PAKISTAN
Received: October 4, 2025; Accepted: July 7, 2026; Published: July 31, 2026
Copyright: © 2026 Debelak, Ziegler. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The raw data and program code (including Python scripts and Jupyter notebooks) underlying the results presented in this study are available in the Open Science Framework (OSF) repository: https://doi.org/10.17605/OSF.IO/NR8HE.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Recent developments in the field of artificial intelligence (AI), such as the wide availability of large language models (LLMs) via models such as OpenAI’s GPT-4 [1], GPT-4o [2] and Google’s Gemini [3] have led to numerous applications in the field of psychological and educational testing. Prominent examples include the automatic scoring of essays or the use of LLMs to generate and evaluate items for educational assessments [4–11]. This line of work is sometimes summarized as Computational Psychometrics or Educational Data Mining [12,13]. The integration of these tools in psychological and educational assessments is the topic of an ongoing debate [14–20] and also includes applications such as automated item generation or the use of AI models to provide feedback. In this context, it is still a point of discussion whether AI models understand the text they process [21], which leads to the necessity of a rigorous evaluation of their output. Despite the numerous methodological innovations of this field, educational assessments based on AI still have to meet professional psychometric standards as they are outlined, for instance, in the Standards for Educational and Psychological Testing [22] or the Guidelines for Technology-Based Assessment [23]. Rich literature discusses the psychometric evaluation of multi-item measurement (for example: [8,24]). This concerns in particular an evaluation of AI-based scores with regard to the reliability, validity and fairness of their interpretation. Scores obtained from AI models use a different methodological framework than traditional testing scores, e.g., natural language processing methods, which do not use trivial scoring functions, such as the sum of correct responses, or a statistical function such as a maximum likelihood estimation to obtain a score. In this aspect, their evaluation differs from frameworks such as classical test theory (CTT) [25], factor analysis (FA) or item response theory (IRT) [26]. Instead, AI models are usually compared based on quantitative metrics in various benchmarks and based on their underlying training data.
This paper wants to review and discuss methods for checking the reliability, validity and fairness of the interpretation of AI-based test scores. By “AI-based scores,” we refer to any numerical evaluation that is given by a machine learning model, such as an LLM, to a human response to a task. Examples for such AI-based scores include, for instance, numerical scores provided by models such as BERT [27] or GPT-4o [2], or scoring systems based on numerical linguistic features such as e-rater [28]. However, we do not consider, for instance, verbal feedback, as it can be provided by generative AI models such as GPT-4o, but briefly discuss possible approaches for evaluating verbal feedback near the end of this work. In the empirical sections below, we use the term “predictions” when referring specifically to the output of a trained model, which we treat as synonymous with AI-based scores in this context.
Besides reliability, validity, and fairness, many additional testing standards are proposed in the literature, such as the standardization and usefulness of a psychological or educational test. However, we focus here on these three standards for two reasons: a) Reliability, validity, and fairness are usually considered as central for the evaluation of tests, and test scores that fail to be reliable, valid, or fair are usually considered problematic. b) The evaluation of these three testing standards usually relies on statistical frameworks, such as CTT, FA, or IRT, which are usually not directly applicable to all aspects of the evaluation of AI-based scores. Although one can still apply simple statistical methods for checking these standards, e.g., the calculation of correlations between AI-based scores and human evaluations as a simple measure of evidence for convergent validity, we will propose additional methods to complement these classical methods.
Classical and AI-based scores
In the following, we distinguish between reliability as the consistency of scores across equivalent inputs and robustness as the stability of scores under minor, construct-irrelevant perturbations.
AI-based scores, in the sense we have outlined above, differ in several important ways from the scoring of CTT or most latent variable models. First, the input of these models is usually not considered as a set of relatively independent units (i.e., items), but more sequential in character. Second, systems for obtaining AI-based scores, such as essay scoring systems based on transformer models and other types of neural networks, are establishing a relationship between the input text and their scoring, which is highly non-linear, typically includes complex interaction effects and is generally significantly more complex than the relationship established in models of IRT or FA.
A practical limitation of the discussed and proposed methods is that they assume that the predicted score is numerical or categorical in nature. In principle, LLMs can provide verbal feedback to texts and documents, including essays; as a simple illustration of this point, readers might use online interfaces to LLMs to ask these models for feedback on the spelling or the style of an arbitrary text. For such evaluations, the proposed methods cannot be applied, and future research might discuss the evaluation of assessment standards for validity, reliability, and fairness for such evaluations.
In summary, we aim to make the following contributions to the literature: a) We provide a systematic discussion of methods for evaluating reliability, validity and fairness for numerical test scores that were obtained from AI models. b) While previous work [29] discussed general frameworks, this paper contributes specific, testable methodologies. c) We illustrate the application of the proposed methods in an empirical example, which is based on automated essay scoring with modern Transformer models [30].
In the following Table 1, we provide a summary of the proposed evaluation steps, and how they relate to traditional psychometric standards. This operational testing protocol translates abstract psychometric concepts into concrete, testable procedures for AI models.
The rest of this paper is set up as follows: First, we provide an overview based on the standards of reliability, validity, and fairness, which is based on their definition in the AERA, APA, and NCME testing standards [22]. Each overview is followed by a review of existing and new methods on how these standards can be checked for AI-based scores. We then illustrate some of these methods using an empirical example and conclude with a discussion.
Reliability
The testing standard of reliability pertains to the precision of measurement, ensuring that scores do not contain large measurement errors [22]. See, for instance, Martinková and Hladká [8], for a general introduction to reliability.
Reliability of AI-based scores in traditional frameworks
Traditionally, reliability is assessed using coefficients like Cronbach’s or split-half correlations in CTT, or via the precision of parameter estimates (e.g., standard errors) in structural equation modeling and IRT [8]. All these concepts share the assumption that the chosen parametric modeling framework provides an accurate description of the data.
In the evaluation of automated essay scoring systems, classical frameworks can sometimes be applied by treating different facets of text quality as “items” to estimate an average rating’s reliability. A similar approach was chosen by Fan et al. [31] to determine the reliability of personality scores obtained by automated systems.
A related challenge stems from issues such as rater drift and halo effects that may affect individual raters. Those challenges are traditionally also faced by human raters. To quantify these effects, one can assess the agreement of various raters – which could include both AI models and human raters – by calculating the interrater reliability of various raters, which can be achieved, for instance, by calculating the intraclass coefficient [32]. A review and discussion of intraclass coefficients was provided by Liljequist et al. [33], Koo and Li [34] and others. Measurement frameworks such as IRT [26] can be used to correct for these effects. Also see Martinková et al. [35] for an overview on flexible modeling of heterogeneity in reliability estimates across groups.
As Attali [36] notes, such measures of interrater reliability are not suitable for assessing the reliability of AI-based systems themselves, since they do not compare the outcome of such systems under different conditions, such as different prompts. Instead, they aim at assessing to which extent AI-based scores can replace human ratings. As the same author writes, many early evaluations (e.g., [37]) of automated essay scoring systems aimed at comparing the agreement between those systems and human raters with the average agreement of human raters.
In summary, we can observe the following methods for assessing the reliability of automated essay systems using traditional methods:
- Treat different outputs based on text partitioning that are provided by automated essay scoring systems as “items,” and apply classical measures from CTT or IRT.
- Treat AI-based scores as additional ratings and compare them with those of human raters.
- If scores from multiple essays written by the same authors are available, one can consider test-retest reliability. However, this approach needs to consider that the writing quality of texts written by the same author can depend on the writing task, including the genre of the text [38].
Additional methods for assessing reliability
When evaluating the reliability of AI-generated scores, it is necessary to consider that such scores depend on characteristics of the input (i.e., the essay) as well as of the model (i.e., the rater). To evaluate this interaction, it is thus necessary to evaluate different parts of it. To this aim, we suggest several methods that aim at manipulating the input to learn about characteristics of the scores, and indirectly about the model as a rater.
As was already outlined, a simple concept for reliability is based on the thought experiment that test takers can take the same test twice, and that the resulting test score should be stable in such a thought experiment. In the context of LLMs and similar AI models, such a thought experiment can indeed be carried out, for instance, by giving the same prompt twice to an AI model and comparing the resulting output. However, it is usually possible to determine the fluctuation of the resulting output to the same input by setting suitable hyperparameters, such as the temperature or the seed parameter, and using suitable hardware. Using these technical settings, it is, therefore, usually possible to optimize this aspect of reliability under specific settings. In other settings, such as multi-cloud settings with variable hardware, AI models may produce non-deterministic outputs due to the inherent variability of the underlying infrastructure. In such cases, it is recommended to quantify the variability of the output by repeatedly providing the same prompt to the model and analyzing the distribution of its responses. This can be done by calculating the standard deviation or variance of the generated scores, or by visualizing the distribution of outputs. For the remainder of the discussion, we assume that our AI model is reliable in the sense that it will always provide the same output to the same input.
Another important aspect of reliability concerns the idea that reliability is measured by the precision of ability estimates. This idea is not directly applicable to AI-based scores such as numerical evaluations obtained in an automated essay scoring system. This has several reasons: First, AI-based scores are usually not based on parametric statistical theory, which prohibits a statistical estimation of measurement errors. Second, the input of such AI systems, such as written text, is of a different nature than the data used in CTT, structural equation modeling or IRT, which are usually responses coded as integer values. While we focus on automated essay scoring here, similar problems arise in other applications of AI in educational assessments, such as simulations or the evaluation of constructed response tasks. As alternatives, we suggest two methods for assessing the precision of an AI-based score that are based on similar ideas such as those outlined for the traditional frameworks.
Both proposed methods share the assumption that the AI model used for obtaining the score already provides predictions of satisfactory accuracy. This assumption can usually be tested using well-known approaches in AI, such as assessing the prediction accuracy in a test set. However, it is important to note in this context that accuracy should not be confused with reliability. Accuracy more closely resembles the notion of model fit in FA or IRT. We will demonstrate the calculation of prediction accuracy briefly in our example below. To evaluate the reliability, that is, the precision of an AI-based score, we check how much this score changes depending on (usually small) changes of the provided input which stems from the same respondent. This pertains to the following simple idea, which also underlies the reliability estimators in traditional frameworks: The observed input can be considered as a sample of a respondent’s behavior, and leads to an observed AI-based score that aims to assess the proficiency of a respondent to show a specific behavior, e.g., writing texts of high quality. If we assume that this proficiency is sufficiently stable – and this assumption is commonly made in psychological and educational assessments -, the AI-based score should remain relatively stable regardless of the provided input. This leads to the following two methods for assessing reliability for AI-based test scores: First, we can divide the provided input in several parts, and obtain scores for each part using our AI model. A measure that summarizes the stability of these scores over the different parts can thus be considered as a measure of the internal consistency of the scores provided by the AI model – it evaluates whether the model provides consistent scores across different sections of the same text. Variations of this method have already been proposed in the context of automated essay scoring systems. Attali et al. [36] discussed an analogous method in the context of automated scoring systems that are based on features (e.g., word length). Here, the reliability of these features was estimated by calculating them for the odd and even sentences of essays and calculating the stepped-up correlation between both features. A similar method was proposed by Fan et al. [31] in the context of personality assessment. Here, the reliability of the prediction was estimated by the correlation between the predictions of the first and second half of the text.
It is well-established that essay length can correlate with both human scores and scores generated by automated essay scoring systems (e.g., [39]), as will also be shown in the empirical illustration below. It needs to be expected that applying a split-half method leads to a drop in the scores for the shorter halves. The purpose of the proposed method is thus not to test for a correlation between essay length and overall score, but to assess the consistency of the scoring function across different parts of the text. It accounts for a possible effect of the essay length on the model score by comparing parts of comparable length.
Another possible disadvantage of the presented method for quantifying the internal consistency of a scoring method is that it might result in unrealistic texts. For instance, extracting every other sentence from a text can create overly artificial texts. As an illustration, the reader can imagine two texts that only consist of the odd and even sentences of this manuscript. Predictions based on such unrealistic texts may not be representative for the scoring task for which reliability should be estimated. In other contexts, such as the analysis of unordered word lists, this approach can be less problematic. As an anonymous reviewer noted, this approach could even account for changes in the text structure that might affect the text ratings. A closer investigation of this method is left as a topic for future research.
A second method is related to approaches that are used for testing the robustness of an AI model. In this method, the input of the AI model is slightly changed, for instance by inserting a few characters or spaces. While the first method aims at evaluating the internal consistency, this method aims at evaluating the robustness of the scores – it assesses whether the model’s score is robust against minor changes to the essay that should be insignificant. Conceptually, adding such additional information can be compared to adding unsystematic variance to items in a psychological or educational test. The stability against such changes can be evaluated by applying such changes multiple times. This second method is related to the idea of adversarial examples in deep learning [40], which aims at checking the robustness of model predictions against minuscule changes in the input, which are assumed to be unrelated to what the AI model is aimed to assess.
For instance, if we consider an essay scoring system that is aimed to assess the overall style or consistency of an essay, it makes sense to include small typos and evaluate their effect on the assessment. If we consider an essay scoring system that assesses the spelling of an essay, it might make sense to replace individual words by synonyms and evaluate the effect of this change on the rating. The empirical example below aims to assess the overall writing quality, as defined by the pooled judgment of human raters.
In general, the definition of which changes are minuscule or irrelevant must be informed by a theory of writing. In a model assessing overall writing quality, extra spaces or punctuation problems can be theorized to have a more minor impact compared to errors in structure or argumentation. This method aims to test whether the model’s scores and sensitivity align with such hypotheses and is related to broader efforts in stress-testing AES and assessing their vulnerability to ‘gaming’ behaviors, where more targeted and substantial manipulations are employed to test system integrity [41,42]. If, for instance, a few typos cause a drastic score change, the model may focus too strongly on such features, which could be considered as a threat to the validity of its scores. The sensitivity of a given model to such changes is, of course, dependent on the specific model architecture and its training data. The outcome of this analysis is thus not a universal estimate for reliability but a statement on the behavior of the scores of a specific model.
A variation of this method, which is available with generative AI models such as OpenAI’s GPT-4o or Google’s Gemini, is the inclusion of small changes in the prompt, that is, the instruction to the generative AI model, and the evaluation of their effect on the score. A similar method was proposed by Attali [36], who compared automated essay scoring models based on different prompts (here: writing instructions to students) to parallel test forms in CTT. Following this author’s reasoning, one might estimate the reliability of these parallel test forms via the Spearman-Brown formula, a classical approach of CTT. In summary, the first proposed method, aiming at evaluating internal consistency, consists of the following steps:
- Divide your input, e.g., text, in two or more parts of comparable length. Calculate the AI-based score for each part of the input.
- As an estimate of internal consistency, calculate a measure of stability between the scores for all parts. For instance, if we divide the input into two parts, we could calculate the correlation between the numerical scores over the complete sample as such a measure. For more than two parts, adaptations of reliability measures from CTT, such as the Spearman-Brown formula, could be an alternative approach. Speer et al. [43] note that the interpretation of this correlation as a measure of reliability requires the assumption that both test halves are parallel.
The second proposed method, aiming at evaluating the robustness of an AI model, can be summarized as follows:
- Make small changes to the input, for instance, by inserting typos. The number of changes might be fixed or varied based on the text length. As outlined above, it is crucial that these changes are unrelated to what the model aims to assess. In models of generative AI, such as GPT-4o or Gemini, one could alternatively vary the prompt that leads to a numerical evaluation of the essay.
- The outcomes of the different changes or different prompts are considered as parallel forms of a test. One can now estimate the reliability of the test by applying traditional measures from CTT, such as the Spearman-Brown prophecy formula. For instance, in the case of two forms, one can calculate the correlation between the two evaluation scores before and after the changes and apply the prophecy formula.
- As a further enhancement, repeat this procedure multiple times with different types of changes (e.g., inserting typos, replacing names with synonyms) and obtain an estimation of the typical change of the ratings (i.e., the average change and its variance) for each type of change.
Since the second method also requires a theory on which changes to the inputs or prompts are inconsequential, it also evaluates an aspect of construct validity. We will discuss this aspect later.
The second method can be further adapted by gradually inducing more changes, e.g., more typos, and evaluating their effect on the overall score. This enhanced procedure would lead to a range of values for measuring the robustness as a facet of reliability instead of a single number for each type of change. If the changes to the text are too extensive to expect no effect of the score, a similar method can be used for assessing the validity of the score. We discuss this topic next.
Summary of Reliability Evaluation: In summary, assessing the reliability of AI-based scores requires moving beyond traditional parametric error estimation. We operationalize AI reliability through two complementary approaches: evaluating internal consistency via split-half text partitioning, and assessing robustness by introducing minor, construct-irrelevant perturbations (e.g., minor typos) to ensure score stability.
Validity
Validity refers to the degree to which empirical evidence and theory support the intended interpretation of test scores ([22], see [8] for a practical application). Here, we focus on validity as the sensitivity of scores to meaningful differences and their alignment with the intended construct.
Validity of AI-based scores in traditional frameworks
Traditionally, validity evidence is gathered from multiple sources, including test content, response processes, internal test structure, relations to external variables, and the consequences of testing [22]. In traditional settings, this involves structural evaluations using latent variable models (e.g., FA or IRT) to confirm construct validity, or correlational analyses to establish convergent and discriminant validity [44,45].
When applied to AI-based scores, particularly automated essay scoring, some of these traditional forms of validity evidence require adaptation [31,36,46]. For example, evidence based on response processes translates to investigating whether an AI model’s ratings are driven by construct-relevant features (e.g., argumentative structure) rather than construct-irrelevant artifacts (e.g., superficial formatting). Similarly, evaluating the relation to external variables involves measuring the degree to which AI-based scores predict relevant outcomes or correlate with independent human expert ratings. Finally, while a full analysis of the intended and unintended consequences of testing [47] is outside the methodological focus of this paper, it remains an essential consideration for the responsible deployment of any AES system to ensure it does not systematically disadvantage specific test-takers [22]. Other forms of validity evidence, such as using correlations as part of the evidence for convergent and discriminant validity, can be applied as in a traditional test analysis.
Additional validity evidence
Some sources of validity evidence, in particular the response processes, are independent from the applied measurement framework. In the context of applications of AI models, one validity aspect of test content can be evaluated by checking the representativeness of the used training, validation and test data sets. Although this is not evidence for content validity in the traditional sense, the representativeness of the training data ensures that the model learns a grading function that is applicable in the entire target population. In the rest of this section, we therefore focus on validity evidence based on the internal structure of the test and the relation to variables external to the test.
Williamson et al. [29] suggested several numerical criteria that could be evaluated and compared, for instance: a) The agreement of AI scores with human ratings, b) the degradation from human-human agreement (i.e., the difference between human-AI agreement and interrater agreement), where such information is available, c) the standardized mean score difference between AI and human ratings. These evaluations correspond to evaluating the convergent validity of AI-based systems. Furthermore, they suggested the group-wise evaluation of external and within-test relationships, which corresponds to the idea of evaluating prediction invariance of the group level.
Using a similar perspective, we can propose additional methods for evaluating validity based on the methods outlined above. If AI models are used to predict external criteria, such as the rating of human raters for texts or pictures, the accuracy of this prediction can serve as evidence for a relation between the test score, as provided by the AI model, and an external criterion, such as human ratings. This idea enhances the classical criteria for convergent, discriminant and criterion validity outlined in the previous section. In computer science, this type of investigation is usually summarized as an evaluation of the model generalizability to new data by evaluating its predictive accuracy in training, validation and test data. In this context, usually two types of prediction models are distinguished: While regression models aim at predicting continuous variables, classification models aim at the prediction of categorical variables. Both model types differ in the metrics that are usually used for assessing the prediction accuracy. Usual metrics to quantify such a relation typically include the accuracy, the F1 score, the sensitivity, and specificity in classification models, as well as the root mean square error (RMSE) in regression models. To understand the F1 score, one can imagine a diagnostic test that results in a positive or negative outcome. In this scenario, “precision” indicates how often a positive outcome is truly correct (avoiding false positives). “Recall” indicates how many true cases were caught (avoiding false negatives). The F1 score is the harmonic mean of precision and recall; it is essentially a single number that rises only when both are good.
This approach for evaluating this aspect of validity is based on the simple idea that the AI score is treated as an outcome of the assessment whose relation to external variables can be directly investigated. This evaluation can be carried out in the training set, that is, in the data that were used for training the model, but also in a test set that was not used for training. In this approach, we treat the AI-based scores as any other score.
A related, but distinct question pertains to explaining how an AI model, for instance for essay scoring, obtained its prediction. This question is related to the second mentioned type of validity evidence, namely, whether the score of the AI models corresponds to its intended interpretation. Such tools are provided by the methods of explainable AI, or interpretable machine learning [48]. While an overview of all available methods is outside the scope of this paper, available methods from the field of natural language processing entail LIME values [49] and SHAP values [50]. For individual data points, these methods help to explain how changing individual features, such as tokens in the context of NLP models (e.g., words or word parts), affect the prediction of the model. For instance, if an AI rating aims to evaluate the spelling of a text, SHAP and LIME values should indicate which words or word parts in a text affect the spelling assessment. Ideally, they would mark all spelling errors, which should affect this assessment negatively. We do not illustrate this method in the empirical example, which is based on essay scoring, later in this text, since it is not clear for this example how specific tokens would relate to the overall rating of the text. In this sense, content validity could be limited in some use cases of essay scoring models. An example application can still be found in the Python code that accompanies this text.
As a variation of this idea, we can induce changes that are related to the trait the assessment aims to measure. For instance, when evaluating an essay scoring system that focuses on spelling, inserting typos should typically reduce the overall rating of an essay. A second aspect concerns the size of the observed changes. Insignificant changes should have insignificant effects on the rating, while significant changes should have significant effects in the desired direction. Therefore, the extent of the changes to the input texts and its expected effect on the ratings determines whether this method is used to assess the robustness, as a facet of reliability, or the validity of the AI-generated scores. A similar point related to construct overlap and the interpretation of correlations as reliability and validity is made by Speer et al. [43].
Optionally, one could plot the range of the observed score in a visualization that is reminiscent of confidence intervals. This approach of inducing changes is conceptually related to the ideas underlying SHAP values (cf. [48]).
Some methods of explainable AI can also be used to investigate the internal structure of AI models. Under the perspective of getting sources of validity evidence, these methods can be seen as belonging to the third type of validity evidence. For instance, software such as TensorBoard, which is part of TensorFlow [51], can help to visualize how information is processed in neural networks. These methods for visualizing the internal structure of neural networks can be considered as a potential future direction for validity research. They could be used to investigate whether individual neurons or neural layers specialize in detecting features that align with specific aspects of the scoring, such as scoring. This type of methodology is currently largely unexplored, therefore we focus on more readily available methods in this paper.
Another method that may help to explain the prediction of an NLP model is to check the scoring of (short) texts for which the true grading is known, but which are not part of a model’s training data. These might also include extreme cases; in the context of essay scoring, such a case might consist, for instance, of an essay with no language errors that is written on the wrong topic. A similar approach was evaluated by Ding et al. ([52]). This method aims at checking whether a given AI model shows undesired or unexpected predictions in specific cases. This method may also help to confirm whether the model applies heuristics that may lead to incorrect predictions in specific cases. Again, this evidence helps to understand the scoring of an AI model, and whether it agrees with the intended interpretation. In summary, the presented methods for checking the validity of AI-based scores can be summarized as follows:
- To assess the validity of an AI model, its predictions can be compared against external criteria. When the criterion is a continuous variable, such as human ratings, the correlation coefficient or the root mean squared error can be used to quantify the accuracy of the AI’s predictions. For categorical criteria, metrics like accuracy, sensitivity, specificity, and the F1 score are appropriate, especially when dealing with binary classifications.
- Use methods of interpretable machine learning and explainable AI, such as LIME and SHAP values, to investigate the relation between individual features and the model prediction. For instance, do such methods indicate correctly that words containing spelling errors decrease the overall quality of a text? Alternatively, investigate the effects of inducing small changes for which the expected effect on the evaluation is known. Using the same example, does inserting more spelling errors reduce the rating of the text?
- Investigate the behavior of specific neurons or specific layers as part of the investigation of the internal structure of an AI model. This method could, in theory, help detect whether individual neurons or layers specialize on aspects of essay scoring, but it is relatively unexplored.
- Evaluate predictions for artificial cases where the true evaluation is known. For instance, does the model detect correctly if a text contains flaws in its argumentation and structure, or does it rather focus on obvious errors in spelling, grammar and punctuation?
A further possible extension arises if ratings from multiple AI models and multiple raters are available. Here, one could follow a traditional approach by applying a multitrait-multimethod analysis (e.g., [53]) that investigates the pairwise correlations for raters and AI models, or a more advanced CTC(M-1) model [54].
Summary of Validity Evaluation: In summary, validating AI-based scores requires a multi-faceted approach. This includes comparing AI predictions against external criteria (e.g., human ratings) to establish convergent validity, using explainable AI (e.g., SHAP/LIME) to verify the relevance of modeled features, and stress-testing the model with targeted, construct-relevant text manipulations (e.g., degradation analysis) to ensure the model reacts appropriately to true quality loss.
Fairness
Fairness is not uniquely defined. Following the Standards for Educational and Psychological Testing ([22], p. 50), “a test that is fair … reflects the same construct(s) for all test takers, and scores from it have the same meaning for all individuals in the intended population; a fair test does not advantage or disadvantage some individuals because of characteristics irrelevant to the intended construct.” Such characteristics typically include gender, ethnicity, socioeconomic status, or cultural background [22]. Important aspects include the fairness with regard to the treatment during the testing process, the absence of measurement bias, the access to the measured constructs, and the validity of the intended test interpretations. Although not all of these can be evaluated using statistical measures, an important group of statistical evaluations is summarized under the term of machine-learning measurement bias [55] or the assessment of prediction invariance [56,57].
In practical evaluations of fairness, it is crucial to move beyond single-axis fairness analyses, since it is often not sufficient to investigate fairness for specific groups defined by only gender or only language. Instead, it is crucial to consider violations of fairness that affect individuals at the intersection of these groups, e.g., female Spanish-language learners.
Another important aspect of the evaluation of fairness is the discussion of individual differences. AI models must not only be fair at the level of groups and their intersections, but they must also account for individual differences that are not relevant to the constructs that are being assessed. For instance, an automated essay scoring system must account for differences in writing styles and cultural backgrounds.
It seems important to note that AI research is usually based on a related concept of fairness, which aims at comparing the accuracy of prediction of an AI model in various groups of interest. The basic idea is to define groups of interests and to compare the predictive accuracy between these groups. For a more detailed discussion of fairness issues in the context of machine learning, see, for instance, Barocas et al. [58] or, in the context of psychological assessments, Goretzko and Israel [59].
There are important aspects of fairness that need to be evaluated before the application of LLMs and other models of machine learning. The first aspect concerns the presence of biases within a pre-trained LLM [60], which was recently discussed in the context of essay scoring by Johnson and Zhang [61], and in the context of NLP models, by Hovy and Prabhumoye [62]. The second aspect concerns the question of whether the data used for training or fine-tuning a machine learning model can be considered as representative for the underlying population. As Schaller et al. [63] argue, training data that are not representative with regard to demographic background variables or cognitive ability can lead to biases or low accuracy in populations that are not represented in the training data.
Evaluating fairness
Statistical methods to detect violations of fairness typically entail methods for the detection of violations of measurement invariance and for the detection of predictive bias. Methods for detecting violations of measurement invariance could be useful in modeling approaches that use latent variable models for developing a joint model for the numerical ratings of human and AI-based raters. Statistically, this can be achieved by a model of IRT, e.g., a multi-facet Rasch model, where the various raters correspond to items, and the essays to persons [64–67]. Conceptually, the basic idea is that all raters aim to assess the same latent trait, but might differ in their strictness (corresponding to the item difficulty) and, depending on the used model, their discrimination and other psychometric characteristics.
While methods of measurement invariance aim at detecting differences in psychometric characteristics of the test items, methods for detecting predictive bias aim at differences in the regression model used for the prediction of external criteria [56,57]. In contrast to methods of measurement invariance, these methods can be used to compare the accuracy of an essay scoring model between various relevant groups of essays. Similar to the concept of group fairness ([68]; see [69] for a critique), they aim at checking whether there are groups of respondents that are systematically disadvantaged by a model. This approach was also advocated by Tay et al. [55], who compared it to measurement invariance. They suggested the following four approaches for detecting differential model accuracy, which they also call machine-learning measurement bias:
- Evaluation based on the score level: Sample individuals from different relevant groups with the same observed rating score, which serves as a proxy for the true score. In the context of essay scoring, these could be individuals from different groups of gender, different language groups, etc., whose essays received the same score by human raters. Let the score of their essays be predicted by a machine learning model, and check for systematic differences in the predicted scores.
- Evaluation based on the distribution level: Sample individuals from different relevant groups with a similar distribution of the rating scores, e.g., with the same mean and variance. As with the first approach, obtain a prediction of their score by the machine learning model, and check whether the distribution of the predictions is comparable over different groups.
- Evaluation based on the predictive accuracy: Evaluate the predictive accuracy of the machine learning model for subgroups of interest. In contrast to the first point, “Evaluation based on the score level,” this point does not concern the investigation of systematic differences between the predictions in various subgroups. The focus of this evaluation is whether the predictions are equally accurate in all relevant subgroups.
- Evaluation based on models for the score level and the score predictions: In a first step, use a simple model, e.g., a linear regression model, to predict the score predictions obtained from the machine learning from the observed human ratings. In a second step, evaluate whether this model is stable for different relevant subgroups. A similar, well-known approach based on moderated regression models was used by Cleary [70].
Expanding on the argumentation of Tay et al. [55], the concept of prediction invariance can be generalized in a straightforward way to scores from non-linear regression models, such as AI models. A similar perspective is taken by Johnson et al. [20] and Johnson and McCaffrey [71] and Williamson et al. [29], who discussed several methods for evaluating fairness in the context of essay scoring systems. Williamson et al. [29] focused on the question of whether it is fair on the level of relevant subgroups to replace a human rater by an AI score. Under this perspective, violations of fairness might occur if the AI model disagrees with human raters for specific subgroups (e.g., female students), or gives systematically higher or lower ratings than human raters. Johnson et al. [20] discuss different perspectives on fairness in AI. For applications in psychology and education, they discuss fairness based on separation and fairness based on sufficiency. To illustrate these terms, we assume that we want to check an essay scoring system, which aims to assess verbal intelligence, for fairness with regard to gender. In a non-technical language, fairness based on separation means in the context of this example that the essay score obtained by AI is independent from, and thus uncorrelated to, gender for persons with the same level of verbal intelligence. Fairness based on sufficiency, on the other hand, means that the true verbal intelligence score is independent from, and thus uncorrelated to, gender for persons with the same predicted essay score level. Johnson et al. [20] also suggest statistical tests to detect violation of these types of fairness which are however based on additional modeling assumptions.
In summary, a fundamental check of the fairness of AI models can be summarized by the following steps:
- In a first step, divide the sample into groups of interest, for instance based on groups of categorical covariates. In metric covariates, create discrete groups based on the observed values, for instance age groups.
- In a second step, train or fine-tune the AI model. Evaluate and compare its accuracy for each group, that is, for each relevant subpopulation obtained in the first step. This assessment includes a) the evaluation of the predictive accuracy, b) the evaluation of whether the model provides systematically lower or higher scores than an AI-based model, or whether there are any systematic deviations of the AI-based scores from human ratings.
This evaluation can be further enhanced with specific metrics and visualizations that compare the model performance between different groups. These methods can be complemented by other, more traditional methods for checking the fairness of AI-based scores, which are based on a) modeling the rating behavior of human and AI-based raters via models of IRT in a first step, and b) checking its invariance across relevant sub-groups. This evaluation aims at checking the invariance of the overall rating behavior of all raters. Further, specific statistical tests for separation and sufficiency fairness can be conducted to evaluate independence assumptions between the predicted score, the true construct or ability, and person covariates. However, these tests might require additional modeling assumptions, therefore we do not focus on them here.
Summary of Fairness Evaluation: In summary, our framework for evaluating AI fairness centers on establishing prediction invariance. This involves systematically comparing model accuracy, error distributions, and score biases across relevant demographic or topic-based subgroups to ensure that the AI model does not systematically disadvantage any specific group of test-takers.
An illustration with empirical data
As a next step, we illustrate several of the methods outlined above with an empirical data set. We showcase these steps using the training data from the Automated Essay Scoring Competition hosted by the Hewlett Foundation on Kaggle, which are available online (https://www.kaggle.com/competitions/asap-aes/data) and were also used for illustration in several other studies on automated essay scoring (e.g., [39]). The complete data set consists of eight essay sets with overall 12976 essays written in English. In each essay set, the essays were written by human students based on a specific prompt. A list of prompts is given in S1 Appendix.
All essays were rated by at least two human raters according to guidelines which differed slightly for each prompt, leading to at least an overall score, which serves as target value. Furthermore, all personally identifying information was removed from the published texts. A limitation of this public dataset is the lack of information on the human rater assignment process. It is unclear whether the same human raters scored all essays, which makes the interpretation of the observer inter-rater reliability problematic. Therefore, the findings of the reported analysis should be seen as an illustration of the methods rather than a definitive assessment of the scores provided by this AI model.
We split the 12976 essays into a training set of 7783 essays (60%), a validation set of 2596 essays (20%), and a test set of 2597 essays (20%), which are all stratified by the essay topics. An important practical restriction of this data set is that there are no known person covariates such as age or gender, so it is not possible to evaluate fairness with regard to such covariates. Further, it is unclear whether the two human raters per essay were the same raters for all essays or not, but it seems plausible that the raters differed for the individual essays.
After suitable preprocessing, the target value takes values between 0 and 1. In the resulting data set, at least two modeling approaches are conceivable: First, one could treat this problem as a regression problem, and aim to predict the specific target value. Second, one could frame this problem as a classification problem instead, by first dichotomizing the target value. In this problem, we would just be interested in whether an essay belongs to a specific group, e.g., the group of essays of high or low quality. Below, we demonstrate the evaluation of validity, reliability and fairness for both regression and classification problems. When the problem was framed as a classification problem, a threshold of .5 was used to distinguish between essays of high and low quality.
In the following, we assume that suitable preprocessing was carried out (e.g., the removal of any special characters) and that we have fitted a machine learning model to solve the classification or regression problem. In this demonstration, a DistilBERT model [30] that was pretrained on an English corpus was used. A Jupyter notebook that allows a replication of these analyses is available in the supplementary materials at https://osf.io/nr8he/ and as a Google Colab under the following link, although the exact results per run may differ slightly because of small differences in the fine-tuned models: https://colab.research.google.com/drive/1G-vBsGBxrZPV8PYxzHelnXaLw_fPNims?usp=sharing. This model is an encoder-based transformer model and leads to a numerical prediction of a target score. In principle, one could also use decoder-based transformer models such as GPT-4o [2] to get verbal or numerical evaluations of these essays. In the following sections, we only assume that a numerical score or label is predicted for each essay, but it is not important which type of model led to this prediction.
Treatment as a classification problem
We start our discussion with the treatment as classification problem, that is, we simply try to predict whether an essay gets a “high” or “low” (or, alternatively, “pass”/“fail”) overall score. From a technical perspective, this is the simpler approach, although the dichotomization of continuous variables cannot be recommended in general. We used cross-entropy as a loss function, which is technically comparable to using maximum likelihood estimation of the model parameters.
We first check the overall agreement between the predicted and the true classes in the validation dataset as an indication of construct validity. In practice, the prediction accuracy can be assessed by multiple criteria, such as the sensitivity and specificity, the F1 score, the classification accuracy, and others. In the essay scoring problem, we obtained a loss of .27 in the training data and of .44 in the validation data after training for three epochs. In the validation data, we observed an accuracy of .85 and an F1 score of .84, which indicate a sufficiently high accuracy for the purposes of this illustration. In a later subsection, we will further investigate the accuracy for different topics of student essays in the training, validation and test data sets.
Assessing reliability.
A first approach for checking reliability is based on the basic idea that an AI model should give comparable scores for different parts of the text, if these scores are meant to represent the same underlying latent characteristics, such as the ability to write with correct spelling and grammar. In the context of essay scoring, there might be criteria where different parts of the texts might not represent the same latent characteristic. For instance, the style of an essay might change during the text. In such instances, the proposed split could be problematic as an estimate of the reliability of the AI model. We therefore determine the scores for different parts of the text. In our example data, we counted the number of sentences in each text, and then split each text into two parts that both contained a similar or equal number of sentences. Very short texts that consist of only one sentence are omitted in this analysis. For each half, we obtained a score prediction, and compared the predicted score for the first and second half of the text. The internal consistency of the predictions was evaluated using a 2x2 contingency table. For each text, the columns of the table represent the predicted outcome (positive or negative) based on the first half, while the rows represent the predicted outcome based on the second half. Each cell thus contains the frequency of texts for each combination of first-half and second-half predictions. The agreement between the predictions of each half can again be calculated by a suitable correlation coefficient, such as Yule’s Q or Cohen’s Kappa. In the following, we calculated Yule’s Q and report analytical confidence intervals (CIs).
In the example data set, we observe the results reported in Table 2 using the example code given in the online supplementary material.
This leads to a Yule’s Q coefficient of Q = .84 (95% CI: [.82; .86]) for the training data, Q = .85 (95% CI: [.82; .88]) for the validation data and Q = .84 (95% CI: [.80; .87]) for the test data. After applying the Spearman-Brown prophecy formula, we obtain values of Q = .91 (95% CI: [.90; .92]) for the training data, of Q = .91 (95% CI: [.90; .93]) for the test data, and of Q = .92 (95% CI: [.89; .93]) for the validation data. These values, especially those for the validation and test data, can be interpreted as measures for the model’s internal consistency. For an additional analysis, one could use alternative criteria to split the text in two halves. Overall, these coefficients can be interpreted as satisfactory when interpreted as split-half coefficients. An interesting finding is that, in the example at hand, most of the predictions are negative for the individual test halves, although they are generally positive for the complete essays. This might indicate a tendency of the model to give worse grades to shorter essays. As was already discussed, such an effect was already reported for human as well as automated scores. In addition, differences between the ratings of text halves could reflect genuine differences in text quality between the text halves. These explanations seem plausible in the context of this empirical example, and we therefore consider these coefficients as robust measures of the model’s reliability.
A second facet of reliability is the robustness of the evaluation. A group of possible evaluations concerns the question of whether insignificant changes of the input texts have no significant effect on the AI scores. In the context of this illustration, we induce small errors, i.e., typos, in the essays of the data set (i.e., training, validation and test data) and evaluate whether these changes lead to an insignificant reduction of the predicted scores. In the example presented here, ten random letters were inserted at random positions in each text, and the prediction of the model before and after this change was investigated. The changes in the probabilities for a positive outcome are presented in the following Table 3 for the training, validation and test data sets. For all three data sets, we observe Yule’s Q coefficients that are larger than .99, indicating a strong correlation of the predicted score with and without 10 typos. However, we also observe that the inclusion of typos overall reduces the predicted score of the essays. This observation helps to understand how the algorithm scores and thereby provides further insights into the score’s validity.
Assessing validity
To further assess the validity of the rating, we investigated the effect of a more drastic change of the essay texts by inserting 50 typos. More advanced options, which were suggested by an anonymous reviewer, could include other types of small errors, such as using the wrong word, word repetitions, or word omissions. This change was expected to result in ratings that would have a smaller correlation with the original ratings, and would be overall more negative.
The changes in the probabilities for a positive outcome are presented in the following Table 4 for the training, validation and test data sets. For all three data sets, we observe Yule’s Q coefficients over .99, indicating a very strong correlation of the predicted score with and without 50 typos. As before, we also observe that the inclusion of typos overall reduces the predicted score of the essays.
For specific criteria, such as the detection of spelling and grammar errors, it might also make sense to use methods from interpretable machine learning, such as LIME and SHAP values. In general, these methods are designed to determine which features are relevant for obtaining a specific outcome. In the context of essay scoring, such values indicate which tokens, or word parts, affect the predicted score of an essay. A useful application of LIME and SHAP values requires that there is a theoretically meaningful relationship between the individual features and the dimension the essay scoring model is trying to assess. If the score aims, for instance, to evaluate the overall level of spelling and grammar, such values should detect and mark spelling errors in the essay as an evidence that the interpretation of the AI score is valid. Since the essay score aims at evaluating the overall quality of the essay, we do not apply LIME and SHAP values in this example.
Assessing fairness.
For assessing the fairness of the AI scoring, we focus on the evaluation of two related criteria that both depends on the availability of a categorical person covariate with regard to which we want to evaluate prediction invariance. The first criterion concerns the requirement that a) the prediction accuracy should be sufficiently high for all relevant groups of essays, and the second criterion concerns the requirement that b) the prediction accuracy should be at a comparable level for all relevant groups of essays. In a non-technical language, the first criterion demands that the AI score should be sufficiently predictive for all relevant groups of essays, and the second criterion demands that there should be no significant differences in the prediction accuracy. In practice, the prediction accuracy can be assessed by multiple criteria, such as the sensitivity and specificity, the F1 score, the classification accuracy, and others, which can be applied to evaluate the fairness of an AI score. As a third criterion for bias, we further demand that there should be no systematic bias for any relevant group of essays. That is, the numerical AI scores should be, on average, comparable to those given by human rater for all relevant groups of essays, and there should be no group where the AI model scores systematically higher or lower than the human raters.
Due to the absence of demographic data in the used dataset, a traditional fairness analysis across protected groups (defined by gender, ethnicity or similar characteristics) is not possible. To illustrate our methodology, we therefore focus in the following on differences of model performance and bias across subgroups defined by essay topics. This analysis should not be interpreted as a substantive evaluation of test fairness.
In the example data set, we define the relevant groups of essays by the topics on which the students should write. To check the first two criteria, we again obtain contingency matrices. In contrast to the evaluation of reliability, we are now interested in whether the agreement between human raters and the predictions of the AI model is comparable for all topics. The results for the training and validation data sets are presented in Tables 5 and 6.
As can be seen, most predictions are close to the human ratings for all essay topics, with accuracy rates (i.e., the rate of correct predictions) usually being above .8 for all essay scores. From this table, we also see that the essay topics differ with regard to their “difficulty,” that is, their probability that a randomly chosen student obtains a positive rating. While positive ratings are more likely for essay topics 1, 3, 6, 7 and 8, we observe more negative ratings for the remaining essay topics. It should be noted that this observation is not interpreted as a violation of fairness. Instead, we interpret it as an indication that the different essay topics are challenging to different degrees.
We are further interested in whether there are groups of essays where the LLM gives systematically higher or lower scores than human raters. In the confusion matrices, such an effect can be detected by investigating the ratio of false positive and false negative predictions, in particular in the validation data set. As we can calculate from Table 5, the ratio of false positive to false negative predictions is between 1.04 and 3.58 for the training data, .85 and 2.1 for the validation data set, and .8 and 3.71 for the test data set. For the validation and test data, positive predictions are more likely than false negative predictions in the essay topics 1, 2, 5, 7 and 8. Three of these topics are among those where positive outcomes are more likely, which might lead to a tendency of the scoring model to give a positive prediction. However, this effect can also be interpreted as a positive bias that affects students working on the mentioned essay topics.
This effect is caused by the general tendency of LLMs to predict classes that occur more frequent in their training data, especially in ambiguous cases where the class membership is unclear. For practical applications, it can still be recommended to use a training set that is mostly representative for the target application of the large language model, if infrequent classes are well enough represented. For an overview on more advanced approaches for addressing imbalanced data, see, for instance, He [72].
Overall, we observe a slight tendency of the AI model to provide more positive scores compared to human raters for some essay topics, which concludes our analysis of fairness.
Intraclass correlations
For illustrative purposes, we further calculated intraclass correlations ICC(1,1), ICC(2,1) and ICC(3,1) for single and average rating scores, using the notation of Liljequist et al. [33]. These coefficients are estimates of the intraclass correlation derived from three distinct statistical models. ICC(1,1) is based on a one-way random effects model, which assumes the absence of systematic measurement errors (bias). A potential scenario is a reliability study using a single rater, provided there are no procedure-related biases. ICC(2,1) is based on a two-way random effects model. It is used when biases are present and are considered a random effect, such as when the raters are a random sample from a larger population of raters. ICC(3,1) is based on a two-way mixed model. It applies when biases are present but are considered a fixed effect – for instance, when a specific group of raters is selected, and they are the only raters of interest for the study. The values of these intraclass correlations were identical for each topic in the training, validation and test data, but differed between single and average rating scores. Depending on the essay topic, the single rating scores took on values between ICC = .47 and ICC = .78 for the training data, between ICC = .47 and ICC = .71 for the validation data, and between ICC = .24 and ICC = .77 for the test data. For the average rating scores, typically larger values were observed. Depending on the essay topic, the average rating scores took on values between ICC = .72 and ICC = .91 for the training data, between ICC = .74 and ICC = .85 for the validation data, and between ICC = .48 and ICC = .91 for the test data.
Overall, these values indicate a moderate to high interrater reliability on the level of the average rating scores, but poor to moderate interrater reliability on the level of the individual rating scores, following the criteria reported by Koo et al. [34]. However, the analysis of the intraclass correlations implicitly assumes that the same two raters rated all essays for each topic, which seems implausible based on the available description of the data. Given the small number of raters, we did not carry out an analysis of differential rater functioning based on models of IRT.
Treatment as a regression problem
We now frame the prediction problem as a regression problem, in which we want to predict the specific value between 0 and 1, which is given to each essay, as accurately as possible. Compared to a classification problem, regression models rely on other metrics for assessing their accuracy, such as mean squared errors (MSE), or root of mean squared errors (RMSE). Their estimation is usually also based on different loss functions which consider the changed scale of the predicted variable. In the current implementation, the MSE was used as a loss function for the regression problem. In the following, we demonstrate how switching from a classification to a regression problem affects the evaluation of validity, reliability and fairness.
Assessing reliability
As was already outlined, the basic idea of assessing the reliability of AI scores consists in comparing their evaluation and comparison for different parts of the text. In the case of regression problems, these predictions are metric in nature, which leads to the option of calculating a Pearson correlation between these predictions. In our example dataset, we counted the sentences in each text and split them into two halves of similar size, ensuring each half contained only complete sentences. We then calculated a predicted score for each half and measured the correlation between these two scores as an index of reliability. We further calculated confidence intervals (CI) for the correlations using the psych package in R [73]. Short texts consisting of only one sentence were discarded from this analysis. In our example data, this leads to a correlation of r = .63 (95% CI: [.61; .65]) in the validation data, r = .66 (95% CI: [.65; .67]) in the training data, and r = .66 (95% CI: [.64; .68]) in the test data. An application of the Spearman-Brown prophecy formula leads to values of r = .77 (95% CI: [.75; .79]) in the validation data, r = .80 (95% CI: [.79; .81]) in the training data, and r = .80 (95% CI: [.79; .81]) in the test data. Compared to the treatment as a classification model, these numbers are lower, but, depending on the specific application and intended use of the model, might still be satisfactory. These results are further illustrated in the following Figs 1–3.
Each point compares the predicted score for the first and second half of a text.
Each point compares the predicted score for the first and second half of a text.
Each point compares the predicted score for the first and second half of a text.
To improve this measure of reliability, one might further fine-tune the underling model or even change the model. However, given that the quality of the first and second part of an essay might differ significantly, it is plausible that there is an upper bound below 1.0 for the correlation between the predictions for the first and second half.
When applying the regression framework, we did not calculate the intraclass correlation coefficient or apply models based on IRT, since these methods are aimed for the analysis of categorical data.
Again, we can further evaluate the effect of inserting ten typos in the text as an evaluation of its robustness. For the example data set, this evaluation is carried out in the following Figs 4–6 for the training, validation and test data, respectively. In the three data sets, we observe a correlation of about r = .98 (95% CI: [.98; .98]) between the predicted scores of texts with and without typos.
Each text was perturbed by inserting 10 random typos at random positions, and the predictions before and after perturbation are compared.
Each text was perturbed by inserting 10 random typos at random positions, and the predictions before and after perturbation are compared.
Each text was perturbed by inserting 10 random typos at random positions, and the predictions before and after perturbation are compared.
As can be seen, the insertion of typos generally reduces the predicted value for the same text, which is in line with theoretical expectations. However, there are a few exceptions, that is, individual texts for which the rating unexpectedly increases. In the Figures above, these texts are visible as dots above the diagonal.
Those unexpected increases, though relatively rare, may be artifacts that stem from the complex, non-linear architecture underlying the neural network. For instance, the insertion of random letters might create a new token (word or word part) that the model associates with higher-scoring texts. This result highlights the nature of the model as a “black box” and also the need of checking the robustness of the model scores.
Assessing validity
Based on the results for assessing the robustness of the model scores, we further evaluated a facet of validity by assessing the effect of inserting 50 typos. The results of this evaluation are illustrated in the following Figs 7–9 for the training, validation and test data, respectively.
The average score predictions decrease by this change, giving insight into the scoring process.
The average score predictions decrease by this change, giving insight into the scoring process.
The average score predictions decrease by this change, giving insight into the scoring process.
As can be seen, the insertion of typos generally reduces the predicted value significantly for the same text, which is in line with theoretical expectations. Again, there are individual texts for which the rating increases. In the three data sets, we observe a correlation of about .93 (95% CI in the training data: [.93; .93]; 95% CI in the validation and test data: [.92; .94]) between the predicted scores of texts with and without typos. This correlation is smaller than the analogous correlation observed after the insertion of 10 typos and underlines that the changes to the input texts had a significant effect on the scores.
As with classification problems, we can obtain further evidence for the construct validity of a model’s predicted score by checking its accuracy in the training, validation and test data sets. For the regression model, we obtain a model with an MSE of .01 in the training data and an MSE of .13 in the validation data after training for three epochs.
Assessing fairness
In the discussion of the evaluation of fairness of AI scores in classification problems we discussed three criteria, which can be directly translated to corresponding criteria for fairness in regression problems: For the first two criteria, we evaluate the metrics of the AI model for different relevant groups of essays, and ensure that the AI scores are a) sufficiently accurate for any group, and b) of comparable accuracy for all groups. As a third criterion, we confirm that c) there are no systematic biases between the predicted and observed scores for any relevant group of essays.
For all eight essay topics, we observed a correlation between the predicted and the observed scores in the range from .83 to .93 in the training data, from .58 to .85 in the validation data and from .65 to .85 in the test data. These numbers are in line with the results of a recent meta-analysis on the agreement of human raters and automated scoring systems [74]. Depending on the intended application of the essay scoring system, they can therefore be interpreted as a satisfactory level of fairness. For biases between the predicted and observed scores, we found mean values between .00 and .03 for all essay topics in the training data, between −.01 and .04 in the validation data, and between −.01 and .04 in the test data. We therefore conclude that there are no systematic biases in the predictions.
Discussion
We have discussed three central standards for psychological and educational assessments, and proposed specific methods for how to evaluate them for numerical scores that were obtained using models of artificial intelligence. This led to an evaluation protocol that is evaluated in Table 1. This protocol was illustrated via an empirical application. In this application, we used an LLM to predict human ratings. This problem was framed as a classification or regression problem, which led to slight differences in the proposed evaluation methods. Compared to existing testing frameworks [29,36,75], we focused on providing specific statistical evaluations that are aligned with psychometric standards. The workflow is illustrated in Fig 10.
The diagram illustrates how a single essay input is processed through three parallel evaluation pipelines to evaluate Internal Consistency, Robustness/Validity, and Fairness.
In our example, the numerical scores of all essays were obtained using a specific model architecture, namely the encoder-based transformer model DistilBERT. In principle, such scores can be obtained by multiple methods from the field of natural language processing (NLP), such as bag-of-words models, recurrent neural networks or even generative transformer models such as GPT-4o or Gemini, which are based on the evaluation of textual prompts. For the proposed methods, the specific nature of the underlying NLP model only plays a role in that it affects how the proposed methods can be technically implemented. On a conceptual level, the prediction model is treated as a “black box,” and the proposed methods focus on their predictions. This characteristic also means that we cannot check all aspects of fairness, reliability and validity that are generally evaluated in, for instance, an IRT framework. For instance, we generally cannot check our black box models for structural invariance, which is related to differential item functioning and measurement invariance in a FA and IRT framework. Furthermore, our empirical results are specific to DistilBERT and the Hewlett Foundation dataset, and may not generalize to other models, domains, or corpora. This slight conceptual weakness is compensated by a strong independence of the discussed methods from the underlying measurement framework.
If significant violations of reliability, validity or fairness are detected in an essay scoring model, this usually indicates that a modification of the model is required. As a first step, longer training or the fine-tuning with additional data might lead to a sufficient improvement of the model. However, in case that additional training or fine-tuning does not resolve these problems, it might be necessary to switch to a different model instead.
For the evaluation of automated essay scoring systems, such as those considered in this text, we can make recommendations how the presented methods should be included in existing evaluation pipelines. First, we recommend that evaluations of such systems report at least one measure of internal consistency (such as a split-half analysis) and a measure of robustness to minor changes. Validity arguments for such systems should at least be supported by empirical evidence of strong predictive accuracy against human scores. These arguments can be further supported by additional methods, such as methods of interpretable machine learning, when this is technical feasible and provides meaningful evidence. For instance, when the essay scoring systems aims at checking the spelling of a text, methods of interpretable machine learning should be able to detect mis-spelled words. Fairness arguments should, at a minimum, compare predictive accuracy and mean score differences across relevant, protected demographic subgroups, while clearly acknowledging any dataset limitations.
Interpretation of the empirical findings
In our empirical example, we found evidence for high internal consistency (Spearman-Brown correlations > .7) and robustness and several important aspects of validity. The reported confidence intervals demonstrate the stability of these results. In the data set used for the demonstration, the automated essay scoring data set of the Hewlett Foundation, we used essay topics as a proxy variable for the evaluation of fairness. The presented framework provides tools for evaluating fairness aspects with regard to gender, education and similar covariates if those are available.
Ethical Implications and Transparency
It is important to note that passing the checks outlined in this framework does not equate to the ethical safety of a system for generating scoring or other feedback, which is particularly relevant in high-stakes educational applications. The “black box” nature of deep learning models poses significant challenges for accountability. This is particularly relevant for the evaluation of fairness, where procedural fairness needs to be covered outside of the statistical evaluation provided here.
In the holistic data example used in this illustration, values of explainable AI such as SHAP values can be noisy, since it is impossible to relate the overall outcome to individual tokens. Instead, systematic experiments based on varying model input can allow an evaluation of the model’s behavior.
Outlook and limitations
The evaluation framework presented in this work has some limitations, but also points to possible future extensions and areas for future research. A limitation of the current empirical illustration is that methods of explainable AI were not applied, as the holistic scoring criterion did not lend itself to token-level interpretation. Future work with more specific scoring rubrics could demonstrate these methods empirically.
Topically, we focused on the specific application of automated essay scoring, and while the presented methods can help to evaluate specific applications of essay scoring models, they do not allow a relative comparison of various essay scoring systems that is independent of their application. For instance, we are not able to conclude based on this techniques which model among a set of given LLMs, such as GPT-4o or Gemini, is best suited for scoring human essays. An interesting field of future research is the construction of a benchmark set of essays which could be used to compare essay scoring systems against each other. Such a set of essays could contain typical essays from various settings (such as schools and universities) and, possibly, various languages, but also essays that contain typical flaws and errors (in various aspects such as spelling, argumentation, text consistency, or grammar) that should be detected by essays scoring systems.
A second extension concerns the development of similar tools and guidelines for models that provide non-numerical output. For instance, generative AI models such as GPT-4o can be instructed to provide verbal feedback on automated essays. Compared to BERT and similar models, evaluating the robustness of the output of these model could include an evaluation of its robustness to changes in the prompt. In contrast to the methods outlined above, such an extension would require methods that are more qualitative in nature while evaluating aspects of reliability, validity and fairness. A general approach for handling verbal output could consist of using numerical linguistic features to quantify specific characteristics of the texts, such as readability, which can be quantified by formulas based on the average word and sentence lengths (e.g., the Flesch–Kincaid readability tests). A related idea is using LLMs to get a rating of the emotional tone of a text or any feature that is of interest. Evidence for the validity of these quantifications can be provided by human inspection.
A third direction for possible extensions could develop similar methods for the evaluation of AI models that evaluate non-verbal responses. A possible use case could be AI models that evaluate drawn figures (e.g., as part of educational or neuropsychological assessments) or videos.
Conclusion
If AI models are used as part of psychological and educational assessments, they must be held to rigorous professional standards. In this work, we moved beyond conceptual discussion and suggested an evaluation framework and protocol for validating AI-based scores.
Our methodological contributions are threefold. First, we combined methods based on split-half analysis with input perturbation to assess aspects of reliability, namely internal consistency and robustness. Second, input perturbation can also serve as a critical stress test for construct validity. For instance, a model that delivers similar outputs despite massive text changes fails to measure writing quality properly. Third, we outlined a fairness check that compares the prediction accuracy across subgroups and which was applied to essay topics as a proxy variable.
Our empirical example provided evidence for high internal consistency (Spearman-Brown correlations > .7), robustness and several important aspects of validity. We also illustrated the evaluation of aspects of fairness by comparing the accuracy of the machine learning prediction across several relevant subgroups.
Under this perspective, we recommend that future deployments of automated essay scoring systems adopt the checklist provided in Table 1 or a similar evaluation system. While AI models can be proficient in score prediction, fairness and validity are not inherent properties of the architecture and must be verified through rigorous, standardized psychometric evaluations. In this paper, we provide some suggestions for such evaluations, but feel that the development of further methods is necessary. Future work should develop standardized benchmark corpora for psychometric applications and extend these methods to non-numerical AI output, as discussed above.
Supporting information
S1 Appendix. List of essay prompts.
Descriptions of the eight essay prompts of the Hewlett Foundation automated essay scoring data set used in the empirical example.
https://doi.org/10.1371/journal.pone.0354680.s001
(PDF)
References
- 1.
OpenAI. GPT-4 Technical Report. 2023.
- 2.
OpenAI. GPT-4o. https://openai.com/index/hello-gpt-4o/ 2024.
- 3.
Google AI. Introducing Gemini: Our Next Generation Model. https://ai.googleblog.com/ 2023.
- 4. Attali Y, Runge A, LaFlair GT, Yancey K, Goodwin S, Park Y, et al. The interactive reading task: Transformer-based automatic item generation. Front Artif Intell. 2022;5:903077. pmid:35937141
- 5. Bezirhan U, von Davier M. Automated reading passage generation with OpenAI’s large language model. Computers and Education: Artificial Intelligence. 2023;5:100161.
- 6. Hao J, von Davier AA, Yaneva V, Lottridge S, von Davier M, Harris DJ. Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI. Educational Measurement. 2024;43(2):16–29.
- 7. Jung JY, Tyack L, von Davier M. Combining machine translation and automated scoring in international large-scale assessments. Large-scale Assess Educ. 2024;12(1).
- 8.
Martinková P, Hladká A. Computational aspects of psychometric methods: With R. Chapman and Hall/CRC. 2023.
- 9. Štěpánek L, Dlouhá J, Martinková P. Item difficulty prediction using item text features: comparison of predictive performance across machine-learning algorithms. Mathematics. 2023;11(19).
- 10. von Davier M. Automated Item Generation with Recurrent Neural Networks. Psychometrika. 2018;83(4):847–57. pmid:29532403
- 11. Zu J, Choi I, Hao J. Automated distractor generation for fill-in-the-blank items using a prompt-based learning approach. Psychological Test and Assessment Modeling. 2023;65(2):55–75.
- 12. von Davier AA. Computational Psychometrics in Support of Collaborative Educational Assessments. J Educational Measurement. 2017;54(1):3–11.
- 13.
von Davier AA, Mislevy RJ, Hao J. Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and Python. Springer Nature. 2022.
- 14.
Bulut O, Beiting-Parrish M, Casabianca JM, Slater SC, Jiao H, Song D. The rise of artificial intelligence in educational measurement: opportunities and ethical challenges. 2024. https://arxiv.org/abs/2406.18900
- 15.
Burstein J, LaFlair GT. Where assessment validation and responsible AI meet. https://arxiv.org/abs/2411.02577 2024.
- 16. Runge A, Attali Y, LaFlair GT, Park Y, Church J. A generative AI-driven interactive listening assessment task. Front Artif Intell. 2024;7:1474019. pmid:39559344
- 17. Ferrara S, Qunbar S. Validity Arguments for AI‐Based Automated Scores: Essay Scoring as an Illustration. J Educational Measurement. 2022;59(3):288–313.
- 18.
Harris DJ. AI: Can You Help Address This Issue?. Educational Measurement: Issues and Practice. 2024. https://doi.org/10.1111/emip.12655
- 19.
Iliescu D, Greiff S, Ziegler M, Fokkema M. Artificial Intelligence, Machine Learning, and Other Demons. 2022.
- 20. Johnson MS, Liu X, McCaffrey DF. Psychometric Methods to Evaluate Measurement and Algorithmic Bias in Automated Scoring. J Educational Measurement. 2022;59(3):338–61.
- 21. Mitchell M, Krakauer DC. The debate over understanding in AI’s large language models. Proc Natl Acad Sci U S A. 2023;120(13):e2215907120. pmid:36943882
- 22.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association. 2014.
- 23.
International Test Commission, Association of Test Publishers. Guidelines for technology-based assessment. International Test Commission and Association of Test Publishers. 2022.
- 24.
Rao CR, Sinharay S. Psychometrics. Amsterdam: Elsevier. 2007.
- 25.
Lord FM, Novick MR. Statistical theories of mental test scores. Addison-Wesley. 1968.
- 26.
van der Linden W. Handbook of Item Response Theory – Three Volume Set. New York, NY: CRC Press. 2016.
- 27.
Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: 2019. https://www.aclweb.org/anthology/N19-1423
- 28. Attali Y, Burstein J. Automated essay scoring with e-rater® v.2. The Journal of Technology, Learning and Assessment. 2006;4(3).
- 29. Williamson DM, Xi X, Breyer FJ. A Framework for Evaluation and Use of Automated Scoring. Educational Measurement. 2012;31(1):2–13.
- 30.
Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. 2020.
- 31. Fan J, Sun T, Liu J, Zhao T, Zhang B, Chen Z, et al. How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores. J Appl Psychol. 2023;108(8):1277–99. pmid:36745068
- 32. Ten Hove D, Jorgensen TD, van der Ark LA. Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs. Psychol Methods. 2024;29(5):967–79. pmid:36048052
- 33. Liljequist D, Elfving B, Skavberg Roaldsen K. Intraclass correlation – A discussion and demonstration of basic features. PLOS ONE. 2019;14(7):1–35.
- 34. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. pmid:27330520
- 35. Martinková P, Bartoš F, Brabec M. Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables. Journal of Educational and Behavioral Statistics. 2023;48(3):349–83.
- 36.
Attali Y. Validity and reliability of automated essay scoring. Validity and reliability of automated essay scoring. New York, NY: Routledge. 2013. p. 181–98.
- 37. Page EB. The imminence of… grading essays by computer. Phi Delta Kappan. 1966;47(5):238–43.
- 38. Johnson AC, Wilson J, Roscoe RD. College student perceptions of writing errors, text quality, and author characteristics. Assessing Writing. 2017;34:72–87.
- 39. Faseeh M, Jaleel A, Iqbal N, Ghani A, Abdusalomov A, Mehmood A. Hybrid approach to automated essay scoring: integrating deep learning embeddings with handcrafted linguistic features for improved accuracy. Mathematics. 2024;12(21).
- 40.
Goodfellow I, Bengio Y, Courville A. Deep Learning. MIT Press. 2016.
- 41. Higgins D, Heilman M. Managing What We Can Measure: Quantifying the Susceptibility of Automated Scoring Systems to Gaming Behavior. Educational Measurement. 2014;33(3):36–46.
- 42. Powers DE, Burstein JC, Chodorow M, Fowles ME, Kukich K. Stumping e-rater:challenging the validity of automated essay scoring. Computers in Human Behavior. 2002;18(2):103–34.
- 43. Speer AB, Oswald FL, Putka DJ. Reliability Evidence for AI-Based Scores in Organizational Contexts: Applying Lessons Learned From Psychometrics. Organizational Research Methods. 2025;29(3):412–40.
- 44. CampbelL DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull. 1959;56(2):81–105. pmid:13634291
- 45. Hubley AM, Zumbo BD. Validity and the Consequences of Test Interpretation and Use. Soc Indic Res. 2011;103(2):219–30.
- 46. Bleidorn W, Hopwood CJ. Using Machine Learning to Advance Personality Assessment and Theory. Pers Soc Psychol Rev. 2019;23(2):190–203. pmid:29792115
- 47. Lane S. Validity evidence based on testing consequences. Psicothema. 2014;26(1):127–35. pmid:24444740
- 48.
Molnar C. Interpretable Machine Learning. 2nd ed. 2022.
- 49.
Hvitfeldt E, Pedersen TL, Benesty M. lime: Local Interpretable Model-Agnostic Explanations. 2022. https://CRAN.R-project.org/package=lime
- 50.
Shapley LS. A value for n-person games. A Value for n-Person Games. Princeton: Princeton University Press. 1953. p. 307–18.
- 51.
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. https://www.tensorflow.org/
- 52.
Ding Y, Riordan B, Horbach A, Cahill A, Zesch T. Don’t take “nswvtnvakgxpm” for an answer –The surprising vulnerability of automatic content scoring systems to adversarial input. In: Proceedings of the 28th International Conference on Computational Linguistics, 2020. 882–92. https://doi.org/10.18653/v1/2020.coling-main.76
- 53. Eid M, Lischetzke T, Nussbeck FW, Trierweiler LI. Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model. Psychol Methods. 2003;8(1):38–60. pmid:12741672
- 54. Nussbeck FW, Eid M, Geiser C, Courvoisier DS, Lischetzke T. A CTC(M−1) Model for Different Types of Raters. Methodology. 2009;5(3):88–98.
- 55. Tay L, Woo SE, Hickman L, Booth BM, D’Mello S. A Conceptual Framework for Investigating and Mitigating Machine-Learning Measurement Bias (MLMB) in Psychological Assessment. Advances in Methods and Practices in Psychological Science. 2022;5(1).
- 56. Millsap RE. Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods. 1997;2(3):248–60.
- 57. Millsap RE. Invariance in Measurement and Prediction Revisited. Psychometrika. 2007;72(4):461–73.
- 58.
Barocas S, Hardt M, Narayanan A. Fairness and Machine Learning: Limitations and Opportunities. MIT Press; 2023.
- 59. Goretzko D, Israel LSF. Pitfalls of Machine Learning-Based Personnel Selection. Journal of Personnel Psychology. 2022;21(1):37–47.
- 60.
Bolukbasi T, Chang KW, Zou J, Saligrama V, Kalai A. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In: 2016. https://arxiv.org/abs/1607.06520
- 61. Johnson M, Zhang M. Examining the responsible use of zero-shot AI approaches to scoring essays. Sci Rep. 2024;14(1):30064. pmid:39627285
- 62. Hovy D, Prabhumoye S. Five sources of bias in natural language processing. Lang Linguist Compass. 2021;15(8):e12432. pmid:35864931
- 63.
Schaller NJ, Ding Y, Horbach A, Meyer J, Jansen T. In: Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 2024. 210–21. https://aclanthology.org/2024.bea-1.18
- 64.
Aomi I, Tsutsumi E, Uto M, Ueno M. Integration of Automated Essay Scoring Models Using Item Response Theory. Lecture Notes in Computer Science. Springer International Publishing. 2021. p. 54–9. https://doi.org/10.1007/978-3-030-78270-2_9
- 65. Goodwin S. A Many-Facet Rasch analysis comparing essay rater behavior on an academic English reading/writing test used for two purposes. Assessing Writing. 2016;30:21–31.
- 66. Myford C, Wolfe E. Detecting and measuring rater effects using many-facet rasch measurement: Part I. Journal of Applied Measurement. 2003;4:386–422.
- 67. Myford CM, Wolfe EW. Detecting and measuring rater effects using many-facet Rasch measurement: Part II. J Appl Meas. 2004;5(2):189–227. pmid:15064538
- 68.
Yang K, Raković M, Li Y, Guan Q, Gašević D, Chen G. Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability. 2024. https://arxiv.org/abs/2401.05655
- 69.
Doewes A, Saxena A, Pei Y, Pechenizkiy M. In: Proceedings of the 15th International Conference on Educational Data Mining, 2022. 206–16.
- 70. Cleary TA. Test Bias: Prediction Of Grades Of Negro And White Students In Integrated Colleges. J Educational Measurement. 1968;5(2):115–24.
- 71.
Johnson MS, McCaffrey DF. Evaluating Fairness of Automated Scoring in Educational Measurement. Advancing Natural Language Processing in Educational Assessment. Routledge. 2023. p. 142–64. https://doi.org/10.4324/9781003278658-12
- 72. Haibo He, Garcia EA. Learning from Imbalanced Data. IEEE Trans Knowl Data Eng. 2009;21(9):1263–84.
- 73.
Revelle W. Psych: Procedures for Psychological, Psychometric, and Personality Research. 2024.
- 74. Yun J. Meta-analysis of inter-rater agreement and discrepancy between human and automated English essay scoring. English Teaching. 2023;78(3):105–24.
- 75.
Burstein J. The Duolingo English Test Responsible AI Standards. Duolingo, Inc. 2025. https://englishtest.duolingo.com/research