Fig 1.
Framework of the MetaRL.Ratio based on forward and backward models.
Empirical choices (non-bold black) were characterized using an RL model (paths “1”, “2” and “3”) leading to choice parameters. The model’s confidence in the empirical choices was determined based on its assessment of the probability that those choices were correct (“5”). These values were rescaled linearly (“6”; using lower and upper bounds LC and HC for subject s) to match the participant’s empirical confidence reports by minimizing the squared error (“7”, “8”). This produces what we call the Forward confidence. In a second comparison step, we also estimated the choice parameters through a Backward method by fitting empirical confidence ratings to the similarly scaled confidence of another RL model to the empirical choices (“9” to “12”). Finally, the behaviour of both the Forward and Backward models was simulated on new instances of the same task (“1”, “13”), yielding estimates of the proportion of trials on which each model chose the best option, which we refer to as the Forward and Backward performance (“4”, “14”). The Backward performance, which captures the influence of both empirical choices and confidence, serves as a measure of metacognitive sensitivity. To ensure a meaningful comparison, this measure was normalized by dividing it by Forward performance, resulting in the metacognitive efficiency measure known as the MetaRL.Ratio.
Fig 2.
Two-outcome reversal learning task.
(A) Participants made choices between two bandits (generated by an LLM: OpenAI ChatGPT, model o4-mini; see OpenAI (2025)) and reported their confidence in the correctness of those choices (on a continuous scale) before receiving feedback on the outcome. (B) The bandits dispensed rewards, the dots, according to two normal distributions with means of 40 and 60, which alternated every trials. The variance for both options was set at 8 in a low-variance condition and set to 16 in a high-variance condition. Each participant completed both conditions in a counterbalanced order, with a time gap of
days.
Fig 3.
Comparison between Forward and Backward models in choice, confidence and parameters.
A) The performance, proportion of trials on which each model chose the best option, of the Backward model was significantly lower than both empirical and Forward performance. Additionally, Forward performance significantly lagged behind empirical performance. B) The confidence bias levels of the Backward model were not significantly different from the Forward model and empirical data, while there was a weak but significant difference between the confidence bias of the Forward model and empirical data. C) The Forward model predicted choices better than the Backward model, as measured by the negative log likelihood. D) The confidence ratings of the Backward model were closer to the empirical data than those of the Forward model. E) The learning-rate was significantly lower in the Backward model compared to the Forward model. F) The inverse temperature parameter was not significantly different between two models. The dots in the plots represent the corresponding estimates for each subject in the low variance condition of the task.
Fig 4.
Consistency with quadratic scoring rule, model-agnostic measure of metacognitive sensitivity.
A) Backward performance was significantly correlated with QSR. B) Backward performance was also significantly correlated with scaled-QSR, which determines a linear scaling of empirical confidence values to maximize QSR. The dots in the plots above represent the corresponding estimations for each subject. The dots in the plots show the estimates for each subject in the low variance condition of the task.
Fig 5.
Relationship between MetaRL.Ratio and empirical choice parameters.
A) The MetaRL.Ratio, our measure of metacognitive efficiency, was independent of empirical performance (the proportion of trials on which participants chose the best option). B) The correlation between MetaRL.Ratio and confidence bias decreased after applying the confidence scaling method. C & D) The MetaRL.Ratio was not significantly correlated with the inverse temperature (C) or the learning-rate (D) of the Forward model. The dots in the plots show the estimates for each subject in the low variance condition of the task.
Fig 6.
Independence of MetaRL.Ratio from task difficulty.
A) Empirical performance (green), Forward performance (blue) and Backward performance (red) were significantly lower in the high- than the low-variance condition (HV versus LV). B) The MetaRL.Ratio was not significantly different between the two levels of task difficulty. C) The confidence bias was significantly lower in high-variance rather than low-variance condition of task according to empirical data (green), Forward model (blue) and Backward model (red).
Fig 7.
Parameter comparison between two levels of task difficulty.
A) The learning-rate of the Forward model was slightly lower in the high-variance (HV) condition than the low-variance (LV) one of the task, while the difference was not significant for the Backward model. B) The inverse temperature was significantly lower in high versus low difficulty, strongly for the Forward model, but modestly so for the Backward model. The dots shown in the above plots correspond to the estimates for each subject.
Table 1.
Correlation of performance and MetaRL.Ratio between Low- and High-Variance Conditions. Empirical and Forward model performance were not correlated between the two task conditions. In contrast, both the Backward performance (red) and MetaRL.Ratio (purple) were.
Table 2.
Correlation of confidence bias between Low- and High-Variance Conditions. The confidence bias was correlated between two task difficulties for empirical data and also Forward and Backward models.