Using think aloud with female adolescents to validate psychological well- and ill-being self-report measures

Sophie C. M. Chatwin; Rebbeca M. Pearson; Haridhan Goswami; Paul R. Appleton

doi:10.1371/journal.pmen.0000551

Abstract

Many existing measures of psychological well- and ill-being are used with young people without testing contemporary adolescent understanding of them. The purpose of this study was to use think aloud interviews to test adolescent understanding of three existing, and one newly created psychological well- or ill-being measures. An initial sample of 40 female participants aged 13–14 years took part in the study. Problematic items were identified based on thematic analysis of adolescent feedback. One item of the Lethargy Scale and one item of the Subjective Vitality Scale were adapted following integration of think aloud findings. The results also indicated that the Brief Serenity Scale was unsuitable for the participants. Subsequently, a second sample of 57 female participants aged 13–14 years completed the think aloud protocol with an alternative measure of serenity, the Child Serenity Scale, and thematic analysis revealed no items were problematic. A 24-item psychological well- and ill-being self-report measure was consequently proposed for use in future research.

Citation: Chatwin SCM, Pearson RM, Goswami H, Appleton PR (2026) Using think aloud with female adolescents to validate psychological well- and ill-being self-report measures. PLOS Ment Health 3(2): e0000551. https://doi.org/10.1371/journal.pmen.0000551

Editor: Lambert Zixin Li, National University of Singapore, SINGAPORE

Received: November 3, 2025; Accepted: January 15, 2026; Published: February 17, 2026

Copyright: © 2026 Chatwin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data is stored in Manchester Metropolitan University’s Research Repository, https://doi.org/10.23634/MMU.00642372.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Globally, 14% of adolescents report having a mental health disorder such as anxiety, depression, eating disorders and addictive behaviours [1]. National Health Service (NHS) data shows one in five children and young people in England aged eight to 25 had a probable mental health disorder in 2023, which is worse since the COVID-19 pandemic [2]. Moreover, the Children’s Society found 11% of 10- to 17-year-olds in the United Kingdom (UK) had low well-being, determined by low life satisfaction scores which were almost half than the European average [3]. Failing to address these issues in adolescence has consequences that stem into adulthood; 50% of adult mental health disorders onset by the age of 14, and ill-being is associated with an increased likelihood of impaired social relationships, substance misuse, and criminal behaviour [4].

Defining psychological well-being is complex because there are numerous conceptualisations, and no universally accepted definition. There is one unanimous theme however, and that is psychological well-being is defined as hedonic or eudemonic [5]. The hedonic approach defines psychological well-being in terms of life satisfaction and subjective happiness. The eudemonic approach focuses on optimal functioning [5]. Ryan and Deci [6] and Ryff [7] suggest due to its multidimensional nature, well-being should include both hedonic and eudemonic aspects. Ill-being has been relatively neglected compared to well-being in literature [8]. Yet again, there is no universally accepted definition of psychological ill-being. Many studies imply ill-being is the opposite of well-being or fail to provide a definition beyond “e.g., depression, stress” [9]. More recently, ill-being has been defined as “unpleasant feelings or emotions that impact the level of functioning” [10], and “negative psychological states” [11]. There is also a debate whether well- and ill-being are independent dimensions, or whether they sit at opposite ends of a bipolar continuum. The continuum theory suggests individuals with higher levels of ill-being would have lower levels of well-being and vice versa [12]. The independence theory suggests higher levels of ill-being would not necessarily be associated with lower levels of well-being and vice versa [12]. Bradburn’s [13] research into independence versus bipolarity discovered the association between positive and negative affect was not strong, despite being inversely correlated. Therefore, scoring high for one construct does not predict scoring for the other construct. The present study will take the view that psychological well- and ill-being are independent, and therefore need to be measured simultaneously but independently.

A criticism of well- and ill-being theories is that they were developed by adults, often for adults [14], but are regularly applied to young people without consideration of their relevance and applicability. Taking two hedonic theories, subjective well-being [15] and quality of life [16] as examples, well-being in the work domain, which is central to these theories, is not relatable for all stages of adolescence. It could be argued that these conceptualisations, and related measures based on these conceptualisations, may not accurately capture an adolescent’s experience of psychological well- and ill-being. To overcome the adult centrism often seen in previous research, some studies have involved adolescents when conceptualising well-being [17,18], but less attention has been paid to ill-being. In one study however, Bracey [19] asked a sample of young people in the UK to write down words they associated with well- and ill-being, and where they felt these feelings (head, body, and/or heart). Younger participants (aged 7–11 years) defined well- and ill-being as unidimensional, singularly occurring feelings. Older participants (aged 12–15 years) had a greater sense of their complexity, describing well-being as both hedonic and eudemonic (“you’d be happy, you’d be healthy…content with…life”), and ill-being as multidimensional (“scared”, “upset”, “you feel bad”) [19]. Bracey’s involvement of adolescents, and consideration of both hedonic and eudemonic perspectives, arguably produced more appropriate definitions for adolescents than existing adult definitions. Focusing on the psychological domain of well- and ill-being, Bracey’s [19] definitions are adopted in this study. Psychological well-being is having life satisfaction, happiness, and optimal functioning determined by positive affect, vitality, and serenity. Psychological ill-being is negative emotional states and undesirable psychological conditions determined by negative affect and lethargy. Participants in Bracey’s [19] work also identified hyperactivity as a facet of ill-being, however in the current study, it was decided to remove this from the definition due to its connotations with neurodivergent conditions such as attention deficit hyperactivity disorder (ADHD). Additionally looking at existing hyperactivity measures, they are often intended for clinical practice, or items reflect an outcome of ill-being rather than a facet of ill-being (e.g., “I was easily distracted, I found it difficult to concentrate” from the Strengths and Difficulties Questionnaire [20]).

A large body of research has examined young people’s psychological well- and ill-being utilising a variety of measures (e.g., Scale of Positive and Negative Experience (SPANE) [21]; Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) [5]; Children’s Society Index of Children’s Subjective Wellbeing (CSICSW)) [22]. Although previous measures are well-validated, they have limitations. They often only consider either eudemonic or hedonic well- (but not ill-) being; as noted above, several measures were developed by adults, for adults, but subsequently used with children/adolescents; and the content validity of these measures is rarely tested. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) can facilitate the assessment of the content validity of self-report measures [23]. According to COSMIN guidelines, relevance, comprehensiveness, and comprehensibility of items from the perspective of the participants and the professional (i.e., researcher) need to be established for a self-report measure to have good content validity. With existing measures, COSMIN guidelines are not always met. For example, regarding the WEMWBS [5] the comprehensibility of items was poor with young participants, as items caused confusion, were misinterpreted, or participants were hesitant to answer [4]. This could be due to the measure stemming from adult conceptualisations of well-being, and/or the reading age not being suitable for younger people. Some measures have been developed from the perspective of adolescents albeit with a limited definition of what constitutes well- and ill-being (i.e., according to COSMIN guidelines, more relevant to the participants but not comprehensive in terms of content). Rees et al. [22] for example, chose to define well-being as unidimensional in terms of quality of life and all five items of the CSICSW include “life”, such as “my life is going well”, which does not capture the multidimensional nature of psychological well-being.

Several specific measures exist which capture the various facets of psychological well- and ill-being proposed by Bracey [19]. For example, the Positive and Negative Affect Scale-Child (PANAS-C) [24,25] measures adolescent positive and negative affect which are defined as “general emotional states which have a positive and negative valence and content, respectively” [19]. The Subjective Vitality Scale (SVS) [26] measures vitality which is defined by Bracey as “a positive state of activation marked by dynamic liveliness and energy” [19]. Although developed for adults, a 5-item version of SVS has been adopted in research with children [27]. Less research attention has been paid to serenity (“a positive state of deactivation marked by composed tranquillity and inner peace” [19]) as an indicator of psychological well-being. Studies that have targeted serenity in adults have employed the 65-item Serenity Scale (SS) [28] or the 22-item Brief Serenity Scale (BSS) [29], although no study has adopted either scale when measuring serenity in young people. With regards to lethargy, defined as “a negative state of deactivation marked by apathetic lethargy and ennui” [19], to date there are no measures that fit this definition. Some researchers have used single items such as “did you feel more tired yesterday or today than you usually do?” (Memorial Symptom Assessment Scale (MSAS 7–12) [30], and “how often do you feel tired when you go to school in the morning?” [31]. However, these items are a better measure of tiredness, rather than lethargy. Another issue is that studies often use lethargy interchangeably with fatigue;. whereas lethargy is a temporary “negative state of deactivation” [19], fatigue is extreme “tiredness resulting from physical or mental exertion or illness” [32]. This suggests measures of fatigue are not a suitable replacement for measures of lethargy. Therefore, a measure which focuses solely on lethargy as defined within Bracey’s [19] work and as experienced by young people is needed.

The present study

Previous research has utilised a variety of scales and questionnaires to capture indicators of young people’s psychological well- and ill-being. Although many scales have acceptable psychometrics, key limitations are discussed above. Moreover, to date, no scale exists that we are aware of that captures the multidimensional nature of psychological well- and ill-being as experienced by young people (in [19]). To begin to address these limitations, the present study used think aloud methods [33] with female adolescents to examine the content validity of four questionnaires that align with the definitions of well- and ill-being developed by Bracey [19]: Positive and Negative Affect Scale-Child (PANAS-C) [25], Subjective Vitality Scale (SVS) [26], and the Brief Serenity Scale (BSS) [29]. As mentioned above, as there are no suitable measures for Bracey’s [19] definition of lethargy, four items were developed in this study (see methods for more detail).

In this study, the focus was specifically on adolescent girls. Girls are more susceptible to lower levels of well-being than boys of the same age [34]. Deighton et al. [34] compared levels of well-being in a sample of 10,889 British adolescents each year for three years (11–14 years old). The results showed significant gender differences as emotional difficulties were higher for girls, behavioural difficulties were higher for boys (although girls’ scores were increasing each year) and subjective well-being was lower for girls [34]. It is clear female adolescents, especially aged around 14 years, are a particularly vulnerable group to these health risks and were therefore the target sample in this study.

Method

Ethics statement

Prior to the study commencing, ethical approval was obtained from the Faculty of Science and Engineering Ethics and Governance Committee at Manchester Metropolitan University, approval number 58960. As participants were children, written parent/guardian consent was obtained first, then the participants’ written assent was obtained. Participants’ identifying information were removed from the transcripts to assure confidentiality and anonymity. Participants were referred to according to identification codes, whereby each unique participant (P) was numbered (e.g., P1).

Transparency and openness

This study’s design and its analysis were not preregistered. All data, analysis code, and research materials are available upon request from the first author. Data were analysed using IBM SPSS Statistics version 29.0 [35].

Participants

Participants (N = 40) were female adolescents from the UK aged 13–14 years old (M = 13.3; SD = 0.46). Data collection started on 05/02/2024 and ended on 12/04/2024. Participants were recruited via emails to gatekeepers at schools (i.e., Headteachers) and sports clubs (i.e., coaches). Potential participants were provided with a: (1) parent/guardian information sheet, (2) parent/guardian consent form, (3) participant information sheet, and (4) participant assent form via onlinesurveys.ac.uk. The information sheets stated that participation was voluntary, interviews would be audio recorded on Microsoft Teams, any quotes included in the analysis would be anonymous, and that participants were free to withdraw from the study at any point. Participants could opt in to a prize draw for a £20 Amazon voucher. Participants were recruited from two schools and two sports clubs in the West Midlands in the UK. Using the Index of Multiple Deprivation tool [36], these locations are 89–92% more deprived than other neighbourhoods in England.

Procedure

In the think aloud method, participants are asked to verbalise their thoughts while completing a task, such as a questionnaire, to reveal their thought processes [33]. There are two variations of think aloud. In concurrent reporting, participants naturally express any thoughts they have about the task without attempting to explain them [33]. In immediate retrospective reporting, the researcher asks the participant questions immediately after the task to elicit additional thoughts [33]. Previous studies have shown that think aloud interviews are useful for identifying usability issues with questionnaires, such as stumbling, rereading and misinterpreting items [37,38]. This study employed immediate retrospective reporting, enabling the researcher to clarify any items that participants may not understand and to gather their suggestions on how to reword problematic items.

Think aloud interviews were conducted either online using Microsoft Teams (N = 1) or in a quiet room at a location convenient to the participant such as their school or sports club (N = 39). All participants completed the questionnaires using onlinesurveys.co.uk on a laptop. At the beginning of the interview, participants completed a short questionnaire to provide demographic information. Participants were then read standardised think aloud instructions adapted from Green and Gilhooly [39] and French, Cooke, McLean, Williams and Sutton [40]. The researcher answered any questions before starting the recording. Next, participants were given a practise task to familiarise themselves with the think aloud process. They completed five randomly selected items of the Me and My School Questionnaire (MMSQ) [41]. The MMSQ was chosen as it has a similar response format to the five questionnaires used in this study. It has also been validated for adolescent use [41]. The participants then completed the think aloud process for the remaining questionnaires.

Power relations, informed consent and confidentiality are the main ethical considerations when interviewing minors [42]. As informed consent was provided by a parent/guardian, assent by the participant, and confidentiality including anonymity assured, to minimise the power relation and influence, when the interview was conducted in person the researcher sat outside the line of sight of the participant. If the participant stopped talking for more than 10 seconds, the researcher told them to “please keep talking”. On average, interviews took 8 minutes 18 seconds (SD = 1 minute 38 seconds). Each transcript was transcribed verbatim.

Materials and measures

Self-report measures were selected to align with Bracey’s [19] definitions of psychological well- and ill-being. To ensure consistency throughout, participants were asked to answer the questionnaires based on their feelings in their general life over the past week.

Positive and Negative Affect.

The 10-item Positive and Negative Affect Scale-Child [25] has five positive affect items (e.g., “happy” and “cheerful”) and five negative affect items (e.g., “sad” and “mad”) which participants rated from 1 (never) to 5 (always).

Vitality.

A 5-item version of the Subjective Vitality Scale [26] was used to measure feelings of psychological energy and vitality. Items such as “I felt I had a lot of energy” and “I looked forward to each day” were rated from 1 (strongly disagree) to 5 (strongly agree).

Serenity.

All nine items from the inner haven subscale of the Brief Serenity Scale [28] were used to measure participants’ serenity. The items (e.g., “I am aware of inner peace” and “when I get upset, I become peaceful by getting in touch with my inner self”) were rated from 1 (never) to 5 (always).

Lethargy.

Where possible, the lethargy scale (LS) items were developed based on responses from participants in Bracey’s [19] study, addressing a key critique that previous measures were created without their involvement. For example, a 15-year-old boy said “sluggish…not want to do anything” [19] which became the items “I was tired a lot” and “I had no energy for things”. Four items were rated from 1 (never) to 5 (always).

Data analysis

Descriptive statistics (means and standard deviations) were calculated using IBM SPSS Statistics (Version 29) [35] to examine the similarity of responses to previous research findings to ensure the think aloud process did not impact how participants answered the questionnaire. The audio recordings were transcribed by the first author. The think aloud transcripts were then thematically analysed using Braun and Clarke’s [43] six-phase guide. The six steps are: (1) familiarizing yourself with your data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and (6) producing the report. Each item was assigned to one of two themes: no problem, or response problem. The response problem theme consisted of four sub-themes: stumbled or reread item, clarified understanding of item, misinterpreted item, and did not understand item. These themes were adapted from French et al. [40]. An item was reviewed if it fell into any of the response problem sub-themes across five or more participants. Finally, the reading age of the reviewed questionnaires were calculated using the Flesch-Kincaid Reading Grade Level (FKRGL) [44] in Microsoft Word. The FKRGL is determined by total number of words, total number of sentences, and total number of syllables [44]. A lower score denotes a lower reading age.

Results

Descriptive statistics

The ethnicity of participants is shown in Table 1. The most common ethnicity was White – British (37.5%) followed by Black – African and Mixed – White and Black Caribbean (12.5% each). Table 2 shows the descriptive statistics for each questionnaire. Any participants who answered “prefer not to answer” were excluded from the analysis of that item. All well-being scores were above the mid-point indicating moderately high levels, and all ill-being scores were below the midpoint indicating moderately low levels.

Download:

Table 1. Ethnicity of participants (Sample 1).

https://doi.org/10.1371/journal.pmen.0000551.t001

Download:

Table 2. Descriptive statistics for each subscale (Sample 1).

https://doi.org/10.1371/journal.pmen.0000551.t002

Thematic analysis

Table 3 shows the number of problems experienced by participants for each questionnaire. A ‘problem’ was an item coded under any of the response problem sub-themes in the thematic analysis. An item was deemed a ‘problematic item’ if this happened across five or more participants. On average, participants experienced 4.4 problems when completing the questionnaires. Only two participants experienced no problems. PANAS-C had a total of five problems across five items, LS had a total of 60 problems across four items, SVS had a total of 13 problems across five items, and the BSS had the most problems with a total of 122 across nine items.

Download:

Table 3. Number of problems for each subscale (Sample 1).

https://doi.org/10.1371/journal.pmen.0000551.t003

Table 4 shows the number of participants experiencing problems per item. Problematic items are shown in bold. Nine out of 28 items had no problems. The item with the most problems was “I felt lethargic” from the LS with 58 problems experienced across 33 participants.

Download:

Table 4. Number and type of problem for each item (Sample 1).

https://doi.org/10.1371/journal.pmen.0000551.t004

Items for review

Items deemed problematic following thematic analysis were reviewed. No items from the PANAS-C needed to be reviewed. One item from the LS, one item from the SVS, and eight items from the BSS were reviewed.

Lethargy Scale.

For the item “I felt lethargic”, 58 problems were experienced across 33 participants. Thirty-one participants explicitly stated that they did not understand what “lethargic” meant. Of these 31 participants, 14 struggled to, or incorrectly pronounced lethargic, and nine participants did not attempt to pronounce “lethargic” before stating they did not know what it meant. These participants opted to point to the item and ask for help. For example, [points to lethargic] “what does that mean?”. The researcher responded by saying “say it out loud, and if you still don’t understand it, tell me”. The participant attempted but still could not pronounce lethargic: “lef..ar… I don’t know” (P7). Participant 37 misunderstood the meaning of lethargic saying “yeah, like sick” after being asked to clarify their understanding of the item by the researcher. Finally, one participant clarified their understanding before answering the item: “what is that, tired?” (P24).

Subjective Vitality Scale.

Six participants had a problem with the item “I nearly always felt alert and awake”. All six participants stumbled or reread the item. For example, “I nearly fell…wait I nearly always felt alert and awake” (P38). Only one participant did not understand the item, “I nearly always felt awake…alert and awake. I don’t understand” (P12).

Brief Serenity Scale.

The most problematic item was “I feel serene”, as 36 participants experienced at least one problem with it. Of these 36 participants, 33 did not know what the word “serene” meant, and three participants had to clarify their understanding of serene. Responses included “I dunno what serene is” (P3) and “what’s serene? Calm?” (P21). Moreover, 18 participants stumbled or had to reread the item. The difficulty in pronouncing the word “serene” is demonstrated by participant 1 “I feel sincere. Is that sincere?” and participant 31 “I feel sincere. What does that mean?”.

Seven participants had problems with the item “I’m aware of an inner source of comfort, strength and security”. Five participants did not understand the item, such as participant 7 who said “I’m aware of an inner…inner source of comfort, strength and security. Uh I dunno”. Participant 4 also stumbled and reread the item as they said: “I’m aware of inner source…I am aware of an inner source of comfort, strength and security”. Lastly, one participant misunderstood the item, as they focused only on the “comfort” aspect of the item, neglecting “strength and security” from their thought process for their answer. They asked “what does that mean? … does that mean like I have my comforts and that sort of stuff?” (P18).

The item “during troubled times I experience an inner source of strength” was problematic for seven participants, with a total of nine problems. Three participants did not understand the item, for example “what? I don’t understand” (P12). Six participants stumbled or reread the question, such as “during trouble…troubled times I…I experience an inner source of strength” (P7) and “during troubled timed. Wait, during troubled times I experience an inner source of strength” (P9).

There was a total of six problems for the item “I experience peace of mind”. Two participants had to clarify their understanding, two participants stumbled or had to reread the item, and two participants did not understand the item. Example quotes include “umm. I experience peace of mind. What’s that? Like peace in your mind?” (P1), “I experience peace of mind…hmm… wait… not sure” (P5), and “I experience peace of mind. Does that mean like peace?” (P12).

Eight participants had a problem with the item “I am aware of inner peace”. Six of these participants did not understand the item, including confusion regarding what “inner peace” meant such as “what’s inner peace?” (P30). Two participants had to clarify their understanding of the item. For example, participant 22 said “what does it mean by aware, because like I know it, but I don’t have it… well I am aware that I have it because if someone says sorry then I’m like peace”. Participant 10 clarified their understanding by asking “does that mean like peaceful on the inside?”.

Another problematic item was “I experience an inner quiet that does not depend on events”. A total of 17 participants experienced a problem, with two participants rereading the item, one participant misinterpreting the item, and 14 participants not understanding the item. Problems occurred due to both the “inner quiet” aspect and the “does not depend on events” aspect of the item. For example, “does that mean your body is like quiet?” (P18) and “the end bit. Like the inner quiet, that’s what’s happening in your mind. But the end bit” (P31).

Three participants did not understand the item “when I get upset, I become peaceful by getting in touch with my inner self”, and five participants stumbled or had to reread the item. For example, participant 1 stumbled by saying “and when I’m…when I get upset, I become peaceful and get in touch with my inner self.” Participant 12 read the item correctly and then said “no, don’t understand”. Similarly, participant 28 had difficulty as they said, “what does that mean?”.

Finally, seven problems occurred across six participants with the item “I can feel angry and observe my feelings of anger and separate myself from it and still feel an inner peace”. Three problems were due to not understanding the item. For example, “These questions! I don’t know” (P12). Four problems were due to stumbling or rereading the item, as shown by participant 4 who said, “I feel angry and observe…I feel…I can feel angry and observe my feelings of anger and separate myself from it and still feel...” and answered “never” without reading the item in full. Participant 9 stumbled, reread the question, and still did not understand it. They said “wait I need to read that again… I can feel angry and observe my feelings of anger and sep-ar-at myself, separate myself from it and feel and still feel an inner peace.” This was followed by a 3 second pause, then they looked at researcher for help.

Changes to items

The items that required changes are shown in Table 5. Initially participants’ suggestions during the think aloud were used to reword problematic items so they could be more easily understood by young people. Some items however were deemed unsuitable and so were removed.

Download:

Table 5. Changes made to items for review (Sample 1).

https://doi.org/10.1371/journal.pmen.0000551.t005

Lethargy Scale.

It was decided to remove the item “I felt lethargic”. Although participant 4 provided a suitable rewording, “what is that, tired?”, this was too similar to the item “I felt tired a lot”.

Subjective Vitality Scale.

Participants seemed to stumble on the “nearly always” part of the item “I nearly always felt alert and awake”. After asking the six participants that stumbled on “almost always” to reword the item, the most frequent (N = 4) suggestion was “mostly”. So, the item was reworded based on this participant feedback to “I mostly felt alert and awake”.

Brief Serenity Scale.

The suggestions from the participants on rewording the BSS items were discussed and considered by the study’s authors. However, after this process, the original and reworded items were deemed by the researchers as too complex for adolescents. The decision was therefore made to search for an alternative serenity scale whose items fit the definition of serenity used for this study, and were less challenging for young people to read and understand. This is described in the section below.

Revised measure of serenity

Materials

The search for an alternative serenity scale revealed the Child Serenity Scale (CSS) [45] which has 6-items measured on a 3-point Likert scale of “yes”, “more or less”, and “no”. The CSS was originally written in Spanish and developed in Argentina with Argentine children. Although items were available in English [45], it was decided to translate the original Spanish items into English, and then translate them back into English with the assistance of two bilingual researchers. The aim of the CSS is to determine a child’s ability to “regulate emotion and their responses to stressful events” [45]. Although this differs from Bracey’s [19] definition of serenity (“a positive state of deactivation marked by composed tranquillity and inner peace”), the items (e.g., “I am often a calm person”) reflect Bracey’s definition making it, according to COSMIN guidelines [23], a relevant and comprehensive measure of adolescents’ serenity.