Aims and analytic overview

To meet the proposed objectives, the study advances through two principal stages: first, the definition of detailed diagnostic criteria and clinical risk profiles; second, the development of an ML-based CAT version of the instrument with an automated administration and reporting application.

As mentioned above, the definition of specific risk profiles based on HRI dimensions is expected to be particularly valuable, as it has the potential to identify specific risk groups, allowing for the analysis of their distinctive characteristics and associated factors. This, in turn, should lead to the development of more personalized and targeted interventions, fully leveraging the tool’s potential. In parallel, the development of an ML-based CAT version of the tool has the potential to reduce respondent boredom and fatigue thereby increasing enrollment rates and survey adherence, facilitating longitudinal monitoring, simultaneously reducing time, costs, and scoring errors.

Diagnostic categorization

To be diagnostically useful, test scores must be translated into explicit decision criteria. In this regard, this study aims to make two primary contributions: the establishment of a cut-off score for identifying hikikomori risk using receiver operating characteristic (ROC) analysis; and the delineation of latent profiles derived from the HRI-15 dimensions through latent profile analysis (LPA).

Establishing a cut-off score for classifying at-risk individuals is critical to enhancing the practical value of any screening or diagnostic instrument. Operationally, ROC analysis provides a rigorous procedure for determining, among all possible test scores, the threshold that most effectively differentiates individuals with and without a specific condition [55]. ROC analyses require two inputs: observed scores on the target measure and an external criterion used as a pragmatic indicator of whether cases meet the target condition. For each possible score, sensitivity (true positive rate) and specificity (true negative rate) are estimated, and the optimal cut-off score is selected as the score that maximizes both [55]. In this study, consistent with the literature indicating a general factor underlying the HRI-15, the total scale score was used as the target measure to establish the optimal cut-off score for the scale [11]. The external criterion was a self-report item assessing how often participants locked themselves in their rooms during the past six months. Responses were recorded on a frequency scale and dichotomized as not at risk (“Never”, “Only once”, “1–2 times per month”) or at risk (“Every week (but not every day)”, “Every day”). This item was considered appropriate for calibrating a practical screening threshold because it directly reflects a core behavioral manifestation of prolonged social withdrawal. Previous work on the HRI-15, using a similar procedure, identified a cut-off score (≥ 37) with encouraging performance (i.e., AUC = 0.80; sensitivity = 0.78; specificity = 0.71; accuracy = 0.72; [11]); however, those analyses relied on a binary external criterion (“Have you ever experienced the tendency to lock yourself in your room for several months, never going out, not even to eat meals or to entertain social relations?”; yes/no). The present study advances this evidence by using a larger sample and a more detailed, frequency-based criterion, thereby enabling a more behaviorally specific cut-off calibration and a more detailed assessment of classification performance.

While establishing precise cut-off scores is crucial for clinical practice, these values are confined to reflecting only the total score. Although suitable for preliminary screening, this approach overlooks potentially meaningful profile differences, thereby limiting the development of individualized interventions and the advancement of understanding of the phenomenon. To address this limitation, the present study proposes a novel contribution through LPA. LPA is a person-centered, model-based approach that partitions a population into latent groups characterized by within-class homogeneity and between-class heterogeneity on a set of indicators [56,57]. It is a rigorous method that yields clinically meaningful information, which in turn can facilitate the development of more individualized prevention and treatment approaches [58,59]. Specifically, in this work, LPA was used to detect latent profiles based on the five HRI-15 dimensions. The emerging profiles were also examined against a set of psychosocial indicators (e.g., anxiety, depression, impulsivity, substance use, and related functional outcomes) to assess their clinical utility. Notably, this profile-based characterization has not yet been undertaken for the HRI; however, it has the potential to substantially expand the instrument’s utility by fully leveraging its value for screening and interventions.

CAT development

In clinical and research contexts, assessment batteries often incorporate multiple instruments to gain a nuanced understanding of individuals and populations. While this approach provides valuable insights, it also increases the workload for respondents, professionals, and organizations. The use of brief scales can be an effective strategy for managing this issue. However, when many constructs need to be evaluated, the overall assessment length can still be quite extensive, even if each construct is measured with a brief scale. CAT is an innovative and effective method for improving measurement efficiency without compromising accuracy [50,51]. Computerized adaptive tests (CATs) personalize the testing experience by presenting respondents with a reduced number of items selected dynamically for each person based on item properties and the responses progressively provided by participants during the testing process. Consequently, unlike fixed-form tests, CATs do not administer the entire set of items to each respondent, but focus on a restricted set of items that are most informative for each respondent at a given time [60,61]. Most CATs originate in the framework of item response theory (IRT) and have demonstrated notable efficiency in in various fields, including education, mental health and personality assessment, as they facilitate the precise measurement of disorder severity using a markedly limited number of items [62–66].

More recent approaches to CAT development draw on ML algorithms based on recursive binary partitioning, such as Classification and Regression Trees (CART). Compared to IRT-based procedures, these methods are generally easier to implement because they do not require calibration of large item banks and adherence to strict distribution or dimensionality assumptions (e.g., unidimensionality). Furthermore, their straightforward integration into administration systems and provision of transparent, clinically relevant decision rules enhance their utility and appeal in psychological and psychodiagnostic settings [34,46,52–54,67,68]. Early work in this area focused on procedures that assigned respondents to discrete categories (e.g., diagnosed vs. not diagnosed; at risk vs. not at risk). This framework is known as computerized adaptive diagnosis (CAD; [46]). Recently, however, CART-based approaches have been extended to estimate continuous outcomes, such as test scores, disorder severity, and trait levels. This enables a more nuanced quantitative assessment that goes beyond simple categorization, thus expanding their range of applications [53,67,69].

In general, CART algorithms operate by developing a tree structure that can be conceptualized as a flowchart defined through a recursive, top-down, divide-and-conquer process that generates a sequence of if-else rules aimed at creating classifications (decision trees, DT) or continuous predictions (regression trees, RT; model trees, MT; [70,71]). The tree comprises a root (the variable from which prediction/classification begins), internal nodes (additional variables considered along the path), branches, and leaves. Branches are constituted by a sequence of nodes, connected through specific decision rules (e.g., a numerical threshold of ≤ 1 vs. > 1 for a 0–5 variable) and leaves correspond to the termination of a branch, representing the final prediction of the defined process (class or score).

As binary partitioning methods, CART algorithms split cases according to the rules coded in the tree structure. The development of the algorithm starts with the selection of a root variable that includes the entire dataset, as well as an associated splitting rule that divides the sample into subsets that are as homogeneous as possible with respect to the outcome (classification or score). The procedure then continues recursively within each subset until specific stopping criteria are met (e.g., maximum depth, minimum node size, or no further gain in homogeneity; [71]). In the context of categorical outcomes, the objective is to maximize node purity, a concept that is typically quantified via information gain and entropy reductions [71–74]. In the context of continuous outcomes, the implementation of splits is based on the objective of minimizing within-node variance. This is frequently operationalized as a reduction in standard deviation [71].

Model definition typically follows a two-step approach. First, the tree is developed using a dataset (training dataset) that contains predictors and outcomes for all cases. Then, its predictive performance is evaluated using an independent dataset (testing dataset) with the same structure, but comprising cases that were not used in the development stage [75,76]. This cross-validation approach provides an estimate of generalizability and practical utility of the learned tree for predicting unseen data.

An emerging body of research indicates that tree-based adaptive procedures are effective in psychometric and diagnostic applications [46,53,67–69,77]. Conceptually, once a tree structure has been developed, it can be used to guide the adaptive administration of tests. Respondents are presented only with items located along the branch to which they are progressively assigned based on their responses [46,52,69,77,78]. This architecture offers several practical advantages. First, the adaptive process employs a precompiled scheme based on the established branching structure, thus allowing for rapid deployment with minimal runtime computation [68]. This contrasts with many IRT-based implementations, which require more complex procedures [60,61]. Second, this approach is less dependent on stringent theoretical assumptions, facilitating its application across clinical, school, and epidemiological contexts [46,54].

Evidence indicates that ML-based CATs perform well for both categorical classification and continuous score estimation, yielding valid and time-efficient assessments that closely align with full-length measures in cross-sectional and longitudinal designs [53,54,69,77–80]. Moreover, agreement with external variables is typically indistinguishable from that obtained with full-scale administrations [53,67].

Building on these foundations, the present study aims to demonstrate that an ML-based CAT can improve psychometric screening and diagnostic assessment by producing clinically meaningful outputs (accurate score estimation, risk-profile categorization, and precise cutoff-based classification) while simultaneously reducing respondent burden and facilitating test administration. This efficiency has the potential to enable large-scale, multi-domain screening, making administration more feasible in real-world settings without compromising accuracy or interpretability. In this context, the HRI-15 is an appropriate test case because it assesses distinct dimensions relevant to hikikomori, allowing the identification of clinically useful risk profiles that can substantially complement the essential cutoff-based classification that remains fundamental for screening. The manuscript details the development of an ML-based CAT and introduces a dedicated administration, scoring, and reporting application that returns dimensional scores in real time, flags cutoff exceedances, assigns LPA-based profiles, and generates a concise, practitioner-oriented report suitable for clinical, educational, and epidemiological contexts.

Participants

Data were drawn from a large, nationally coordinated, school-based survey of Italian adolescents aged 11–17 years. The sampling plan was designed to produce a representative student cohort across the country and employed probabilistic selection with stratification by age band (11–13; 14–17), school type, and geographical area. To achieve at least 4,000 valid questionnaires per age group—and assuming a response rate of 15–20%—676 schools across all Italian regions were invited. Final participation rates were 22.7% for middle schools (ranging 20.0–26.2% across areas) and 22.6% for high schools (range 18.5–26.8%). All regions contributed data, ensuring broad national coverage.

In total, 10,181 questionnaires were submitted. Following data-quality checks, 1,426 records (14.0%) were excluded due to ineligibility (age outside target range), incomplete responses, or duplicate submissions caused by technical issues during online entry. The final analytic dataset thus included 8,755 valid cases.

Written informed consent was obtained from students and their legal guardians prior to participation. Data collection and processing complied with national privacy regulations, guaranteeing respondent anonymity. The study received ethical approval from the National Ethics Committee of the Italian National Institute of Health (Protocol PRE BIO CE 0010655, March 22, 2022) and adhered to the principles of the 1964 Declaration of Helsinki and its subsequent amendments.

The final sample comprised 8,755 students (M age = 14.03 years, SD = 1.98). Of them, 3,623 were middle-school students (41.4%; M age = 11.99, SD = 0.81) and 5,132 were high-school students (58.6%; M age = 15.48, SD = 1.08). Within the high-school subsample, 7.7% attended fine-arts secondary schools, 31.8% vocational institutes, 29.4% technical institutes, and 31.1% other types of high schools. The gender distribution was approximately balanced (4,187 females, 47.8%; 4,291 males, 49.0%), with 277 students (3.2%) not disclosing gender. Most participants reported Italian nationality (N = 7,535; 86.1%).

Measures

All measures were collected using closed-ended items through a computer-assisted web interviewing platform during school hours. Trained supervisors were present to oversee the process. The battery comprised: demographic indicators (age, gender, region), lifestyle and behavioral indicators (e.g., participation in social-media challenges, alcohol use, cannabinoids use), and standardized scales assessing hikikomori risk and related psychosocial constructs.

ROC analysis.

ROC analyses were conducted on the HRI-15 total score using an external criterion based on the item: “Over the past six months, have you stayed in your room for extended periods without going out—not even for meals or social interactions?”. Responses were dichotomized as positive (“Every week, but not every day” or “Every day”) versus negative (“Never” through “One to two times per month”). The dataset was randomly partitioned into a training set (approximately 67% of the total sample) and a test set (remaining 33%). The training set (n = 5,866; 48.9% male; M age = 14.00, SD = 1.99) was used to estimate the ROC curve, compute the area under the curve (AUC) with its 95% confidence intervals, and determine the optimal cut-off via Youden’s index (Sensitivity + Specificity – 1; higher values indicate better combined sensitivity and specificity). The selected threshold was then applied to the independent test data set (n = 2,889; 49.3% male; M age = 14.10, SD = 1.96), to compute sensitivity, specificity, and overall accuracy with their 95% confidence intervals. ROC analyses were performed in the statistical environment R [91] using the pROC package [92]. To examine the robustness of the threshold to alternative operationalizations of the external criterion, sensitivity analyses were conducted using different dichotomizations of the room-confinement item. Specifically, the primary coding defined positivity as responses 4–5 (“Every week (but not every day)” / “Every day”), whereas a stricter coding treated only the response “Every day” as positive. This comparison was used to assess whether the resulting cut-off remained stable under a narrower definition of social withdrawal. Finally, supplementary calibration analyses were conducted by estimating logistic regression models in the training dataset and applying them to the testing dataset to obtain predicted probabilities. Calibration was evaluated by comparing predicted risk probabilities with observed outcomes in the testing dataset. The Brier score was used as a summary index of the overall discrepancy between predicted probabilities and observed outcomes, with lower values indicating better probabilistic accuracy [93]. Calibration intercept and calibration slope were also examined: the ideal values are 0 for the intercept and 1 for the slope; negative or positive intercept values indicate systematic overestimation or underestimation of risk, respectively, whereas slopes below or above 1 indicate overly extreme or overly moderate predicted probabilities, respectively [94].

Latent Profile Analysis (LPA).

LPA was employed to detect latent profiles based on the five HRI-15 scales. LPA is a person-centered approach that classifies individuals into groups defined by comparable response patterns on a set of indicators, providing a holistic view of individual differences that can be especially informative in applied research and clinical practice [95–99]. To date, no latent-profile solution for the HRI-15 has been reported, making this analysis a novel contribution. The analysis was therefore conducted on the full sample to maximize precision and obtain realistic parameter estimates. In particular, a sequence of one- through five-class models was estimated and compared, using standardized scores on the five HRI-15 scales as indicators. To identify the preferred solution, multiple statistical and substantive criteria were considered jointly, including Akaike’s Information Criterion (AIC), the Bayesian Information Criterion (BIC), the sample-size–adjusted BIC (SABIC), and entropy. Entropy is a measure of classification accuracy ranging from 0 to 1. Higher values indicate a clearer separation between classes. Conversely, for AIC, BIC, and SABIC greater accuracy is suggested by lower values [57]. Two comparative tests were also considered for model evaluation: the Vuong–Lo–Mendell–Rubin likelihood ratio test (VLMR; [57]) and the Lo–Mendell–Rubin likelihood ratio test (LMR; [100]). These tests are used to compare a model with C latent classes against a model with C–1 classes. Significant p-values indicate that the model with C classes better fits the data than a more parsimonious one (i.e., the model with one fewer class). In contrast, non-significant p-values suggest retaining the more parsimonious model [57,100]. Finally, class sizes and the interpretability of the resulting profiles were taken into account in selecting the solution to be retained. As mixture-model fit indices often fail to converge on a single optimal solution, no single statistic was considered to be conclusive. Instead, model selection was based on an overall balance of relative model fit, classification quality, parsimony, class size, and interpretability [57,93,97]. Analyses were conducted using Mplus 7.4 [56], with the robust maximum likelihood estimator (MLR; [101]) and mixture specification. Between-class differences on a set of psychosocial indicators were examined using ANOVA and χ² statistics, with eta-squared (η²) and Cramer’s V as effect size measures.

Machine-Learning Computerized Adaptive Test (CAT).

CAT development: A real-data simulation approach with cross-validation was used to develop the CAT. Specifically, a multivariate conditional-inference tree (ctree) was estimated on the training dataset (67%; random split). In the model the 15 items of the HRI-15 [11,44] were used as predictors, while the HRI-15 total score (sum) and the five standardized (M = 0, SD = 1) scores on its subscales (anthropophobia, agoraphobia, paranoia, lethargy, depressive mood) were the joint outcome variables. The ctree algorithm, as with other regression trees, employs recursive binary partitioning [102]. However, in this framework, in contrast to conventional approaches, both variable and splitting rule selection are determined by permutation tests. This allows for the management of multiple outcomes within a single tree and reduces selection bias [103–105]. At each node, the algorithm tests a global null hypothesis of independence between the predictors and the outcome variable(s); if the null hypothesis is not rejected, the node becomes a leaf. If the null hypothesis is rejected, the predictor with the smallest p-value (adjusted for multiplicity) is selected for splitting; and for that predictor, all possible binary partitions are evaluated, choosing the split that maximizes the test statistic across branches. For a single continuous outcome, the split statistic can be interpreted as a standardized difference between the mean outcome values in the two child nodes defined by a candidate split. For ordered outcomes, the comparison is instead based on a linear-rank statistic computed from the outcome scores/ranks. For multivariate outcomes, the statistic combines the differences between the two child nodes across the outcome vector into a single quadratic-form test statistic [11,44]. At each split, specific node-size constraints have to be met. Specifically, the constraints pertain to the minimum number of observations required in the nodes (or the sum of the case weights) to attempt a split (i.e., the minimum split or “minsplit”) and the minimum number of observations that must be included in each leaf (i.e., the minimum bucket or “minbucket”). The thresholds for these parameters are set to balance tree parsimony against predictive accuracy (higher thresholds favor simplicity; lower thresholds allow for a closer fit, at the risk of overfitting). At the end of the branches growing process, leaves provide piecewise-constant predictions representing the mean of the outcome variable(s) in a specific subset (which is equivalent to the intercept of an intercept-only model for that subset). The branches growth stops when no covariate shows sufficient association with the outcome(s) at the prespecified significance threshold (i.e., “mincriterion”). This threshold is the p-value that determines whether a node should be split (it is typically set as 1 − α; e.g., a mincriterion of .95 corresponds to α .05, meaning that p ≤ 0.05 is required for a split) or when node/leaf sample-size constraints (i.e., minsplit/minbucket) are not met; therefore, no pruning or smoothing is required. In the present study, candidate tuning configurations were examined over a broad grid of plausible values for mincriterion (.10, .20, .50, .70, .90), minsplit (10, 20, 50), and minbucket (10, 20, 50). Candidate configurations were first compared by 5-fold cross-validation within the training subset. They were then refitted on the full training subset and evaluated on the corresponding independent test subset to examine out-of-sample performance and the sensitivity of agreement, prediction error, and item savings to alternative hyperparameter settings. The final configuration was chosen to balance predictive accuracy and administration efficiency. Analyses were performed in the statistical environment R [91] using the partykit package [104,106,107] (Fig 1).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the conditional-inference tree procedure and schematic illustration of adaptive routing.

Note. Panel A summarizes the node-wise procedure used to grow the conditional-inference tree. At each node, the algorithm tests the global null hypothesis of independence between the predictors and the outcome(s). If the null hypothesis is rejected, the predictor with the smallest adjusted p value is selected, and the admissible binary split that maximizes the test statistic is chosen; otherwise, the node becomes terminal. Panel B provides a schematic illustration of the resulting adaptive routing structure. The dark nodes and terminal node connected by dashed lines indicate one possible branch of the tree. Starting from the root node, all respondents receive the same initial item; subsequent items depend on the responses provided at each preceding node. This recursive process continues until a terminal node (TN) is reached. Each TN corresponds to a subgroup of respondents defined by a specific response pattern and stores the predicted score(s) associated with that branch.

https://doi.org/10.1371/journal.pone.0355595.g001

Evaluation of CAT performance: Once the tree structure had been defined using the above-described procedure on the training dataset, it was used to instruct an adaptive assessment procedure based on respondents’ answers in the test dataset, for which complete responses were available (i.e., real data simulation). The procedure simulated the administration to participants of only the items that corresponded to the branch they were moving along based on their responses, and it produced total (sum score) and subscale (z-score) score estimates. On this dataset (not used to develop the tree), the CAT performance was evaluated in terms of efficiency and accuracy. The efficiency of the CAT procedure relative to the full-length test was measured by item savings (the average number of items administered and the corresponding reduction percentage) and evaluated using t-tests. Accuracy, interpreted as score reproduction, was quantified using Pearson’s correlation coefficients (r) and mean absolute error (MAE). Correlations indicate the degree to which CAT scores align with those obtained by administering all items (a higher r indicates better performance), while MAE captures the average absolute deviation between CAT and full-length scores (a lower MAE indicates closer alignment). Paired samples t-tests were also run to compare CAT-derived scores with full-length scores. The t-tests assess the null hypothesis that the mean of CAT scores does not differ from that of full-length scores.Agreement between CAT- and full-length scores was assessed also using intraclass correlation coefficients computed on the independent test set for each HRI-15 scale and the total score, with a two-way model, absolute agreement, single measure (ICC(A,1)). Interpretation followed [108]: ICC < .50 poor, .50–.75 moderate, .75–.90 good, > .90 excellent.

The performance of the CAT was also evaluated in terms of its accuracy in classifying risk. The cut-off score derived from the ROC curve estimated on the training data set was applied to both the total scores obtained through the CAT and the total scores obtained from the full-length administration. For both sets of scores, the following metrics were checked: sensitivity, specificity, overall accuracy, and Cohen’s κ coefficients. Cohen’s κ represent chance-corrected agreement between each method’s classifications and the criterion variable (e.g., external variable indicating the tendency to social withdrawn). Higher values indicate that the method’s predicted labels better reflect the actual condition. For the purpose of interpretation, conventional criteria were employed: 0–0.20 mild, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and ≥ 0.81 almost perfect agreement [109]; see also [71]). To directly compare the two methods, we used McNemar’s χ² test (df = 1) on the paired classifications relative to the criterion variable. A p-value less than .05 indicates different error rates between CAT and full-length scores.

Finally, a Latent Transition Analysis (LTA) was conducted to examine the coherence of latent class assignment between the profiles estimated from the full-length and CAT versions of the HRI-15. LTA is a multi-occasion extension of LPA that enables the evaluation of latent profiles across different occasions and the estimation of transition probabilities between them [95]. In this work, the two considered conceptual “occasions” were the two set of scores (i.e., full-length and CAT scores). In the analysis the indicators were the standardized scores on the five HRI-15 dimensions. First, a configural model was specified, holding the structure and number of profiles constant across the two sets of scores, while allowing means and variances of the indicators to be freely estimated by profile and set of scores. This specification permits level differences between full-length and CAT scores while preserving the conceptual alignment of profiles across occasions [95]. Estimation was performed in Mplus 7.4 [56] using the robust maximum likelihood estimator (MLR; [101]) within a mixture-model framework, with time-2 classes regressed on time-1 classes to obtain the transition matrix (row-standardized transition probabilities from the full-length–based profile to the CAT-based profile). Entropy was used as a classification-quality index (values close to 1 indicate more precise classification), as in the preceding LPA. Concordance between the full-length and CAT solutions at the profile level was also considered. Finally, particular attention was paid to the profile transition probabilities across score type (full-length vs. CAT); low off-diagonal probabilities (i.e., high diagonal stability) were taken as evidence that the CAT procedure yields categorizations comparable to the full-length administration.

Application for adaptive assessment and automatic reporting: A Shiny application [91,110,111] was developed for the adaptive administration of the HRI-15 [11], based on the tree structure derived in the training dataset. Computerized administration follows a deterministic traversal of a pre-populated decision tree. At runtime, two external tables are loaded: (i) a tree sheet that lists the internal nodes, the splitting rules, and the outputs of the terminal leaves (i.e., sum score, subscale z-score, and LPA derived profile); and (ii) a dictionary that maps item codes to the item text and scale to which they belong.

Administration begins at the root (the first non-leaf node not referenced as a child). At each step, the engine presents the item indicated by the current node’s variable, records the response, evaluates the split, and advances to the next item based on the response and tree structure. If a subsequent node requests a variable that has already been answered, the engine automatically advances using the stored response until a new variable is requested or a terminal leaf is reached. The stopping rule is the first terminal leaf encountered. The application then returns leaf-specific predictions: the HRI-15 total score, the z-scores of the five subscales, and the latent class. The results are immediately displayed in a concise, profile-based report. The total score is interpreted based on the ROC-derived decision rule (cutoff ≥ 42), and the dimensional profile is graphically represented to support a two-step workflow in the applied settings: threshold-based triage followed by profile-driven personalization. The system is implemented in Shiny and relies on external files (Excel) for both the tree structure and the dictionary, allowing content to be updated without code changes. Main dependencies include shiny, readxl, dplyr, stringr, and ggplot2, along with custom functions for parsing and matching of path rules. The interface presents standardized instructions and, for each item, and a short guide to interpret results.

Receiver operating characteristic (ROC) analysis

ROC analysis was conducted on the training dataset in order to identify the optimal cut-off score for the instrument. This analysis was then validated on the testing dataset, with the HRI-15 total score serving as the dependent variable and the item on the tendency to self-confine over the past six months as the external criterion (recoded as positive for responses 4–5: “Every week (but not every day)”/ “Every day”). According to this criterion, there were 374 positive cases in the training dataset and 184 in the testing dataset. In the training data set, the area under the ROC curve (AUC) was .846, CI [.826, .865] (Fig 2). The Youden index indicated an optimal cut-off score of ≥ 42. The performance metrics achieved using this cut-off score were .751, .791, and .748 for accuracy, sensitivity, and specificity, respectively. The corresponding 95% confidence intervals were [.740, .762] for accuracy, [.747, .832] for sensitivity, and [.737, .760] for specificity. When the same cut-off score was applied to the testing dataset, the corresponding values were AUC = .837, 95% CI [.809, .865], accuracy = .726, 95% CI [.710, .742], sensitivity = .783, 95% CI [.716, .840], and specificity = .722, 95% CI [.705, .739] (further details are reported in S1 and S2 Tables in the S1 File). In summary, a cut-off score of ≥ 42 showed acceptable ability to identify individuals at risk.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. ROC on Training dataset (N = 5,866).

Note. Area under the curve was .846.

https://doi.org/10.1371/journal.pone.0355595.g002

A sensitivity analysis using a stricter criterion definition, in which only the response “Every day” was coded as positive, yielded the same optimal integer cut-off score (≥ 42). In the training dataset, the AUC was .819, 95% CI [.780, .858], with accuracy = .725, 95% CI [.713, .736], sensitivity = .771, 95% CI [.685, .843], and specificity = .724, 95% CI [.712, .735]. In the testing dataset, the corresponding values were AUC = .794, 95% CI [.731, .858], accuracy = .700, 95% CI [.683, .717], sensitivity = .764, 95% CI [.630, .868], and specificity = .699, 95% CI [.682, .716]. Overall, the convergence of the primary and sensitivity analyses supported the robustness of the ≥ 42 threshold.

Supplementary calibration analyses based on logistic regression models estimated in the training dataset indicated acceptable agreement between predicted and observed risk probabilities in the testing dataset. In the primary analysis, the Brier score was 0.052, the calibration intercept was −0.087, and the calibration slope was 0.941, indicating only modest deviations from ideal calibration (see Supplementary Tables S3–S5 Tables and S6 Fig in S1 File).

Latent profile analysis

A total of five models, with one to five classes, were estimated. The results were evaluated using a combination of statistical and qualitative criteria (Table 1). The information criteria (i.e., AIC, BIC and SABIC) decreased sharply up to four classes, whereas the improvement from four to five classes was more modest. Entropy was highest for the four-class solution (0.877), indicating clearer classification. The VLMR/LMR tests were significant at the transitions from 2 to 3, 3–4, and 4–5 classes (p < 0.01), indicating continued improvement in fit with increasing model complexity. However, the incremental improvement from four to five classes was comparatively limited, whereas the four-class model yielded the clearest classification and an interpretable configuration with well-defined class proportions (52.4%, 17.5%, 15.0%, and 15.2%). The four-profile solution was therefore retained as the preferred solution, as it offered the best balance among fit improvement, interpretability, parsimony, and classification quality. Additional classification diagnostics for the retained solution, including average posterior probabilities and class-specific parameter estimates, are reported in (S7–S10 Tables in S1 File). Supplementary robustness analyses conducted in the training and testing subsets also supported the stability of the four-profile configuration (S11 Table and S12 Fig in S1 File).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Fit indices for latent profile analysis.

https://doi.org/10.1371/journal.pone.0355595.t001

Fig 3 illustrates the profiles derived from the four-class solution, which clearly show distinct profiles. The first group, which is by far the largest, exhibits sub-mean levels across all dimensions, indicating the absence of clinically significant signs of social withdrawal. The second group is primarily characterized by a depressed mood and low energy levels. Social avoidance (agoraphobia and anthropophobia) is a secondary characteristic, while suspicion or distrust is minimally present. This pattern can be regarded as a form of “secondary” withdrawal, where social disengagement is a consequence of pervasive sadness and reduced activity. The third profile most closely aligns with the true hikikomori pattern, which is characterized by social avoidance alongside symptoms such as paranoia, lethargy, and depressed mood, indicating a more pronounced withdrawal syndrome. The fourth group is characterized by anxiety-driven withdrawal, where the main challenges involve social interactions and environmental contexts (agoraphobia and anthropophobia), while depressed mood and low energy levels are comparatively less pronounced. In essence, the identified groups appear to distinguish between individuals without clinical indicators (Class 1, “non-at-risk”); individuals whose withdrawal is predominantly mood-driven (Class 2, “Depressive–lethargic”); individuals exhibiting a characteristic hikikomori pattern (Class 3, “at-risk”); and individuals whose withdrawal is primarily influenced by phobic and social anxiety traits (Class 4, “Social-anxiety/phobic”).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Profiles emerging from latent profile analysis.

Note. Bars represent standardized scores on Anthropophobia, Agoraphobia, Paranoia, Lethargy, and Depressive Mood for the profiles identified via latent profile analysis (LPA). Positive values indicate scores above the sample means; negative values indicate scores below the sample means. The four profiles are: (1) Non-at-risk, (2) Mood-driven, (3) At-risk, and (4) Social anxiety/phobic. Scores are standardized within-sample (M = 0, SD = 1). Within-class standard deviations were 0.585 for Anthropophobia, 0.547 for Agoraphobia, 0.806 for Paranoia, 0.702 for Lethargy, and 0.499 for Depressive Mood; these values were constrained to be equal across classes in the retained model.

https://doi.org/10.1371/journal.pone.0355595.g003

Differences between profiles extended systematically to the other psychological variables and risk behaviors considered. ANOVAs revealed large, significant differences for depressive symptoms (F(3, 2742) = 2496.7, p < .001, η² = .518) and social anxiety (F(3, 2688) = 2665.6, p < .001, η² = .545), as well as medium differences for impulsivity (F(3, 3055) = 301.4, p < .001, η² = .103). As shown in Fig 3, Class 1 (i.e., the non-at-risk profile) displays below-average levels of social anxiety, depression and impulsivity. Class 2 (i.e., the mood profile) shows slightly above-average anxiety and impulsivity, and moderately elevated depression. Class 3 (i.e., the at-risk profile) combines high anxiety and depression with moderately high impulsivity, constituting the highest-risk profile. Class 4 (i.e., the social anxiety profile) is characterized by above-average social anxiety. Overall, this pattern aligns with the expected profile for each class (Fig 4).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Depression, anxiety, and impulsivity by LPA-derived profile.

Note. Bars represent standardized (z) scores for depression, anxiety, and trait impulsivity across the four profiles identified by latent profile analysis (LPA). Higher values indicate greater symptom severity/impulsivity. Scores are standardized within-sample (M = 0, SD = 1). Error bars show 95% confidence intervals around the mean; substantial overlap suggests non-significant differences. Post hoc pairwise comparisons indicated that all between-class differences were statistically significant (p < .01).

https://doi.org/10.1371/journal.pone.0355595.g004

Notable differences across profiles were also observed concerning risky behaviors and habits (Tables 2 and 3). For instance, daily engagement with social media and video games use varied systematically according to the hikikomori risk profile. Regarding gaming, the χ² test yielded a significant result (χ² = 145, p < .001, V = .074), indicating a small effect size. For social media use, the association was more pronounced (χ² = 580, p < .001, V = .149), indicating a small to medium effect size. Examination of standardized residuals (|z| > 1.96) indicated that profiles characterized by depressed mood and those defined as “at-risk” (Classes 2–3) were more likely to report high daily social media engagement (from ≥ 3 to > 9 hours/day), whereas “non-at-risk” and “social anxiety” profiles (Classes 1 and 4) more frequently reported low levels of use. With respect to gaming, very intensive engagement (7–8 hours/day) was more likely in Class 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Associations between LPA-derived profiles and behavioral indicators.

https://doi.org/10.1371/journal.pone.0355595.t002

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Associations between LPA-derived profiles and daily video-game and social-media use.

https://doi.org/10.1371/journal.pone.0355595.t003

Participation in social media challenges was weakly but significantly associated with class membership (χ²(3) = 9.92, p < .05, V = .034). Standardized residuals indicated greater participation among individuals categorized on the “depressive” profile (Class 2). Regarding sexting, assessed only in the high-school subsample, small but statistically significant differences emerged across classes (χ²(3) = 81.10, p < .001, V = .126), with the “non-at-risk” group (Class 1) showing a lower propensity to send erotic/sexual content, and the depressive and “at-risk” groups (Classes 2 and 3) showing a higher propensity.

With respect to substance use, a small-to-moderate association was found between alcohol consumption and class membership (χ²(3) = 321, p < .001, V = .192), whereas cannabis use demonstrated a significant but small association (χ²(3) = 77.10, p < .001, V = .123). Standardized residuals indicated that the probability of alcohol use was higher in classes characterized by depressive mood and in the “at-risk” group (Classes 2 and 3) but lower in the “non-risk” and “anxious” profiles (Classes 1 and 4). Cannabis use was more likely in the depressive class (Class 2) and less likely in the anxious-phobic class (Class 4).

Finally, the protective tendency to seek support by talking with parents or caregivers differed markedly across profiles (χ²(3) = 1119, p < .001, V = .357). Standardized residuals indicated that Classes 2, 3, and 4 were more inclined to speak with reference figures, whereas only the “non-at-risk” profile (Class 1) was less inclined to do so.

It is interesting to note that the profile membership from LPA was associated with the risk status both relative to the criterion variable (χ²(3) = 284, p < .001, Cramér’s V = .313) and to the dichotomous classification derived from the ROC cut-off score (χ²(3) = 1833, p < .001, Cramér’s V = .796). The pattern was consistent in the testing dataset and in the total sample. In the testing dataset, standardized residuals (|z| > 1.96) indicated that Class 3 (i.e., “at-risk” profile) was more likely to be positive and to be classified as positive (ps < .001), whereas Class 1 (i.e., “non-at-risk” profile) was more likely to be negative and to be classified as negative (ps < .001). Classes 2 and 4 (i.e., “depressive” and “anxious” profiles) were more likely to be classified as at-risk (ps < .001; in the total sample Class 2 also showed a higher likelihood of being positive according to the criterion, p < .001) (Table 4).

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Associations of LPA-derived profiles with the external Criterion and ROC-based classification.

https://doi.org/10.1371/journal.pone.0355595.t004

CAT performance

The ctree algorithm was tuned on the training data set by examining a grid of plausible hyperparameter settings for mincriterion, minsplit, and minbucket, taking into account 5-fold cross-validated performance together with item savings and tree complexity. The cross-validation and sensitivity analyses indicated that the configuration mincriterion = .10, minsplit = 10, and minbucket = 10 provided a balanced compromise between predictive accuracy and administration efficiency (detailed tuning and sensitivity results are reported in Tables S13 to S16 in the S1 File).

When refitted on the full training data set with the selected hyperparameter configuration, the ctree algorithm yielded a tree structure with 349 leaves, with item 15 (“I feel a sense of inner emptiness”) at the root. Leaves stored predictions for the HRI-15 total score and the standardized scores on the five subscales; branch depth ranged from 3 to 14 nodes.

This structure was then applied to the independent testing set to drive adaptive administration. The number of administered items ranged from 3 to 14, with an average of 7.03 items, corresponding to a 53.13% saving relative to the full-length version (15 items). A t-test on the number of administered items confirmed that the mean was significantly lower than 15 (t(2888) = −143, p < .001, d = −2.66).

Regarding accuracy, agreement between scores obtained with full administration and those estimated adaptively was high: correlations ranged from .767 to .997, with good to excellent values of ICC (from .751 to .997) and low MAEs (Table 5). In addition, paired-samples t-tests indicated no significant differences between full-length and CAT scores on any scale. Moreover, applying the ROC-based cutoff score to the two sets of score yielded comparable classification performance: CAT scores achieved .73 accuracy, with sensitivity .73, specificity .73 and Cohen’s κ .17 (mild), while full-length scores achieved the same overall accuracy (.73) with sensitivity .78, specificity .72, and κ .18 (mild). Consistent with this, McNemar’s test showed no significant difference in error rates when applying the cutoff of ≥ 42 to CAT or full-length scores, χ²(1) = .194, p = .66 (further details on administration diagnostics, CAT classification, and score-discrepancy analyses between CAT-derived and full-length scores are reported in Tables S17–S23 in the S1 File).

Further evidence for the comparability of CAT and full-length scores came from a latent transition analysis (LTA). The five HRI-15 dimensions served as indicators at two occasions (full-length scores represent the first occasion; and CAT scores the second). A configural LTA model with four profiles per occasion was estimated. For interpretability, CAT-based classes were remapped to align with full-length-based profile meanings based on class-specific means. The results showed excellent classification quality (entropy = .959; AIC = 53,180.38; BIC = 53,568.35; SABIC = 53,361.82; N = 2,889). Profile-level agreement between full-length and CAT profiles was high (diagonal transition probabilities from .927 to .954; M = .943), indicating that the CAT yields categorizations comparable to the full-length administration (Table 6). Profile shapes were substantially superimposable across occasions (Fig 5), supporting conceptual alignment.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Agreement between full-length and CAT scores on the HRI-15: descriptive and comparative statistics (N = 2,889; testing dataset).

https://doi.org/10.1371/journal.pone.0355595.t005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Profiles emerging from LTA based on full-length test and CAT administration (N = 2,889; testing dataset).

Note. Profile means from full-length test (solid) and CAT administration (dashed) across the five HRI dimensions. Four profiles are shown: (1) Non-at-risk, (2) Depressive–lethargic, (3) At-risk, and (4) Social-anxiety/phobic. Lines are plotted in black with distinct markers per profile.

https://doi.org/10.1371/journal.pone.0355595.g005

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Latent transition probabilities.

https://doi.org/10.1371/journal.pone.0355595.t006

Development of App

The app is provided in the S1 File. It administers the test adaptively and returns results in real time, including: (a) the total score on the general scale, (b) standardized scores for each subdimension, (c) a bar chart displaying the subdimension scores, and (d) an indicator of whether the ROC-based cut-off is exceeded, (e) along with profile membership according to the LPA classification.

The home screen presents instructions; users proceed by tapping “Start test” (Image A, Fig 6). This initiates item administration: respondents select an option, which routes them to the subsequent item (Image B, Fig 6). Instructions remain visible on the page, and a button is available to restart the assessment if needed.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Shiny app for adaptive administration and automatic reporting of the HRI-15.

Note. Screens from the adaptive assessment application (S1 File). A) Landing page with standardized instructions and “Start test.” B) Item screen showing a single prompt with five response buttons (1–5; “Not true for me” to “Very true for me”). Items are administered adaptively based on prior answers; instructions remain visible and a “Restart” option is available. C) Results screen displaying the total score, standardized subscale scores (z), LPA-based profile assignment, a ROC–cutoff risk flag, and a bar chart of subscale scores. Subscale scores are standardized within-sample (M = 0, SD = 1). The risk indicator reflects the ROC-derived cutoff (≥ 42). Color coding of the flag is not visible in grayscale of this screen.

https://doi.org/10.1371/journal.pone.0355595.g006

Upon completion, a results screen displays the scores, guidance for interpretation, and a bar chart of dimensional scores (Image C, Fig 6).