Burden and public-health significance

Presenteeism, broadly defined as attending work despite physical or mental health problems, is increasingly viewed as a major contributor to productivity loss and diminished well-being. Early empirical work suggested that working while ill is common in working populations and is shaped by occupational conditions (e.g., replaceability and job demands) in addition to health status [1]. Building on this, later syntheses emphasize that presenteeism is not only prevalent but also consequential. A longitudinal review of sickness presenteeism reports that repeated or sustained presenteeism is associated with adverse health and well-being over time, indicating that it is more than a short-term coping behavior [2]. Accordingly, reviews and critical overviews position mental health as a central upstream factor for workplace productivity and highlight that the productivity burden is mediated through both absenteeism and presenteeism pathways [3,4].

Although the burden is observed internationally, its magnitude and expression depend on local measurement practices and organizational context. For example, studies in both workplace and clinical settings quantify productivity loss by jointly considering absenteeism and presenteeism, reinforcing that the key outcome is functional impact rather than symptoms alone [5,6]. In Japan, employer- and cohort-based analyses further indicate that productivity loss attributable to presenteeism can account for a substantial share of total health-related costs [7–9]. Longitudinal evidence also suggests that psychological distress can be followed by persistent productivity impairment, implying that missed early signals may translate into prolonged functional loss [10,11]. These findings motivate scalable, low-burden approaches for early identification of work-function impairment in occupational settings.

Limitations of current detection and monitoring approaches

Existing detection and monitoring approaches largely rely on self-report instruments, periodic screenings, and occasional specialist assessments. While useful, these approaches have three persistent limitations in real-workplace deployment: (i) bias and introspective limits in self-report, (ii) high burden and low temporal resolution, and (iii) construct heterogeneity and context dependence.

First, self-reports can systematically underestimate strain. Organizational research suggests that employees may defensively downplay stress and strain, implying divergence between reported states and actual burden even when an instrument is well designed [12]. More broadly, self-report presupposes accurate introspection, yet perceived “being able to work” may not align with externally observable functional impairment.

Second, questionnaires and screenings impose burden and therefore cannot be deployed at high frequency. In practice, completing multi-item instruments can be time-consuming and disruptive, making daily or near-daily administration unrealistic in many workplaces. Consequently, periodic screening tends to capture only coarse changes and can miss shorter-term within-person dynamics of work limitation and functioning [13]. Specialist-led assessments provide richer judgment but are costly and difficult to maintain routinely at scale.

Third, heterogeneity in measurement and definition complicates both inference and implementation. Two major constructs—“sickness presenteeism” (working despite illness) and “impaired work function” (reduced functional capacity while at work)—are related but not equivalent, and estimates based on them are not interchangeable [14]. Moreover, organizational context can reshape both reporting and attendance behavior, which can alter the validity of self-report and presence-based indicators. Structural factors such as job flexibility and job security further affect mental-health outcomes and work attendance, limiting one-size-fits-all monitoring schemes [15]. This conceptual non-equivalence complicates benchmarking, label design, and the interpretation of predictive models, particularly when models are trained on symptom proxies rather than direct assessments of functioning during work.

Taken together, these limitations argue for objective, low-burden, and higher-frequency indicators that complement questionnaires and periodic assessments. Such indicators should be feasible in naturalistic work settings and sufficiently interpretable to support screening and initial response decisions in occupational health.

Advances in digital and multimodal AI for work-function assessment

Digital phenotyping and multimodal machine learning (MML) provide a technical foundation for translating everyday behavioral signals into quantitative indicators. Single-modality approaches (e.g., speech or facial cues) can capture mental-state–related variation, but they are vulnerable to missingness and context in short, everyday recordings. Multimodal fusion can integrate complementary cues and stabilize inference under such constraints [16,17]. At the deployment layer, smartphone-based sensing studies illustrate low-burden designs that can be embedded into daily routines [18,19].

However, prior multimodal studies have primarily targeted proximal labels such as affective states, stress markers, or questionnaire-derived severity scores [16,17,20,21], which are not direct assessments of work functioning during active employment. This leaves a methodological disconnect between advances in multimodal behavioral modeling and psychiatrist-consensus–grounded assessment of work-function impairment.

Study aim and methodological contributions

Motivated by this methodological gap, we develop and evaluate a psychiatrist-consensus–anchored multimodal model that estimates ordinal work-function impairment, interpreted here as a functional manifestation of presenteeism in occupational settings, from brief smartphone-based dialogues. Rather than predicting affective states or questionnaire scores, the proposed model is directly supervised by psychiatrist consensus ratings of functional impairment during work.

The model integrates linguistic content with nonverbal behavioral cues, including vocal prosody, facial signals, and head-motion dynamics, and is designed for short interactions (10–30 s) and low-burden capture in naturalistic workplace settings. Conceptually, multimodal behavioral signals captured by smartphones are treated as a low-burden statistical approximation of psychiatrist judgment, not as a replacement for clinical assessment.

Our contributions are threefold. First, we show that brief smartphone-recorded dialogues contain sufficient multimodal information to approximate psychiatrist consensus ratings of work-function impairment. Second, by explicitly modeling ordinal severity, the proposed approach preserves graded transitions between functional states, enabling more fine-grained assessment than nominal classification. Third, we analyze modality contributions and disagreement cases to clarify how nonverbal cues refine decision boundaries in conjunction with language.

Study design and data collection

This prospective observational study employed a custom iOS application installed on company-issued iPhones to collect short self-report videos (front camera; face and speech) and ecological momentary assessments (EMA) in naturalistic work settings. Fig 1 provides an end-to-end overview of the data-collection and analysis workflow, including on-device prompting, secure upload, and offline processing for model development.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the study workflow.

Participants responded to short video prompts and EMA items via a custom iOS application in daily work settings. Data were uploaded through encrypted channels to a study server and processed offline to extract multimodal behavioral signals and train the proposed models. Screenshots show the application interface as implemented in the study.

https://doi.org/10.1371/journal.pone.0354342.g001

At first launch, participants completed a one-time baseline questionnaire covering demographics, personality, sleep, and psychological distress (S5 Table). At each prompt thereafter, participants positioned their face within an on-screen frame, recorded a 10–30 s video reply to “How are you feeling today?”, and answered brief EMA items (S6 Table). All responses were transmitted via encrypted channels to a study server for storage and preprocessing. Collected media were subsequently processed offline to extract facial, acoustic, and linguistic features and to train and evaluate multimodal models (Fig 1).

Daily sampling followed standard EMA practice [22]. Affect was recorded using Russell’s circumplex coordinates (valencearousal) [23,24] and is summarized descriptively in S6 Table. Baseline psychometric scales and daily EMA items were collected to support contextual understanding during psychiatrist consensus labeling and for descriptive characterization of the cohort (S5 and S6 Tables). They were not incorporated as input features in the machine-learning models.

Participants were white-collar employees from a single company (enrolled , age 20–60 years), including 13 women. Of these, provided valid video submissions (mean age = 41.4 years); the subset entering the supervised analysis (after feature-extraction and labeling availability, detailed in Feature extraction and preprocessing) was . Participation was voluntary, with written informed consent obtained from all participants. Data were collected from February 15 to May 20, 2024 (approximately 3 months), during which participants were asked to submit at least two responses per day (pre- and post-work). The protocol was approved by the Institutional Review Board of the University of Tsukuba (Approval No. 1662−6). Cohort composition and engagement are summarized descriptively in the Results section.

Baseline and ecological momentary assessments

At study entry, participants completed a baseline questionnaire comprising demographics and three validated psychometric scales: TIPI/TIPI-J [25,26], AIS [27,28], and K6 [29]; full items and response formats are in S5 Table. During the study period, daily ecological momentary assessments (EMAs) captured location, work status, previous-night sleep, affect on Russell’s circumplex [23,24], workload factors, self-rated job performance, and working hours (full items in S6 Table). Both the baseline questionnaire and the EMA were used to characterise the cohort descriptively and to provide contextual reference information to psychiatrists during labeling; they were not used as predictive input features in the multimodal models.

Psychiatrist consensus labeling

Each valid video clip was independently reviewed by three psychiatrist raters together with the corresponding EMA responses. Raters assigned one of three ordinal labels representing work-function impairment, which we interpret as an indicator of presenteeism: 0 = Healthy, 1 = Moderate, and 2 = Unwell. To reduce ambiguity and enhance consistency, raters were instructed to base their judgments primarily on observable behavior in the video, including facial expressions, speech characteristics, and spoken content. Information from baseline questionnaires and EMA responses was used only as secondary contextual cues to support interpretation when needed.

All three raters were board-certified psychiatrists with occupational mental health expertise (one Designated Mental Health Physician with additional social-medicine credentials; two certified occupational physicians).

A single psychiatrist consensus label was obtained for each clip by majority vote across the three ratings. Clips showing complete dispersion (i.e., one rater assigned each of the three classes—Healthy, Moderate, and Unwell) were regarded as ambiguous and excluded from both model training and evaluation; this criterion removed 10 clips ( of the 1,791 labeling-eligible clips), leaving 1,781 labeled clips before the participant-level inclusion criterion described in Feature extraction and preprocessing was applied.

Inter-rater agreement across the three psychiatrists was substantial (Fleiss’ ; pairwise Cohen’s [R1–R2], 0.819 [R1–R3], 0.617 [R2–R3]; raw agreement 81.5%/90.5%/80.0%), supporting the validity of aggregating individual ratings via majority vote. Accordingly, the majority-vote label was adopted as the ordinal target for supervised learning. The structure of inter-rater disagreement and its implications for per-class performance are detailed in the Limitations section.

The overall labeling protocol builds on our prior response-based assessment framework for camera-mediated self-reports [19]. Because the endpoint in the present study is a psychiatrist consensus ratings of work-function impairment rather than an affective proxy or questionnaire score, the supervisory signal encodes clinically grounded judgments about functional status during everyday work. Accordingly, the proposed models are trained to capture statistical regularities that align with psychiatrist reasoning in naturalistic occupational contexts.

Feature extraction and preprocessing

To operationalize psychiatrist-labeled work-function impairment from observable behavior, we extracted multimodal behavioral representations from each valid video. Specifically, three primary feature streams were derived to capture complementary aspects of facial expression, speech acoustics, and linguistic content.

Facial features were obtained via OpenFace and included facial action units, head pose, and gaze [30]. Acoustic features were computed using a Praat-compatible paralinguistic pipeline and comprised fundamental frequency (F0), intensity, harmonics-to-noise ratio, formants, spectral measures, and MFCCs [31]. Linguistic features were derived by transcribing speech with Whisper [32] and embedding utterances using transformer encoders from the Sentence-BERT family [33,34].

To integrate these heterogeneous streams at the temporal level, Whisper timestamps were used to define utterance boundaries. Text embeddings and turn identifiers were broadcast across their corresponding time spans to align linguistic information with continuous facial and acoustic signals. Facial and acoustic streams were temporally aligned, resampled to 100 Hz, and aggregated into 50 ms frames with 50 ms hops (20 Hz).

A limited set of survey-derived attributes was retained only to support normalization procedures and stratified descriptive summaries. All normalization statistics were computed exclusively within each training fold and applied to the corresponding held-out fold, thereby preventing information leakage across participants. The resulting aligned multimodal sequences served as input to the modeling pipeline described below and illustrated in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed multimodal modeling pipeline.

Each smartphone-dialogue clip is first segmented into speech segments, from which three parallel feature streams (face, voice, text) are extracted and independently encoded by per-stream Attention-MIL blocks. The three stream-level representations are combined by late fusion with logit adjustment and decoded into the three ordinal classes (Healthy / Moderate / Unwell) by a CORN ordinal head. Modality dropout on the face and voice streams is applied during training only. Implementation details (feature dimensions, time alignment, fusion equation, CORN decoding, and training loss) are described in Methods, Feature extraction and preprocessing and Model and evaluation.

https://doi.org/10.1371/journal.pone.0354342.g002

Model and evaluation

We trained a three-stream Attention-MIL model [35] that maps temporally aligned facial, acoustic, and linguistic feature sequences () to a probability over the three ordinal classes (Fig 2). The per-modality encoders are attention-pooled and combined via learnable, simplex-constrained late fusion before being passed through a rank-consistent ordinal head:

(1)

where is a per-modality attention-pooled encoder [35], are simplex-constrained learnable fusion weights, and is the rank-consistent ordinal head of [36] with logit adjustment () [37] for class imbalance. The network is trained against a fused-logit ordinal loss with an auxiliary text-stream regulariser:

(2)

where is the fused logit, is the text-stream logit, and weights an auxiliary CORAL-style ordinal BCE [38]. During training, modality dropout [39] independently masks the face and voice streams with . Optimisation uses AdamW (weight decay 10⁻⁴); hyperparameters are selected per outer fold by Optuna Bayesian optimisation [40] (TPE sampler, 10 trials). Full implementation details are released alongside the code.

To evaluate generalization performance while preventing identity leakage, we employed 25-fold repeated participant-disjoint cross-validation (5 repeats 5-fold GroupKFold), assigning all clips from the same participant to a single fold. All out-of-fold (OOF) predictions across all 25 folds () were aggregated as the primary evaluation set. OOF predictions were used for all reported classification, discrimination, and calibration metrics. For modality ablation comparisons, fold-level metric vectors (n = 25) were used for paired t-tests with df = 24; since the 55 repeated structure makes the 25 folds non-independent, single-configuration uncertainty is additionally quantified by the participant-level cluster bootstrap described below. For subgroup analyses with limited stratum size, fold-level paired t-tests are restricted to folds whose test partition contains (i) at least 5 stratum clips and (ii) more than one true class. This pre-specified criterion depends only on the test partition’s clip count and class membership, not on any model’s predictions, so it cannot preferentially favour any configuration; small-sample F1 is otherwise unstable (a single misclassification on fewer than 5 clips shifts macro-F1 by 0.2–0.3) or mathematically uninformative (single-class macro-F1 collapses to one per-class term). A sensitivity analysis replacing excluded folds with and re-testing on the full 25-fold set is reported alongside the primary result in Results. Metric uncertainty was quantified by a participant-level (cluster) bootstrap with B = 2,000 resamples of the 29 participants, yielding 95% percentile confidence intervals. Holm–Bonferroni correction [41] was applied across the family of six paired modality contrasts against the Text-only baseline to control the family-wise error rate. Throughout this paper, we use the phrase statistically indistinguishable to mean “no detectable difference at after Holm–Bonferroni correction”; this is a non-rejection of the null and does not constitute formal equivalence in the sense of a TOST procedure with a pre-specified margin. Subgroup-level contrasts (Results, Subgroup analysis: where nonverbal modalities contribute) are reported uncorrected and explicitly labelled as exploratory; the multiplicity caveat for the speech-duration clinical-state grid is discussed in Discussion, Statistical multiplicity and the post-hoc nature of subgroup contrasts.

Software and reproducibility

All analyses were conducted with fixed data splits and fold-wise preprocessing, using a consistent software environment across experiments. This section summarizes the execution conditions under which the figures and tables in this manuscript were generated, thereby supporting reproducibility. Across folds, identical preprocessing, training, evaluation, and calibration procedures were applied.

Any proprietary components were confined to non-essential engineering (e.g., logging or deployment wrappers) and were not part of the scientific workflow defining the dataset, psychiatrist consensus labeling, feature extraction, model specification, or evaluation protocol. The overall analytical workflow and psychiatrist consensus labeling context were designed to be consistent with our previously reported camera-mediated self-report framework [19].

Engagement summary

The final analysis cohort comprised 29 participants contributing 1,768 OOF clips (exclusion flow detailed in Methods, Feature extraction and preprocessing). Per-participant clip counts were mean 61.0, SD 43.6, median 56, range 8–155, with a median of 44 observed days and a mean of 1.63 responses per participant-day (median share of days with responses: 0.514). This density supports both the participant-level cluster bootstrap and the modality-stratified analyses reported below.

Overall performance on the out-of-fold evaluation

We evaluated classification performance on the aggregated OOF clip-level predictions (; protocol detailed in Methods, Model and evaluation). The OOF set is imbalanced across the three ordinal classes (0 = Healthy, 1 = Moderate, 2 = Unwell; supports 1,130 / 416 / 222), so macro-averaged F1 was used as the primary summary alongside accuracy, with AUROC and AUPRC computed on the same predictions for ranking-based comparison [42]. Headline metrics for the Full configuration are reported with cluster-bootstrap 95% confidence intervals in Table 1 (macro-F1 = 0.772, accuracy = 0.840).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Overall metrics across the four text-containing configurations on the 25-fold OOF evaluation (, 29 participants). This table documents the aggregate-level equivalence of the four text-containing configurations on every overall metric—the empirical basis for the subgroup-conditional hypothesis tested in Table 3; it is not intended to establish a within-family ordering. Confidence intervals (in brackets) were obtained by resampling participants (not individual clips) with replacement, so that the uncertainty estimate reflects between-participant variability rather than the larger but non-independent clip count^a.

https://doi.org/10.1371/journal.pone.0354342.t001

Per-class F1 was approximately 0.93 (Healthy), 0.70 (Moderate), and 0.69 (Unwell), broadly comparable across the four text-containing configurations; per-class confidence intervals are tightest for Healthy and substantially wider for Moderate and Unwell, reflecting their lower support (S4 Table).

To further examine the structure of classification errors across the four text-containing configurations, we summarized the relationship between predicted and true classes using per-configuration confusion matrices on the OOF evaluation (full matrices in S7 Table). The error structure is highly consistent across configurations: misclassifications concentrate between adjacent severity levels (01 and 12), whereas direct confusions between the extremes (02) are rare in every configuration ( of Healthy clips misclassified as Unwell, of Unwell clips misclassified as Healthy).

Discrimination and calibration analyses

Beyond hard classification at a single operating point, we assessed ranking ability and probability quality on the same OOF set. Discrimination metrics (macro-AUROC, macro-AUPRC) for the four text-containing configurations are reported in Table 1.

Probability calibration was assessed on the OOF predictions for each of the four text-containing configurations. Overall (winning-class) expected calibration error (ECE) was uniformly small across configurations: Text-only 0.025, Face+Text 0.035, Audio+Text 0.021, Full 0.024 (Fig 3; per-class panels in the same figure). Because reliable risk communication on imbalanced ordinal targets requires that confidence is well calibrated for each class rather than only on average, we additionally computed per-class (one-vs-rest) ECE. Per-class ECEs are small and broadly comparable across the four text-containing configurations ( for any classconfig cell; Healthy ECE is slightly higher for Face+Text at 0.045 versus 0.036–0.039 elsewhere). For the Full configuration, per-class ECE was 0.038 (Healthy), 0.024 (Moderate), and 0.025 (Unwell), with corresponding Brier scores of 0.067, 0.102, and 0.059. The Moderate class carries the largest Brier score (0.102) across all four configurations, reflecting larger probabilistic-loss magnitudes near the Healthy/Moderate and Moderate/Unwell boundaries even when the binned ECE itself is small. This pattern is consistent with the inter-rater ambiguity quantified in the Methods section, in which clinically equivocal clips are over-represented near the Moderate boundary.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Per-class reliability diagrams across the four text-containing configurations.

Reliability diagrams for each severity class (Healthy / Moderate / Unwell, rows) under each text-containing configuration (Text-only / Face+Text / Audio+Text / Full, columns), computed on the out-of-fold (OOF) clip-level one-vs-rest probabilities ( from 29 participants; 25-fold repeated participant-disjoint cross-validation). Predicted one-vs-rest probabilities were assigned to ten equal-width bins on the interval [0,1]; each bar plots the empirical fraction of positives within a bin against the mean predicted probability in that bin, with the connecting line tracing the bin centres. The dashed diagonal indicates perfect calibration (predicted probability = empirical positive rate); shaded regions to the right of the diagonal indicate over-confidence and those to the left indicate under-confidence. Each panel reports per-class expected calibration error (ECE); overall (winning-class) ECE per configuration is Text-only 0.025, Face+Text 0.035, Audio+Text 0.021, and Full 0.024. Per-class ECEs are uniformly low () and broadly comparable across configurations, with Face+Text slightly worse on the Healthy class.

https://doi.org/10.1371/journal.pone.0354342.g003

Modality ablation

To test our central hypothesis—that multimodal fusion provides additional cues beyond text alone—we retrained the Attention-MIL architecture under seven feature configurations partitioned into two families: a text-containing family (Text-only, Face+Text, Audio+Text, Full) and a text-free control family (Audio+Face, Audio-only, Face-only). The text-containing family represents progressive enrichment of the linguistic baseline with nonverbal streams; the text-free control family quantifies the contribution of nonverbal information in the absence of the dominant linguistic channel. For each configuration, the remaining modalities, model architecture, and training protocol were kept identical; performance was evaluated using fold-level paired t-tests on the 25 GroupKFold folds with Holm–Bonferroni correction across the six contrasts against the Text-only baseline (Table 2; a visual summary is provided in the Supporting Information).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Modality ablation across seven configurations (n = 25 folds; 5 repeats 5-fold GroupKFold; same splits across configurations). The four text-containing configurations (top block) form the multimodal family; the three text-free configurations (bottom block) are reported as a control to quantify the necessity of the linguistic channel. values are computed against the Text-only baseline. , , (paired t-test, df = 24; Holm–Bonferroni adjusted across the six non-reference contrasts).

https://doi.org/10.1371/journal.pone.0354342.t002

At the aggregate level, the four text-containing configurations are statistically indistinguishable from one another on macro-F1 (all Holm-adjusted p_adj > 0.40; absolute vs. Text-only). By contrast, every text-free configuration is substantially worse than the text-containing family ( vs. Text-only, p_adj < 0.001), confirming that the linguistic channel is necessary for the headline classification task. This aggregate uniformity within the text-containing family does not contradict the multimodal hypothesis; rather, it indicates that the additional value of nonverbal modalities is not uniformly distributed across clips. Consistent with this view, Text-only and Full agree on 96.3% (1,703 / 1,768) of OOF clips, with the residual disagreements concentrated between adjacent severity levels. The question therefore is not whether multimodal fusion changes aggregate macro-F1, but where and when nonverbal cues contribute—an analysis we report next in Subgroup analysis: where nonverbal modalities contribute.

Subgroup analysis: Where nonverbal modalities contribute

The aggregate result above leaves open the central question of this study: does multimodal fusion provide additional cues beyond text alone, and if so, where? To localize this contribution, we stratified the 1,768 OOF clips along two axes simultaneously: speech-duration (a proxy for the amount of linguistic content available to the text channel) and clinical state (Healthy L0 vs. non-Healthy L1 + L2; i.e., the contrast of clinical interest). Speech-duration was partitioned at the lower tertile (<0.89 s, n = 581), the upper tertile ( s, n = 589), and the intermediate band (0.89–2.67 s, n = 598). The intersection of the two axes yields six strata. Below we describe the cell-level pattern across these strata in prose; the clinically ambiguous Short non-Healthy cell, which is the principal subgroup-level finding of this study, is shown in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Subgroup analysis: paired contrasts vs. Text-only on the clinically ambiguous subgroup (short-speech <0.89 s non-Healthy, n = 77 clips from 13 participants). Fold-level paired t-tests on 18 of 25 GroupKFold folds (per the pre-specified small-stratum exclusion criterion defined in Methods). Only Audio+Text shows a statistically detectable advantage over Text-only at . A sensitivity analysis replacing the 7 excluded folds with and re-testing on all 25 folds yields t₂₄ = 2.26, p = 0.033 for Audio+Text vs. Text-only, confirming the result is not driven by the exclusion. This subgroup contrast is exploratory and uncorrected; see Discussion (Statistical multiplicity and the post-hoc nature of subgroup contrasts) for the multiplicity caveat.

https://doi.org/10.1371/journal.pone.0354342.t003

The pattern is consistent with the multimodal hypothesis. In Healthy clips, all four text-containing configurations approach ceiling accuracy in the short-speech stratum ( on n = 504 clips) and decline together as speech-duration grows—Healthy clips are essentially saturated for any text-containing model, so additional nonverbal information cannot help. In non-Healthy clips with sufficient linguistic content (mid and long speech, n = 227 and n = 334), the four configurations are again indistinguishable on macro-F1 (, no configuration reaches significance vs. Text-only).

The exception—and the key empirical signal of this study—is the short-speech non-Healthy stratum (n = 77 clips from 13 participants). In this clinically ambiguous cell, Audio+Text significantly outperforms Text-only in a fold-level paired t-test (, t₁₇ = 2.34, p = 0.032 uncorrected; full numerics and sensitivity in Table 3), whereas Face+Text and Full do not reach significance (p = 0.84 and p = 0.30, respectively). The 95% CI lower bound of +0.004 excludes zero but lies close to it, so the magnitude of the multimodal benefit should be regarded as modest in absolute terms, even though the contrast is statistically detectable; a pre-registered replication is required before this conditional advantage can be claimed with full confidence. In practical terms, corresponds to roughly two additional correctly classified clips per participant within this stratum, on a base of approximately six short non-Healthy clips per participant; whether this magnitude is clinically actionable is a question for prospective deployment evaluation. The acoustic complement is therefore specific to the cell in which (i) the response is brief, limiting the information available to the text channel, and (ii) the speaker’s clinical state has deteriorated, where the linguistic signal alone is uncertain. Importantly, this stratum is not an arbitrary post-hoc cut: it corresponds directly to the operational regime targeted by the present study design—brief (10–30 s) smartphone-based self-report dialogues collected in the workplace—so the conditions under which nonverbal acoustic cues are statistically detectable are precisely the brief-interaction context for which the system is intended. The subgroup contrast is exploratory and uncorrected; the multiplicity caveat is detailed in Discussion, Statistical multiplicity and the post-hoc nature of subgroup contrasts.

Three corollaries follow. First, the headline conclusion of the multimodal hypothesis is supported by a localized but statistically reliable contrast rather than by aggregate dominance. Second, the cell in which the contrast emerges is clinically meaningful: brief responses from non-Healthy individuals are precisely the cases in which a screening system has the least linguistic content to act on, and where acoustic prosodic cues—known to track mood and fatigue—carry relevant signal. Third, the absence of an effect in long-speech non-Healthy clips suggests a substitution rather than additivity pattern: sufficient verbal content saturates the prediction, and nonverbal modalities operate as a fallback rather than as parallel uplift.

Principal findings and interpretation

Through these experiments, we demonstrated that brief smartphone-based dialogues can yield clinically meaningful estimates of ordinal work-function impairment when supervision is anchored to psychiatrist consensus ratings. On 25-fold OOF predictions (n = 1,768 clips), the proposed multimodal framework yielded strong discrimination and well-calibrated probabilistic outputs across the three severity classes (0 = Healthy, 1 = Moderate, 2 = Unwell), with full numerical results across the four text-containing configurations in Table 1 (per-class breakdown in S4 Table).

Taken together, these results support two primary conclusions. First, even under substantial class imbalance, short multimodal behavioral samples contain sufficient information to rank clips by degree of functional impairment with high fidelity. Second, the alignment between predicted probabilities and empirical accuracy indicates that the model outputs reflect graded uncertainty rather than overconfident or arbitrary scores.

Importantly, the observed error structure further contextualizes these findings. Most misclassifications occurred between adjacent severity levels (Healthy vs. Moderate, and Moderate vs. Unwell), whereas direct confusions between extremes were rare. This pattern is consistent with the ordinal nature of work-function impairment: ambiguity is concentrated near clinical boundaries rather than across qualitatively distinct states, and ordinal misclassifications are not uniformly harmful (confusing Moderate with Unwell differs fundamentally from confusing Healthy with Unwell). Preserving this graded structure retains clinically relevant nuance that would be lost under binary formulations, particularly in screening contexts where gradual functional decline is of interest.

Relation to existing assessments and deployment perspective

From a measurement standpoint, the proposed approach is best viewed as complementary to established presenteeism questionnaires rather than as a replacement. Traditional instruments such as the Work Limitations Questionnaire or the Work Productivity and Activity Impairment scale are designed for periodic self-report and retrospective aggregation, yielding standardized scores suitable for longitudinal monitoring and population-level comparison.

In contrast, the present method captures in-situ behavioral expressions during brief interactions and can be administered repeatedly with low marginal burden. Accordingly, the two approaches address different measurement needs. Questionnaires remain appropriate for structured documentation and long-term trend analysis, whereas brief multimodal dialogues are better suited to capturing momentary functional states and detecting emerging impairment in everyday contexts.

The calibrated and ordinal nature of the model outputs further supports this complementary role: rather than producing binary decisions, the system provides graded estimates aligned with psychiatrist judgments, which can inform human-centered workflows in which interpretation and action remain under professional or organizational control. A methodological consequence of this design is that the estimation target is not self-reported productivity loss or affective state but clinician-judged functional consequences during work—a behaviorally grounded, expert-anchored complement to existing self-report instruments.

Modality roles and interpretation of nonverbal signals

The 25-fold ablation reveals a two-tier structure (Table 2): the four text-containing configurations are statistically indistinguishable from one another, whereas every configuration that drops the text channel degrades sharply. This aggregate uniformity within the text-containing family does not refute the multimodal hypothesis; rather, the subgroup analysis (Table 3) shows that the contribution of nonverbal modalities is conditionally distributed—contributing precisely where the text channel is most vulnerable (short-speech responses from non-Healthy participants, with sparse linguistic content and subtle clinical signal).

Limitations and future directions

Several limitations should be noted. First, the cohort size was modest and drawn from a single company, which limits statistical power and external validity. Organizational culture, job demands, and reporting norms may shape how impairment is expressed in short dialogues. Thus, the present results establish feasibility but do not guarantee transportability. Multi-site validation across industries and countries is a necessary next step.

Second, the unwell-impairment class was rare, increasing uncertainty in class-specific estimates. This reflects a common challenge in occupational screening, where the prevalence of Unwell cases is typically low. Future work should therefore consider larger samples, targeted enrichment strategies, or evaluation protocols that more directly reflect deployment prevalence and operational costs. Calibration also varies across classes: the Healthy class shows the largest per-class expected calibration error despite carrying the most support, suggesting that risk communication on the dominant class would benefit from class-conditional recalibration (e.g., temperature scaling per class) or training-time adjustments such as soft-label or weighted losses.