Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Unveiling productivity: The interplay of cognitive arousal and expressive typing in remote work

  • Samiul Alam,

    Roles Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliation Department of ECE, University of Houston, Houston, Texas, United States of America

  • Saman Khazaei,

    Roles Methodology, Writing – review & editing

    Affiliation Department of Biomedical Engineering, New York University, New York City, New York, United States of America

  • Rose T. Faghih

    Roles Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Writing – review & editing

    rfaghih@nyu.edu

    Current address: Department of Biomedical Engineering, New York University, New York City, New York, United States of America

    Affiliations Department of ECE, University of Houston, Houston, Texas, United States of America, Department of Biomedical Engineering, New York University, New York City, New York, United States of America

Abstract

Cognitive Arousal, frequently elicited by environmental stressors that exceed personal coping resources, manifests in measurable physiological markers, notably in galvanic skin responses. This effect is prominent in cognitive tasks such as composition, where fluctuations in these biomarkers correlate with individual expressiveness. It is crucial to understand the nexus between cognitive arousal and expressiveness. However, there has not been a concrete study that investigates this inter-relation concurrently. Addressing this, we introduce an innovative methodology for simultaneous monitoring of these elements. Our strategy employs Bayesian analysis in a multi-state filtering format to dissect psychomotor performance (captured through typing speed), galvanic skin response or skin conductance (SC), and heart rate variability (HRV). This integrative analysis facilitates the quantification of expressive behavior and arousal states. At the core, we deploy a state-space model connecting one latent psychological arousal condition to neural activities impacting sweating (inferred through SC responses) and another latent state to expressive behavior during typing. These states are concurrently evaluated with model parameters using an expectation-maximization algorithms approach. Assessments using both computer-simulated data and experimental data substantiate the validity of our approach. Outcomes display distinguishable latent state patterns in expressive typing and arousal across different computer software used in office management, offering profound implications for Human-Computer Interaction (HCI) and productivity analysis. This research marks a significant advancement in decoding human productivity dynamics, with extensive repercussions for optimizing performance in telecommuting scenarios.

Introduction

The intricate relationship between stress and productivity has been a subject of extensive research across various disciplines [13]. Pioneering theories like the Yerkes-Dodson law [1] have proposed an inverted U-shaped function, suggesting an optimal level of stress that maximizes productivity. This law has profound implications for understanding the dynamics of workplace performance, individual differences in stress response, and the overall impact on mental and physical health. However, the universality and applicability of this law have been topics of debate, prompting further exploration into the nuanced relationship between stress and productivity across different contexts and populations.

Delving deeper into the neuroscientific perspective, stress regulation and productivity are intricately linked with specific brain regions and their interconnections. The amygdala, known for its role in processing emotions, especially fear and stress-related response, is a critical player in this domain. It interacts closely with the prefrontal cortex, which is central to executive functions, decision-making, and emotional regulation. The hippocampus, another key region, is vital for memory consolidation and is particularly sensitive to stress, which can impact its function and structure. These brain regions form a complex network that governs our responses to stress and influences cognitive capacities like attention, memory, and decision-making processes [46]. Understanding these neural mechanisms is crucial for comprehending how stress affects cognitive functions and, consequently, productivity.

In the context of this study, the physiological signals under investigation—such as autonomic nervous system (ANS) activations and HRV—are reflections of the underlying neural activities. The ANS, with its sympathetic and parasympathetic branches, is directly influenced by brain activity, particularly in regions like the amygdala and prefrontal cortex [7]. Heart rate variability, as a measure of ANS function, offers insights into the body’s stress response and its regulation by the central nervous system. Thus, exploring these physiological markers in the context of neural substrates provides a more comprehensive understanding of the stress-productivity nexus.

This study takes a closer look at the physiological dimensions of productivity, exploring parameters such as ANS activation, typing dynamics, and HRV. The assessment of ANS activations through EDA data via deconvolution techniques [822] provides valuable insights into the body’s arousal levels and stress response. HRV data, derived from ECG measurements [23], further complement these findings by offering a window into the heart’s response to stress, reflecting the dynamic balance between the sympathetic and parasympathetic nervous systems. Cognitive arousal, as influenced by these physiological factors, has significant implications for memory, attention, and overall cognitive performance [24, 25].

The investigation extends to typing dynamics, an innovative measure of cognitive state and productivity. The correlation between typing speed and brain activity levels [26] reveals how cognitive and emotional states can manifest in physical activity. Expressive writing, requiring significant emotional and cognitive engagement, serves as a critical area of study. This type of writing, often reflective and personal, can induce physiological changes correlating with heart rate and emotion [27, 28]. The typing patterns observed during such tasks offer a unique lens through which to view an individual’s cognitive and emotional state, providing a practical method to assess productivity [29].

A comprehensive approach that integrates different physiological signals enhances the accuracy of depicting arousal and emotion. Studies have shown that a combination of HRV, EDA, and typing dynamics can provide a more nuanced understanding of an individual’s cognitive state and emotional well-being [19, 3047]. In our study, the simultaneous analysis of these signals aims to paint a comprehensive picture of the interplay between physiological responses and cognitive states, especially in the context of productivity and stress management.

The emergence of advanced IoT devices like the Empatica E4 and Emotibit [48, 49] has revolutionized the field of physiological measurement, allowing for high-frequency, non-intrusive data collection. These technological advancements facilitate deeper insights into the interaction between physiological signals and cognitive processes, opening new avenues for research in productivity, especially in remote work settings [5053]. Additionally, these insights have the potential to inform the development of innovative closed-loop control systems for performance regulation and mental health monitoring [5460].

In response to these advancements and the identified gaps in the literature, our study proposes a state-space approach to concurrently track cognitive arousal and expressive typing states, utilizing EDA, HRV, and typing dynamics as key indicators. This approach, grounded in control systems engineering and informed by Bayesian filtering techniques, enables us to model and monitor internal brain states and their physiological manifestations. Our methodology differs from previous works, which predominantly focused on either EDA or HRV for measuring cognitive arousal [10, 30, 33, 39, 41]. By integrating these diverse data streams, we aim to provide a more holistic view of the cognitive and emotional factors that influence productivity.

The inclusion of typing dynamics in our analysis is particularly novel, offering a tangible measure of cognitive engagement and expressive capacity. This aspect of the study is grounded in the understanding that cognitive processes are not only reflected internally but also externalized through physical behaviors like typing. By examining these dynamics, we can gain insights into the real-time cognitive states of individuals, especially in work environments where typing is a central activity.

Furthermore, the study aims to explore the role of stress in cognitive processes more deeply. Stress, while often perceived negatively, can also be a driving force for productivity when managed effectively. Understanding the optimal levels of stress, as suggested by the Yerkes-Dodson law, and their impact on different cognitive functions such as memory, attention, and problem-solving, is essential for this analysis. This exploration is particularly relevant in the context of remote work, where traditional stressors may be replaced or compounded by new challenges related to isolation, work-life balance, and technological reliance.

With the increasing prevalence of remote work, understanding the interplay between physiological responses, cognitive states, and productivity has taken on new significance. Our study aims to contribute valuable insights into how remote workers can optimize their performance by managing stress and leveraging their cognitive and emotional states. The application of our findings could lead to the development of tools and strategies to enhance productivity and well-being in remote work settings. Moreover, the potential for integrating these insights into the design of ergonomic workspaces, productivity software, and mental health support systems is vast.

Our methodology involves detailed data processing and employs cutting-edge technologies and sophisticated Bayesian models. The findings from this research are expected to not only validate existing theories but also reveal new patterns and insights into the relationship between stress, cognitive arousal, and productivity. We will present results from both simulated and experimental data, offering a robust evaluation of our approach.

Finally, the discussion and conclusion sections of this manuscript will reflect on the implications of our findings, considering both their theoretical and practical applications. We will also outline potential future research directions, emphasizing the importance of continued exploration in this field to fully harness the benefits of understanding the stress-productivity relationship in modern work environments.

Methods

In this section, we delineate the methodologies employed in our study. We describe the dataset [61] that we used, emphasizing the multimodal nature of the dataset which includes physiological signals, computer logging, and behavioral observations under varying stress conditions. The heart of our methodological framework lies in the application of a state-space model to capture the latent dynamics of cognitive arousal and expressive typing state, as represented by physiological and psychomotor parameters. This model is complemented by a multi-state filtering algorithm, following an Expectation-Maximization (EM) structure, to robustly estimate the system states and parameters [53, 62]. The algorithm integrates both forward and backward data [34, 40], refining the state space and parameter estimates to enhance the accuracy and reliability of our findings. Through these sophisticated statistical techniques, we aim to offer a comprehensive understanding of the interplay between cognitive arousal, stress, and physiological responses within the context of knowledge work.

Data

In this study, we utilize the SWELL Knowledge Work (SWELL-KW) dataset [61], which is particularly significant due to its comprehensive collection of multimodal data like computer logging, facial expression from camera recordings, body posture from a Kinect 3D sensor, as well as physiological signals like heart rate and skin conductance obtained from body sensors. The dataset, encompassing recordings from 25 subjects performing typical desk work tasks under varied conditions, provides a rich source for analyzing the complex interactions between physiological states and knowledge work activities. The subjects were exposed to different stressors, such as email interruptions and time constraints, making it an ideal context for exploring the relationship between cognitive load, stress, and physiological responses. Despite its robustness, the dataset does have limitations, including the relatively small sample size and the controlled experimental setup, which might not fully capture the complexity of real-world work settings. In our study, we focus specifically on EDA and HRV to assess stress and cognitive load, complemented by detailed analysis of computer logs for mouse and keyboard activities to measure the expressive typing state. The participants’ subjective experiences, assessed through validated questionnaires, add qualitative depth to our multi-faceted analysis, providing a holistic view of the impact of stressors in knowledge work.

State-space model

Assume that our state vector xk involves two states (cognitive arousal state and expressive typing state) and evolves with time following (1) where , We observe two sets of binary variables. These are the deconvolved skin conductance impulse events nk,1 and the key press events nk,2 which we consider as Bernoulli-distributed random variables with mass functions and , respectively. Here pk,1 = P(nk,1 = 1) and pk,2 = P(nk,2 = 1). These variables are related to xk as (2) where m ∈ 1, 2 and βm are constant coefficients to be determined and xk,1 and xk,2 are independent random variables. We determine empirically similar to [44] assuming that x0,m ≈ 0 ∀j at the very beginning of the random walk. Therefore, (3)

We use HRV as our continuous variable rk which we assume is related to xk linearly as (4) where is sensor noise and are parameters relating rk with the hidden state variable xk.

Multi-state filter algorithm overview

The multi-state filter algorithm is based on the Expectation-Maximization (EM) approach, used for estimating parameters in a statistical model. The algorithm involves iterative calculations over two main steps: the E-step (Expectation) and the M-step (Maximization).

E-Step.

The algorithm begins by approximating the posterior density as a Gaussian distribution following [62]. The filter update process involves solving for the state estimate xk|k, which results in the following update rules: (5) (6)

Backward smoother equations.

For improving state space estimates by using both backward and forward data, the following backward smoothing update equations are utilized: (7) (8) (9)

M-step derivations.

This next step focuses on updating the model parameters based on the expected log-likelihood. The updates for γ0, γ1 and γ2 can be derived by solving for the following system of linear equations: (10)

The updates for , σϵ,1 and σϵ,2 are as follows: (11) (12)

The full derivations for these equations can be found in the S1 Appendix. Following this steps the algorithm executes iteratively, alternating between the E-Step and M-Step for a set number of iterations. Each iteration refines the parameter estimates, thereby improving the model’s fit to the observed data.

An overview of the algorithm is shown in Algorithm 1

Algorithm 1 Overview of the Multi-State Filter Algorithm

Initialize Variables

for N iterations do

x1 ← 0

W1σe

for j from 2 to J do      ▷ E − Step

  XjXj−1

  Solve for Xj|j−1 from Eq (5)

  

end for      ▷ Smoothing

xk|K = xk|k + Ak(xk+1|Kxk+1|k)

     ▷ M Step

 Update γ[0, 1, 2] via Eq (10)

 Update σv via Eq (11)

 Update σϵ,[1,2] via Eq (12)

end for

Results and discussion

Evaluation using simulated data

To validate the multi-state filter, we compare its accuracy on simulated datasets. We generated ten sets of simulated data. To generate the data we randomly assigned typical values of such that they are in range [−0.7, 0.7]. We also assigned σe = [0.003, 0.003] and σv = 0.03. We created a random vector of length K = 2500 and generated the state values x according to Eq (1). We then calculated the value of vector r from Eq (4). Following this, we calculate as shown in Eq (3) based on approximating p0,1 and p0,2 by the average probability of an impulse occurring in the data similar to [44]. We simulated two point processes having a true value of p0,1 = 0.2 and p0,2 = 0.3. Using these generated data, we tested our algorithm’s capacity to estimate the unobserved sympathetic arousal and expressive typing states and recover the model parameters from the set of observations. The results of this simulation for 2 datasets are shown in Fig 1. The QQ plots indicate that we obtain good fits to the simulated data. The state estimate can deviate from the true value in regions where there are large gaps between impulses for an extended period of time as we see for the 1st simulated data in the left panel. The estimates are closer to the true state in regions where more impulses tend to occur.

thumbnail
Fig 1. Latent state estimations for simulated data.

The two panels above in order from top show the simulated binary random variables n1 and n2, the continuous variable r, the estimated latent variables and (shown in blue) alongside the ground truths x1 and x2 (shown in red). Following that are the occurrence probabilities and (shown in blue) alongside the ground truths p1 and p2 (shown in red). Lastly, the quantile-quantile (QQ) plot for the residual error of x is shown.

https://doi.org/10.1371/journal.pone.0300786.g001

We estimate observed system parameters for 10 different simulated datasets and the percentage deviations from the true values are shown in Table 1. In the table, we see that, γ0, γ2, and σv show relatively smaller errors overall. It is important to note that these system parameters can have multiple solutions due to the greater degrees of freedom afforded by multiple states. Therefore, the technique may not always converge to the exact values with which the simulated data were created.

thumbnail
Table 1. Estimation Error of system parameters for our approach on simulated data.

https://doi.org/10.1371/journal.pone.0300786.t001

Experimental results and discussions

To illustrate the effectiveness of the algorithm, we analyze its performance on experimental data from the SWELL-KW dataset. In the experiment, the subject is tasked with writing a general essay under different conditions as explained in the Data section. Figs 24 show examples of decoded arousal and expressive typing states for each session. We can expect the subject to feel more cognitive stress initially when they start searching for resources online, which gradually decreases once the subject feels more confident about the topic. Conversely, after gaining a good handle on the topic, the expressive typing state will also increase after some time as the stress decreases. Figs 24 also indicate that this trend is common for all sessions and is appropriately captured by the algorithm.

thumbnail
Fig 2. Latent state estimations for experimental data with no stressors.

Each of the panels above shows the experimental data for no-stressor sessions. From the top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated x1 and x2 (purple line) x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint, and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown. The panel on the left is for subject 1 and the one on the right is for subject 11.

https://doi.org/10.1371/journal.pone.0300786.g002

thumbnail
Fig 3. Latent state estimations for experimental data with time limit.

Each of the panels above shows the experimental data for no-stressor sessions. From the top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint, and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown. The panel on the left is for subject 1 and the one on the right is for subject 15.

https://doi.org/10.1371/journal.pone.0300786.g003

thumbnail
Fig 4. Latent state estimations for experimental data with interruptions.

Each of the panels above shows the experimental data for no-stressor sessions. From the top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint, and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown. The panel on the left is for subject 1 and the one on the right is for subject 13.

https://doi.org/10.1371/journal.pone.0300786.g004

It is essential to note that unlike in [34] where the absence of neural stimuli caused the EM estimate to fail in predicting minute changes in the latent state, here, the algorithm does not suffer from that problem. This is due to the fact that the algorithm uses multiple binary random variables to predict the latent states and it will only fail when both impulses become scarcer which we see exemplified in Fig 1.

Influence of stressors

In our analysis of the SWELL dataset, where three distinct sessions were recorded for each experiment, we observe significant behavioral patterns emerging across different stress conditions. Each session involved subjects writing essays, a task requiring both information gathering and opinion expression. The first session serves as a baseline with no stressors, providing a control scenario for comparison. The second session introduces a time constraint, adding a layer of urgency and thereby testing the subjects’ ability to manage time pressure. In the final session, subjects face frequent email interruptions, challenging their focus and multitasking abilities. These varying conditions, analyzed in Figs 24, reveal clear patterns in cognitive and emotional responses to stressors. Notably, the hidden states captured in the data effectively track the arousal and expressive typing states across different sessions. This is evident when correlating the active applications used by the subjects with the respective state values. The significance of these findings lies in their potential to enhance our understanding of stress impacts in knowledge work environments. By analyzing how subjects adapt their cognitive and emotional responses under varying stress conditions, we can gain insights into designing more effective work strategies and tools, particularly in scenarios replicating real-world challenges such as time constraints and frequent interruptions.

Baseline session.

For the first session or the baseline session, where no external stressors were introduced, subjects exhibited a unique pattern of cognitive and expressive behaviors, as seen in Fig 2 for subjects 1 and 11. This session was characterized by a more relaxed environment, allowing subjects to engage in the essay-writing task with relative ease. Initially, the subjects focused on gathering information, which is reflected in the elevated cognitive arousal state, denoted as x1. This heightened state of arousal during the information gathering phase indicates an intensive cognitive engagement, a critical aspect of the knowledge work. As the session progressed and the subjects shifted from information gathering to writing, there was a noticeable transition in their cognitive state. The arousal state gradually stabilized or oscillated, indicating a shift in cognitive demands. Concurrently, the expressive typing state, x2, which initially started at a lower level, began to rise. This increase in x2 aligns with the subjects’ transition into the writing phase, where expressive typing becomes more pronounced. The elevated expressive typing state towards the latter part of the session underscores the subjects’ engagement in the creative and expressive aspect of essay writing. This observation from the baseline session provides valuable insights into the natural progression of cognitive and expressive states in a stress-free environment, serving as a crucial reference point for comparing subject behaviors under different stress conditions in subsequent sessions.

Timed sessions.

In the timed sessions of our study, while the general patterns of expressive typing state and cognitive arousal observed across all sessions align with the trends described in Experimental Results And Discussions, a distinct dynamic emerges under the time constraints. Subjects in these sessions faced added pressure to complete their tasks within a set time frame, influencing their cognitive and expressive behaviors, as depicted in Fig 3. A notable observation is the peak in the middle of the session in the expressive typing state for most subjects, indicating a concentrated effort to accomplish the majority of the writing task. This peak is a significant indicator of increased typing activity, possibly reflecting a heightened state of focused work. The cognitive arousal state, denoted as x1, is observed to be high at the beginning and end of these sessions. Initially, the subjects engage in intensive information gathering, which requires significant cognitive processing, thereby elevating the arousal state. As they transition from gathering to writing, a decrease in cognitive arousal is observed, coinciding with an increase in the expressive typing state. This inverse relationship between the two states highlights the shift from cognitive-intensive to expression-intensive phases of the task. Towards the end of the session, the cognitive arousal rises again, likely due to the subjects’ efforts to review and finalize their work within the time limit. This rise at the end underscores the return to cognitive-intensive activities, such as editing and revising the written content. The behavioral patterns observed in these time-limited sessions offer insightful revelations about the impact of time pressure on cognitive and expressive states, shedding light on how individuals manage their workload under constrained conditions.

Email interrupted sessions.

The email-interrupted sessions in our study present an intriguing scenario where subjects encounter continual distractions in the form of emails, leading to varied patterns in cognitive and expressive states. This variability among subjects is particularly evident in their responses to interruptions, as some exhibit a marked change in their arousal state while others maintain a more steady level, reflecting their individual coping strategies with such disruptions. These interruptions inevitably impact the expressive typing state, typically causing a degradation, which can be attributed to the frequent breaks in concentration and the resultant shift in focus away from the primary task of essay writing. Unlike the time-limited sessions where distinct peaks in arousal and expressive typing states were observed, the patterns in the email-interrupted sessions do not exhibit such definitive forms. The lack of pronounced peaks in these sessions suggests a more erratic and disrupted workflow, indicating how the constant email notifications influence cognitive load and the ability to maintain a consistent level of expressive output. This observation sheds light on the impact of intermittent distractions on task performance, highlighting the need for strategies to manage such interruptions in work environments where maintaining focus is crucial for productivity.

State distribution for different office applications.

In our analysis, the visualization of cognitive arousal and expressive typing states with respect to the use of various applications offers insightful revelations about the subjects’ interaction with technology. By examining these states in the context of specific applications, we gain an understanding of how different tasks influence cognitive and expressive behaviors. For instance, as shown in Fig 5, subjects exhibit a higher level of cognitive arousal when engaged in responding to emails. This heightened arousal could be attributed to the immediate cognitive processing required to comprehend and respond to incoming information [1, 63]. Contrastingly, when subjects use Microsoft Word, a platform primarily for writing, they demonstrate a high expressive typing state, coupled with a relatively lower level of cognitive arousal. This pattern suggests a more focused and sustained engagement in the expressive aspect of the task. Furthermore, the use of applications like PowerPoint, which necessitates a combination of typing and cognitive efforts to create and structure documents, results in overlapping variations in the histograms of both cognitive arousal and expressive typing states. These overlapping variations highlight the dual demands of such applications, requiring both cognitive processing for conceptualizing content and expressive output for its articulation. The distinct patterns observed across different applications underscore the nuanced relationship between the nature of the task and the associated cognitive and expressive states, providing a window into how specific activities can differentially engage cognitive and expressive faculties.

thumbnail
Fig 5. State distribution for different applications.

The histograms above show the variations of state vectors x1 representing cognitive arousal state and x2 representing expressive typing state for four applications Word, PowerPoint, Internet Explorer, and VLC Player in order from left. The expressive typing state variations are shown in orange bars whereas the arousal state by blue bars. Notice the variations depend on the nature of the applications.

https://doi.org/10.1371/journal.pone.0300786.g005

The findings of this study have significant practical implications, especially in remote work environments where managing stress and maintaining productivity are crucial. Understanding the impact of different stressors and work activities on cognitive states can inform the design of more effective work strategies and tools. Additionally, the study lays the groundwork for future research, expanding the understanding of cognitive responses in work environments. Translating these findings into practice involves developing interventions and tools to manage stress and enhance productivity. By leveraging the study’s insights, practitioners can better understand and mitigate the impact of stressors in various work settings. Therefore, this line of research is crucial for advancing our understanding of workplace stress and productivity. It opens avenues for developing more adaptive and responsive work environments, ultimately contributing to improved well-being and efficiency. However, while the study provides valuable insights, it has some limitations, including the controlled nature of the experimental setup and the dataset’s sample size. Despite these limitations, the study’s multimodal approach and detailed analysis offer a comprehensive understanding of the interaction between stressors, cognitive states, and productivity.

Implications of the study

The field of cognitive response analysis in work environments, a critical aspect of occupational psychology and human-computer interaction, is undergoing rapid evolution. This change is primarily driven by the technological advancements in data collection and analysis methodologies. In today’s era, marked by an increasing reliance on digital tools and remote working arrangements [64], understanding the nuances of cognitive responses to various stressors in the workplace is not just academically intriguing but also practically essential [65]. It has significant implications for enhancing both employee well-being and workplace productivity [66].

Our study builds upon the foundational work in state space modeling and Bayesian filters [30, 31, 33, 34, 3747, 62], extending the knowledge frontier by integrating refined multi-state filtering methodologies. We delve deeper into the nuanced effects of stressors on cognitive and expressive states and their inter-relation, a subject that has been of perennial interest but inadequately explored due to technological and methodological limitations in the past. By employing a multi-state filtering technique, and leveraging a more diverse dataset, this research provides intricate insights into the dynamic interplay between various work-related stressors and cognitive responses. These insights are pivotal in our understanding of the cognitive load, a concept critical in designing effective and humane work environments.

The implications of our findings are far-reaching and multifaceted. Primarily, they contribute significantly to the discourse on workplace productivity and stress management. By systematically unraveling how different stressors—ranging from time constraints to intermittent distractions—affect cognitive arousal and expressive typing states, our study provides empirical evidence that can inform the design of workplace strategies and tools. These tools and strategies are not just theoretical constructs but have real-world applicability in enhancing work efficiency and reducing the detrimental effects of stress [67].

In the context of the evolving dynamics of the workplace, particularly with the surge in remote work and digital collaboration, the insights from our study are especially pertinent. Remote work, while offering flexibility and perceived freedom, brings with it unique challenges, including the blurring of work-life boundaries and the potential for increased stress due to isolation or overwork. Our findings offer a roadmap for navigating these challenges [68], helping organizations and individuals to strike a balance between productivity and well-being in these digitally mediated work environments.

Moreover, our research methodology and the resulting findings lay a solid foundation for future studies. The detailed analysis of cognitive responses to different stressors, conducted with rigor and precision, opens up new avenues for research. These include personalized stress management strategies and productivity enhancement techniques. Our study thus acts as a catalyst, spurring further exploration and innovation in this domain.

One of the most exciting prospects of this research is its potential contribution to HCI, especially to the development of intelligent tools for real-time stress and productivity monitoring by using underlying states. Such tools, harnessing the power of artificial intelligence and machine learning, could provide immediate feedback and interventions, helping individuals manage their workloads and stress levels more effectively as showcased in [60, 69]. This aligns with the broader trend of digitalization in the workplace [70] and represents a significant leap forward from traditional, more reactive approaches to workplace well-being.

The ultimate goal of this line of research is ambitious yet profoundly impactful: to achieve a comprehensive understanding of the cognitive impacts of various work-related stressors and to translate this understanding into practical tools and strategies. These tools and strategies aim to enhance productivity and well-being in the workplace, addressing one of the fundamental challenges of modern work life.

However, reaching this goal is not without its challenges. The primary challenge lies in the translation of theoretical knowledge and experimental findings into practical, real-world applications that are effective across diverse work environments and individual differences. This requires not just academic insight but also a deep understanding of the complexities and variabilities of real-world work settings in addition to strategic data collection that mimic practical use-cases.

To achieve this ambitious goal, interdisciplinary knowledge spanning psychology, data science, human-computer interaction, and occupational health is required. Moreover, technological advancements, particularly in non-invasive physiological monitoring [71], sophisticated data analytics [72], and AI-driven predictive modeling [73], are crucial. The development and refinement of such technologies will play a pivotal role in the practical application of our findings, bridging the gap between research and real-world application.

As workplaces continue to evolve, particularly with the increasing prevalence of remote work, the insights gained from this research will be invaluable. They will not only shape the future of work but also ensure that this future is more adaptable, efficient, and conducive to mental health and well-being. This research, therefore, stands at the intersection of technological innovation and human-centric design, heralding a new era in workplace productivity and health.

Conclusion

In this study, we have explored the intricate relationship between expressive typing state, and cognitive arousal. Our research demonstrates the potential of multi-state filters in accurately tracking these states, offering a promising tool for performance tracking in both traditional and remote work environments. The utilization of such advanced algorithms could revolutionize how we understand and manage productivity and stress in the workplace. Theoretically, our findings contribute to the broader discourse on cognitive workload and its manifestation in behavioral patterns. Practically, the application of this research could lead to the development of intelligent systems capable of enhancing workplace efficiency by providing real-time feedback on cognitive states. While our results with simulated data show promise, we recognize the need for further refinement, particularly in modeling the complexity of real-world interactions. Future research directions include expanding the scope of variables considered, such as incorporating analysis of correct and incorrect key responses and utilizing marked point process observations based on physiological data. Additionally, integrating more complex interactions, like mouse usage and application switching, could provide a richer understanding of cognitive states. However, limitations in current datasets, such as the SWELL dataset’s lack of comprehensive typing data, highlight the need for more detailed experimental studies tailored to the evolving landscape of remote work.

Supporting information

S1 Appendix. Summary derivations.

Derivations for different steps of the multi-state filter.

https://doi.org/10.1371/journal.pone.0300786.s001

(PDF)

S1 Fig. Latent state estimations for experimental data with no stressors for subject 1.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s002

(TIF)

S2 Fig. Latent state estimations for experimental data with time limit for subject 1.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s003

(TIF)

S3 Fig. Latent state estimations for experimental data with interruptions for subject 1.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s004

(TIF)

S4 Fig. Latent state estimations for experimental data with no stressors for subject 2.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s005

(TIF)

S5 Fig. Latent state estimations for experimental data with time limit for subject 2.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s006

(TIF)

S6 Fig. Latent state estimations for experimental data with interruptions for subject 2.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s007

(TIF)

S7 Fig. Latent state estimations for experimental data with no stressors for subject 3.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s008

(TIF)

S8 Fig. Latent state estimations for experimental data with time limit for subject 3.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s009

(TIF)

S9 Fig. Latent state estimations for experimental data with interruptions for subject 3.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s010

(TIF)

S10 Fig. Latent state estimations for experimental data with no stressors for subject 4.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s011

(TIF)

S11 Fig. Latent state estimations for experimental data with time limit for subject 4.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s012

(TIF)

S12 Fig. Latent state estimations for experimental data with interruptions for subject 4.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s013

(TIF)

S13 Fig. Latent state estimations for experimental data with no stressors for subject 5.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s014

(TIF)

S14 Fig. Latent state estimations for experimental data with time limit for subject 5.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s015

(TIF)

S15 Fig. Latent state estimations for experimental data with interruptions for subject 5.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s016

(TIF)

S16 Fig. Latent state estimations for experimental data with no stressors for subject 6.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s017

(TIF)

S17 Fig. Latent state estimations for experimental data with time limit for subject 6.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s018

(TIF)

S18 Fig. Latent state estimations for experimental data with interruptions for subject 6.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s019

(TIF)

S19 Fig. Latent state estimations for experimental data with no stressors for subject 7.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s020

(TIF)

S20 Fig. Latent state estimations for experimental data with time limit for subject 7.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s021

(TIF)

S21 Fig. Latent state estimations for experimental data with interruptions for subject 7.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s022

(TIF)

S22 Fig. Latent state estimations for experimental data with no stressors for subject 8.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s023

(TIF)

S23 Fig. Latent state estimations for experimental data with time limit for subject 8.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s024

(TIF)

S24 Fig. Latent state estimations for experimental data with interruptions for subject 8.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s025

(TIF)

S25 Fig. Latent state estimations for experimental data with no stressors for subject 9.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s026

(TIF)

S26 Fig. Latent state estimations for experimental data with time limit for subject 9.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s027

(TIF)

S27 Fig. Latent state estimations for experimental data with interruptions for subject 9.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s028

(TIF)

S28 Fig. Latent state estimations for experimental data with no stressors for subject 10.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s029

(TIF)

S29 Fig. Latent state estimations for experimental data with time limit for subject 10.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s030

(TIF)

S30 Fig. Latent state estimations for experimental data with interruptions for subject 10.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s031

(TIF)

S31 Fig. Latent state estimations for experimental data with no stressors for subject 12.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s032

(TIF)

S32 Fig. Latent state estimations for experimental data with time limit for subject 12.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s033

(TIF)

S33 Fig. Latent state estimations for experimental data with interruptions for subject 12.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s034

(TIF)

S34 Fig. Latent state estimations for experimental data with no stressors for subject 13.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s035

(TIF)

S35 Fig. Latent state estimations for experimental data with time limit for subject 13.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s036

(TIF)

S36 Fig. Latent state estimations for experimental data with interruptions for subject 13.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s037

(TIF)

S37 Fig. Latent state estimations for experimental data with no stressors for subject 15.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s038

(TIF)

S38 Fig. Latent state estimations for experimental data with time limit for subject 15.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s039

(TIF)

S39 Fig. Latent state estimations for experimental data with interruptions for subject 15.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s040

(TIF)

S40 Fig. Latent state estimations for experimental data with no stressors for subject 16.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s041

(TIF)

S41 Fig. Latent state estimations for experimental data with time limit for subject 16.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s042

(TIF)

S42 Fig. Latent state estimations for experimental data with interruptions for subject 16.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s043

(TIF)

S43 Fig. Latent state estimations for experimental data with no stressors for subject 17.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s044

(TIF)

S44 Fig. Latent state estimations for experimental data with time limit for subject 17.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s045

(TIF)

S45 Fig. Latent state estimations for experimental data with interruptions for subject 17.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s046

(TIF)

S46 Fig. Latent state estimations for experimental data with no stressors for subject 18.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s047

(TIF)

S47 Fig. Latent state estimations for experimental data with time limit for subject 18.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s048

(TIF)

S48 Fig. Latent state estimations for experimental data with interruptions for subject 18.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s049

(TIF)

S49 Fig. Latent state estimations for experimental data with no stressors for subject 19.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s050

(TIF)

S50 Fig. Latent state estimations for experimental data with time limit for subject 19.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s051

(TIF)

S51 Fig. Latent state estimations for experimental data with interruptions for subject 19.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s052

(TIF)

S52 Fig. Latent state estimations for experimental data with no stressors for subject 20.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s053

(TIF)

S53 Fig. Latent state estimations for experimental data with time limit for subject 20.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s054

(TIF)

S54 Fig. Latent state estimations for experimental data with interruptions for subject 20.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s055

(TIF)

S55 Fig. Latent state estimations for experimental data with no stressors for subject 21.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s056

(TIF)

S56 Fig. Latent state estimations for experimental data with time limit for subject 21.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s057

(TIF)

S57 Fig. Latent state estimations for experimental data with interruptions for subject 21.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s058

(TIF)

S58 Fig. Latent state estimations for experimental data with no stressors for subject 22.

The panel shows the experimental data for no stressor sessions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s059

(TIF)

S59 Fig. Latent state estimations for experimental data with time limit for subject 22.

The panel shows the experimental data with time limit. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s060

(TIF)

S60 Fig. Latent state estimations for experimental data with interruptions for subject 22.

The panel shows the experimental data with interruptions. From top, the binary variables n1 and n2 derived from deconvolved EDA data and typing data respectively, the continuous variable r denoting the RR intervals derived from heart rate (red line) and estimated from latent variables x1 and x2 (purple line), x1 and x2 in order from top indicating cognitive arousal state and expressive typing state respectively. p1 and p2 show the estimated probabilities. Patches of green, red, and cyan indicate what application the subject was using at the time of measurement. Green indicates applications for information search like internet explorer, red is for typing like Microsoft word and PowerPoint and cyan is for when subjects are looking at their emails. The Blue vertical line indicates the time email notifications were sent. Finally, the QQ plot for the residual error of r is shown.

https://doi.org/10.1371/journal.pone.0300786.s061

(TIF)

Acknowledgments

Samiul Alam would like to thank Dr. Dilranjan Wickramsuriya for his valuable help in developing the code for pre-processing the data and deriving the expectation maximization equations for the filter.

References

  1. 1. Yerkes RM, Dodson JD. The relation of strength of stimulus to rapidity of habit-formation. Journal of Comparative Neurology and Psychology. 1908;18(5):459–482.
  2. 2. MacFadyen JS. The dance between innovation, stress, and productivity. Holistic nursing practice. 2015;29(4):187–189. pmid:26086461
  3. 3. Wilke PK, Gmelch WH, Lovrich NP Jr. Stress and productivity: Evidence of the inverted U function. Public Productivity Review. 1985; p. 342–356.
  4. 4. Tortora F, Hadipour AL, Battaglia S, Falzone A, Avenanti A, Vicario CM. The Role of Serotonin in Fear Learning and Memory: A Systematic Review of Human Studies. Brain Sciences. 2023;13(8). pmid:37626553
  5. 5. Tanaka M, Szabó Á, Vécsei L, Giménez-Llort L. Emerging Translational Research in Neurological and Psychiatric Diseases: From In Vitro to In Vivo Models. International Journal of Molecular Sciences. 2023;24(21). pmid:37958722
  6. 6. Battaglia S, Nazzi C, Thayer JF. Heart’s tale of trauma: Fear-conditioned heart rate changes in post-traumatic stress disorder. Acta Psychiatrica Scandinavica. 2023;148(5):463–466. pmid:37548028
  7. 7. McCorry LK. Physiology of the Autonomic Nervous System. American Journal of Pharmaceutical Education. 2007;71(4):78. pmid:17786266
  8. 8. Amin MR, Pednekar DD, Azgomi HF, Van Wietmarschen H, Aschbacher K, Faghih RT. Sparse System Identification of Leptin Dynamics in Women With Obesity. Frontiers in endocrinology. 2022;13.
  9. 9. Amin MR, Tahir M, Faghih RT. A State-space Investigation of Impact of Music on Cognitive Performance during a Working Memory Experiment. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE; 2021. p. 757–762.
  10. 10. Steele AG, Parekh S, Azgomi HF, Ahmadi MB, Craik A, Pati S, et al. A Mixed Filtering Approach for Real-Time Seizure State Tracking Using Multi-Channel Electroencephalography Data. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021;29:2037–2045. pmid:34543199
  11. 11. Genty JX, Amin MR, Shaw ND, Klerman EB, Faghih RT. Sparse deconvolution of pulsatile growth hormone secretion in adolescents. IEEE/ACM Transactions on Computational Biology and Bioinformatics. 2021;19(4):2463–2470.
  12. 12. Amin MR, Faghih RT. Identification of sympathetic nervous system activation from skin conductance: A sparse decomposition approach with physiological priors. IEEE Transactions on Biomedical Engineering. 2020;68(5):1726–1736.
  13. 13. Amin MR, Faghih RT. Robust inference of autonomic nervous system activation using skin conductance measurements: A multi-channel sparse system identification approach. IEEE Access. 2019;7:173419–173437.
  14. 14. Amin MR, Faghih RT. Sparse deconvolution of electrodermal activity via continuous-time system identification. IEEE Transactions on Biomedical Engineering. 2019;66(9):2585–2595. pmid:30629490
  15. 15. Amin MR, Faghih RT. Tonic and phasic decomposition of skin conductance data: A generalized-cross-validation-based block coordinate descent approach. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2019. p. 745–749.
  16. 16. Amin MR, Wickramasuriya DS, Fagih RT. Wearable-Machine Interface Architectures for Mental Stress; 2019.
  17. 17. Faghih RT. From physiological signals to pulsatile dynamics: a sparse system identification approach. In: Dynamic Neuroscience. Springer; 2018. p. 239–265.
  18. 18. Faghih RT, Dahleh MA, Adler GK, Klerman EB, Brown EN. Quantifying pituitary-adrenal dynamics and deconvolution of concurrent cortisol and adrenocorticotropic hormone data by compressed sensing. IEEE Transactions on Biomedical Engineering. 2015;62(10):2379–2388. pmid:25935025
  19. 19. Faghih RT, Stokes PA, Marin MF, Zsido RG, Zorowitz S, Rosenbaum BL, et al. Characterization of fear conditioning and fear extinction by analysis of electrodermal activity. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2015. p. 7814–7818.
  20. 20. Faghih RT, Dahleh MA, Adler GK, Klerman EB, Brown EN. Deconvolution of Serum Cortisol Levels by Using Compressed Sensing. PLOS ONE. 2014;9(1):1–12. pmid:24489656
  21. 21. Amin R, Faghih RT. Physiological characterization of electrodermal activity enables scalable near real-time autonomic nervous system activation inference. PLOS Computational Biology. 2022;18(7):1–28. pmid:35900988
  22. 22. Faghih RT. System identification of cortisol secretion: Characterizing pulsatile dynamics; 2014.
  23. 23. Becker DE. Fundamentals of electrocardiography interpretation. Anesthesia progress. 2006;53(2):53–64. pmid:16863387
  24. 24. Tyng CM, Amin HU, Saad MNM, Malik AS. The Influences of Emotion on Learning and Memory. Frontiers in psychology. 2017;8:1454–1454. pmid:28883804
  25. 25. Tyng CM, Amin HU, Saad MN, Malik AS. The influences of emotion on learning and memory. Frontiers in psychology. 2017;8:1454. pmid:28883804
  26. 26. Duprez J, Stokkermans M, Drijvers L, Cohen MX. Synchronization between keyboard typing and neural oscillations. Journal of cognitive neuroscience. 2021;33(5):887–901. pmid:34449844
  27. 27. Bourassa KJ, Allen JJ, Mehl MR, Sbarra DA. The impact of narrative expressive writing on heart rate, heart rate variability, and blood pressure following marital separation. Psychosomatic Medicine. 2017;79(6):697. pmid:28481761
  28. 28. McGuire KMB, Greenberg MA, Gevirtz R. Autonomic effects of expressive writing in individuals with elevated blood pressure. Journal of Health Psychology. 2005;10(2):197–209. pmid:15723890
  29. 29. White HG. Typing Performance as Related to Mental Abilities and Interests: A Preliminary Study. The Journal of Educational Research. 1963;56(10):535–539.
  30. 30. Wickramasuriya DS, Crofford LJ, Widge AS, Faghih RT. Hybrid Decoders for Marked Point Process Observations and External Influences. IEEE Transactions on Biomedical Engineering. 2022;. pmid:35839187
  31. 31. Faghih RT, Wickramasuriya DS, Amin MR. Systems and methods for estimating a nervous system state based on measurement of a physiological condition; 2022.
  32. 32. Yaghmour A, Amin MR, Faghih RT. Decoding a Music-Modulated Cognitive Arousal State using Electrodermal Activity and Functional Near-infrared Spectroscopy Measurements. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE; 2021. p. 1055–1060.
  33. 33. Wickramasuriya DS, Faghih RT. A mixed filter algorithm for sympathetic arousal tracking from skin conductance and heart rate measurements in Pavlovian fear conditioning. PloS one. 2020;15(4):e0231659. pmid:32324756
  34. 34. Wickramasuriya DS, Faghih RT. A marked point process filtering approach for tracking sympathetic arousal from skin conductance. IEEE Access. 2020;8:68499–68513.
  35. 35. Pednekar DD, Amin MR, Azgomi HF, Aschbacher K, Crofford LJ, Faghih RT. Characterization of cortisol dysregulation in fibromyalgia and chronic fatigue syndromes: a state-space approach. IEEE Transactions on Biomedical Engineering. 2020;67(11):3163–3172. pmid:32149617
  36. 36. Ravindran AS, Nakagome S, Wickramasuriya DS, Contreras-Vidal JL, Faghih RT. Emotion recognition by point process characterization of heartbeat dynamics. In: 2019 IEEE Healthcare Innovations and Point of Care Technologies, (HI-POCT). IEEE; 2019. p. 13–16.
  37. 37. Ahmadi MB, Craik A, Azgomi HF, Francis JT, Contreras-Vidal JL, Faghih RT. Real-time seizure state tracking using two channels: A mixed-filter approach. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers. IEEE; 2019. p. 2033–2039.
  38. 38. Wickramasuriya DS, Faghih RT. A cortisol-based energy decoder for investigation of fatigue in hypercortisolism. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2019. p. 11–14.
  39. 39. Wickramasuriya DS, Amin M, Faghih RT, et al. Skin conductance as a viable alternative for closing the deep brain stimulation loop in neuropsychiatric disorders. Frontiers in neuroscience. 2019; p. 780. pmid:31447627
  40. 40. Wickramasuriya DS, Faghih RT. A novel filter for tracking real-world cognitive stress using multi-time-scale point process observations. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2019. p. 599–602.
  41. 41. Wickramasuriya D, Faghih R. A mixed-filter algorithm for arousal tracking from galvanic skin response and heart rate measurements. In: Proc. IEEE-EMBS Int. Conf. Biomed. Health Inform.; 2019. p. 1.
  42. 42. Wickramasuriya DS, Qi C, Faghih RT. A state-space approach for detecting stress from electrodermal activity. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2018. p. 3562–3567.
  43. 43. Deng X, Faghih RT, Barbieri R, Paulk AC, Asaad WF, Brown EN, et al. Estimating a dynamic state to relate neural spiking activity to behavioral signals during cognitive tasks. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2015. p. 7808–7813.
  44. 44. Wickramasuriya DS, Faghih RT. A Bayesian Filtering Approach for Tracking Arousal From Binary and Continuous Skin Conductance Features. IEEE Transactions on Biomedical Engineering. 2020;67(6):1749–1760. pmid:31603767
  45. 45. Valderas MT, Bolea J, Laguna P, Vallverdú M, Bailón R. Human emotion recognition using heart rate variability analysis with spectral bands based on respiration. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2015. p. 6134–6137.
  46. 46. Lane RD, McRae K, Reiman EM, Chen K, Ahern GL, Thayer JF. Neural correlates of heart rate variability during emotion. NeuroImage. 2009;44(1):213–222. pmid:18778779
  47. 47. Sunagawa K, Kawada T, Nakahara T. Dynamic nonlinear vago-sympathetic interaction in regulating heart rate. Heart and Vessels. 1998;13(4):157–174. pmid:10442397
  48. 48. Garbarino M, Lai M, Bender D, Picard RW, Tognetti S. Empatica E3—A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: 2014 4th International Conference on Wireless Mobile Communication and Healthcare-Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH). IEEE; 2014. p. 39–42.
  49. 49. Montgomery SM, Nair N, Chen P, Dikker S. Introducing EmotiBit, an open-source multi-modal sensor for measuring research-grade physiological signals. Science Talks. 2023;6:100181.
  50. 50. Wickramasuriya DS, Tessmer MK, Faghih RT. Facial expression-based emotion classification using electrocardiogram and respiration signals. In: 2019 IEEE Healthcare Innovations and Point of Care Technologies, (HI-POCT). IEEE; 2019. p. 9–12.
  51. 51. Yadav T, Atique MMU, Azgomi HF, Francis JT, Faghih RT. Emotional valence tracking and classification via state-space analysis of facial electromyography. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers. IEEE; 2019. p. 2116–2120.
  52. 52. Parshi S, Amin R, Azgomi HF, Faghih RT. Mental workload classification via hierarchical latent dictionary learning: A functional near infrared spectroscopy study. In: 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). IEEE; 2019. p. 1–4.
  53. 53. Wickramasuriya DS, Faghih RT. Online and offline anger detection via electromyography analysis. In: 2017 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT). IEEE; 2017. p. 52–55.
  54. 54. Branco LR, Ehteshami A, Azgomi HF, Faghih RT. Closed-Loop Tracking and Regulation of Emotional Valence State From Facial Electromyogram Measurements. Frontiers in computational neuroscience. 2022;16:747735. pmid:35399915
  55. 55. Azgomi HF, Faghih RT. Enhancement of Closed-Loop Cognitive Stress Regulation Using Supervised Control Architectures. IEEE Open Journal of Engineering in Medicine and Biology. 2022;3:7–17. pmid:35399789
  56. 56. Amin MR, Wickramasuriya DS, Faghih RT. A Wearable Exam Stress Dataset for Predicting Grades using Physiological Signals. In: 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT). IEEE; 2022. p. 30–36.
  57. 57. Azgomi HF, Cajigas I, Faghih RT. Closed-loop cognitive stress regulation using fuzzy control in wearable-machine interface architectures. IEEE Access. 2021;9:106202–106219.
  58. 58. Azgomi HF, Faghih RT. A wearable brain machine interface architecture for regulation of energy in hypercortisolism. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers. IEEE; 2019. p. 254–258.
  59. 59. Azgomi HF, Wickramasuriya DS, Faghih RT. State-space modeling and fuzzy feedback control of cognitive stress. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2019. p. 6327–6330.
  60. 60. Fekri Azgomi H, Hahn JO, Faghih RT. Closed-loop fuzzy energy regulation in patients with hypercortisolism via inhibitory and excitatory intermittent actuation. Frontiers in neuroscience. 2021; p. 909. pmid:34434085
  61. 61. Koldijk S, Sappelli M, Verberne S, Neerincx MA, Kraaij W. The SWELL Knowledge Work Dataset for Stress and User Modeling Research. In: Proceedings of the 16th International Conference on Multimodal Interaction. ICMI’14. New York, NY, USA: Association for Computing Machinery; 2014. p. 291–298. Available from: https://doi.org/10.1145/2663204.2663257.
  62. 62. Brown EN, Frank LM, Tang D, Quirk MC, Wilson MA. A Statistical Paradigm for Neural Spike Train Decoding Applied to Position Prediction from Ensemble Firing Patterns of Rat Hippocampal Place Cells. The Journal of Neuroscience. 1998;18(18):7411–7425. pmid:9736661
  63. 63. Storbeck J, Clore GL. Affective Arousal as Information: How Affective Arousal Influences Judgments, Learning, and Memory. Social and Personality Psychology Compass. 2008;2(5):1824–1843. pmid:25067943
  64. 64. Bernhardt A, Kresge L, Suleiman R. The Data-Driven Workplace and the Case for Worker Technology Rights. ILR Review. 2022;76(1):3–29.
  65. 65. Singh P, Bala H, Dey BL, Filieri R. Enforced remote working: The impact of digital platform-induced stress and remote working experience on technology exhaustion and subjective wellbeing. Journal of Business Research. 2022;151:269–286. pmid:35847196
  66. 66. Battisti E, Alfiero S, Leonidou E. Remote working and digital transformation during the COVID-19 pandemic: Economic–financial impacts and psychological drivers for employees. Journal of Business Research. 2022;150:38–50. pmid:35706830
  67. 67. Alhejaili R, Alomainy A. The Use of Wearable Technology in Providing Assistive Solutions for Mental Well-Being. Sensors. 2023;23(17). pmid:37687834
  68. 68. Marsh E, Vallejos EP, Spence A. The digital workplace and its dark side: An integrative review. Computers in Human Behavior. 2022;128:107118.
  69. 69. Fekri Azgomi H, F Branco LR, Amin MR, Khazaei S, Faghih RT. Regulation of brain cognitive states through auditory, gustatory, and olfactory stimulation with wearable monitoring. Scientific reports. 2023;13(1):12399. pmid:37553409
  70. 70. Kraus S, Ferraris A, Bertello A. The future of work: How innovation and digitalization re-shape the workplace. Journal of Innovation & Knowledge. 2023;8(4):100438.
  71. 71. Barreto A, Zhai J, Adjouadi M. Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. In: Human–Computer Interaction: IEEE International Workshop, HCI 2007 Rio de Janeiro, Brazil, October 20, 2007 Proceedings 4. Springer; 2007. p. 29–38.
  72. 72. Orphanidou C. A review of big data applications of physiological signal data. Biophysical Reviews. 2019;11(1):83–87. pmid:30627871
  73. 73. Zhang A, Wu Z, Wu E, Wu M, Snyder MP, Zou J, et al. Leveraging physiology and artificial intelligence to deliver advancements in healthcare. Physiol Rev. 2023;103:2423–2450. pmid:37104717