Introduction

Clinical practice and public health guidelines aim to provide recommendations based on the best available evidence to support health professionals in (clinical) decision-making and ultimately improve patient care and public health. Guideline developers thereby draw primarily on evidence from clinical research, such as randomized controlled trials and systematic reviews thereof, and to a lesser extent on cohort studies, case series and expert opinions [1,2]. Despite the value of evidence from clinical research, there is increasing recognition of the importance of incorporating the experiences and values of relevant stakeholders, such as patients and healthcare professionals, alongside clinical evidence [3]. Patients can, for instance, offer insights into the burden of disease and treatment, areas of concern, needs, preferences, and patient-relevant outcomes as well as prioritize guideline topics. Guidelines that integrate this type of knowledge have been shown to be of higher quality, utility, and relevance to medical practice [4,5]. As a result, significant efforts have been made to develop methods and tools for more effectively incorporating experiential knowledge from diverse actors. Current strategies for involving the public and patients at various stages of guideline development include consultative methods (such as interviews, focus groups, questionnaires and comments on drafts), participatory methods (e.g., involving patient representatives as panelists in the guideline development process) and communication strategies (providing information to patients to support their individual healthcare decisions), or a combination of these [6]. Moreover, insights into experiential knowledge can also be gained through reviewing relevant literature and synthesizing qualitative evidence [7]. The degree and type of involvement may depend on the topic of the guideline, the availability of patient organizations, time, and resources. Despite the recognized added value, incorporating patient experience and preferences remains a major challenge in the development of clinical practice and public health guidelines, hindered by constraints such as a lack of funding and expertise, methodological unclarity, uncertainty about roles and expectations, and the complexities of integrating this knowledge with other forms of evidence in prevailing evidence appraisal frameworks [8,9].

The challenge of effective and meaningful patient engagement is exacerbated in the case of developing guidance for infectious diseases. Patient organizations are uncommon, making it harder to recruit and involve patients, which is often needed for traditional engagement methods. In some cases, a wider scope of involvement is needed, e.g., of contacts of an infected case or the general public (e.g., for a COVID-19 guideline), which is more time-consuming and complex to realize. Lastly, when developing guidance amid constantly evolving evidence coupled with the urgency to provide rapid guidance during outbreaks like COVID-19 or Mpox, traditional engagement methods have proven difficult to implement in a short timeframe [10,11].

This study presents an innovative approach to address these challenges of including patients’ experiential knowledge in guideline development when traditional engagement methods are challenging to implement. This approach uses AI-based methods, particularly from the field of natural language processing (NLP) and was applied and examined in the case of the revision of the Dutch public health guideline for scabies. Over the past decade, numerous studies have employed NLP techniques to analyze patient experiences of, e.g., healthcare services [12], health technologies [13], and diseases [14] from patient-generated text data in surveys, on social media and online health communities. Despite substantial research and discussions on the potential of these AI-based methods, there persists a well-known implementation gap [15]. Many of these techniques offer marginal or unclear practical utility and few studies test medical NLP applications in real-world settings, leaving results and discussions largely confined to academic discourse [16].

In the field of guideline development, the AI technologies currently being developed primarily focus on supporting the collection and synthesis of scientific literature [17,18]. More recently, there has been an emerging interest in utilizing “real-world evidence” for guideline development, which, however, is primarily targeted at quantitative, routinely collected data from clinical registries and less at the inclusion of patient experiences [19]. There is no literature available that explores the use of AI-based methods to integrate patients’ experiential knowledge into guideline development.

In this study, we demonstrate the implementation and added value of employing NLP methods to learn about patients’ experiences in the case of the revision of the Dutch public health guideline for scabies. Scabies is a skin infestation, caused by the human scabies mite (Sarcoptes scabiei var. hominis). In common scabies, the scabies mites are transmissible through skin-to-skin contact. Because the mite can survive outside the human body, medical treatment of the patient and contacts is necessary, in addition to decontamination measures such as machine washing of clothing and cleaning contact surfaces [20]. Scabies treatment can be a challenging and lengthy process, involving prolonged efforts to eliminate scabies that significantly affect the individual and their surroundings. In recent years, the incidence of scabies has risen continuously worldwide as well as in many European countries, including the Netherlands [21]. The reported figures are likely still underestimated due to insufficient monitoring and data. Unfortunately, there are no patient organizations for scabies in the Netherlands, and furthermore, the disease carries a social stigma that may inhibit willingness to participate in available patient engagement methods. Our method of systematically gathering large amounts of diverse patient experiences shared online holds great potential in this context and may overcome potential limitations of traditional patient engagement methods.

The aim of this study is to showcase the potential of NLP methods as an innovative, additional tool for guideline development to gain insights into patients’ experiential knowledge and preferences and to incorporate these into public health guidance, by presenting the results and practical application of these techniques in the context of the revision of the Dutch public health guideline on scabies.

Methods

Results

Using structural topic modelling, we identified 13 overarching topics that were raised in the online forum on scabies (Table 1). In the following, we will focus on those topics that were most pertinent to the revision of the public health guideline for scabies, as they led to specific changes in the guideline text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Topics identified in patients’ experiences shared on dokter.nl, their assigned labels, proportions, and the most frequent and simultaneously most exclusive (FREX) words, translated from Dutch to English (n = 5644).

https://doi.org/10.1371/journal.pone.0339358.t001

Many of the topics identified revolve around and point towards the various aspects of the psychosocial burden experienced by people infected with scabies. In topic 1, with the highest topic proportion (11.2%), forum users express their appreciation for the forum community as a space for exchanging experiences, receiving support, consolation, understanding, and encouragement. Severe itching—one of the most common symptoms of scabies infection—is the third largest topic (11%) in our analysis. Those affected describe feeling dirty and that the itching usually worsens at night, which can lead to sleep deprivation, as reported by several forum users (topic 3). Patients and their close contacts describe a constant state of alarm and anxiety caused by uncertainty about the success of the treatment and the possible return of the mite (topic 6). The fear of (re)infecting others or themselves combined with feeling ashamed and dirty has led some patients to self-isolate, sometimes for weeks or months. They express that unsuccessful treatment(s) led to despair, frustration, and exhaustion (especially for members of large families) from trying to closely adhere to treatment plans (topic 2) and decontamination measures (topic 8). In some cases, this fear and desperation drove individuals to exceed recommended treatment frequency and doses hoping to enhance treatment effectiveness (topic 5), and to experiment with alternative remedies.

The exploration of alternative substances in the form of homemade creams surfaced as a distinct theme in our analysis (topic 7). Among the most frequently mentioned substances were sulfur (seen in 251 [63.5%] of the posts assigned to this topic), tea tree oil (in 239 [60.5%] posts), and neem oil (in 62 [15.7%] posts). Patients resorted to these remedies when medication was unavailable, e.g., when travelling abroad, or as an additional treatment method.

Lastly, forum users reported very sensitive and damaged skin following scabies medication and treatment (topic 12, with a topic proportion of 4.7%). Despite successful treatment, those affected still had to deal with considerable skin problems, such as susceptibility to bacterial and mold infections, eczema, and allergic reactions.

Discussion

This is the first study to use AI-based methods to gain insights into patients’ experiences to inform and contribute to the development of clinical and public health guidelines. We have demonstrated how employing NLP techniques to analyze patients’ experiential knowledge online can serve as an additional, innovative method in the toolkit of methods for integrating patient knowledge into guideline development. In the case of scabies, we have showcased how the application of this method has generated valuable insights that could not have been easily obtained otherwise. These patient experiences have led to meaningful and practical additions in the guideline. The methods presented can thus contribute to meeting the persistent challenge in guideline development to incorporate experiential knowledge into guideline development in a methodologically sound manner.

A key advantage offered by this analysis of patient-generated text collected from web-based platforms is that patients describe their experiences in “natural language”, i.e., accounts are spontaneously expressed in an environment that is often familiar to them rather than solicited in a research setting [32]. Patients may benefit by being able to express their experiences more candidly and anonymously in such an environment. Consequently, this unfiltered and unprompted analysis allows to uncover authentic themes that truly matter to patients. Another advantage is that this approach enables insights into patient experiences on a larger scale. The experiences of a larger and likely more diverse group of patients can be consulted, including views not represented by patient organizations. Such analyses can be carried out fairly quickly, remotely, with relatively little organizational and logistical effort, without any burden for the patients, and at low cost. While this study focuses on scabies in the Netherlands, the methods presented can be applied to a variety of other diseases, online platforms—including social media—and different national or cultural contexts. Research across countries and disease areas shows that patients increasingly use social media and online health communities to share their personal health experiences [33]. These methods may be particularly valuable in contexts and disease areas where conventional patient involvement methods prove difficult to implement. This is for example evident when patient organizations are scarce (e.g., for most infectious diseases) or when the disease is associated with a societal stigma (e.g., sexually transmitted diseases) that may hinder active patient involvement and when experiential knowledge is evolving as is the case with outbreaks of infectious diseases. Furthermore, NLP analyses can also provide more nuanced and diverse insights, precisely when patient organizations are highly organized, have strong viewpoints and advocate for specific subjects. Lastly, patient-generated online content could also be analyzed across countries—for instance, through the use of multilingual topic models [34]—but requires careful consideration of language, platform norms, and local healthcare contexts. For all such uses, a shared requirement is that there needs to be a critical number of patients discussing and sharing their experiences in online health communities, patient forums, or on social media for this method to be beneficial.

Careful awareness is needed that, while there are invaluable experiences shared online, there may be individuals—especially in cases with greater public attention—who do not share their experiences truthfully or deliberately spread false information, possibly aided by bots [35]. This can be partially mitigated by implementing bot detection filters during the data scraping process, removing duplicate posts, analyzing topics at an aggregated level giving less weight to individual posts, and by following a human-in-the-loop approach, where automated analyses constantly alternate with close reading and qualitative analysis of the data (for an extensive review of detecting and mitigating online misinformation see [36]). For example, if a post appears suspicious, researchers can revisit the forum to examine the user’s profile and posting history, identifying potential red flags such as a newly created account, continuous 24/7 activity, or repetitive posting of identical content. The “human-in-the-loop” aspect is a defining strength of our method. Yet it might also pose a limitation in that it is not a fully automated process but necessitates foundational computer science expertise to ensure the safe and effective execution of these methods. Analyzing digital patient knowledge offers insights into the experiences of a larger and likely more diverse group of patients. Nevertheless, it is crucial to acknowledge that this data represents but a specific selection of patients sharing their experiences online, which also vary by channel. Despite increasing social media uptake across demographics, a selection bias towards native language writers, the young, wealthy, and technologically savvy still exists [31]. Additionally, it is important to recognize that, while patients turn to online platforms for a range of reasons [33], the content shared may disproportionately highlight negative and extreme experiences, thereby overshadowing more typical cases [37,38]. Guideline developers need to be mindful of which patient groups and which experiences they may not be reaching and consider other methods from the toolkit to include these patients. Finally, it is important to recognize that, although our method can render patients’ experiential knowledge and preferences more readily available, the validation and the meaningful integration of this type of knowledge into guideline development remains a challenge. One promising avenue might be to collaborate with a patient representative to review experiential knowledge from online sources and ensure that it is accurately represented and meaningfully integrated into the guideline development process. Where dedicated patient organizations do not exist, findings could be compared with analyses of additional online platforms, published literature on patient experiences, or other methods such as surveys. Innovative digital approaches could also be considered—for example, engaging directly with patients online to gather feedback or ask targeted questions.

Conclusion

Through the purposeful application of established NLP methods, we provide guideline developers with a new, methodological sound way to incorporate experience-based knowledge into the guideline development process. We are hereby addressing the persistent challenge in evidence-based guideline development where, despite the long-recognized and proven value of this type of knowledge, its meaningful and effective integration into the guideline process remains difficult to achieve. While not a panacea, this novel approach represents a significant stride in reconciling evidence-based guideline development with its founding principles of equally drawing on clinical evidence, patient experience and preferences, and clinical expertise in the development of guidance [39], which is crucial for developing high-quality, robust, relevant, and inclusive guidelines for complex clinical and public health questions.