Introduction

Immune checkpoint inhibitors (ICIs) are an increasingly used type of cancer immunotherapy that has been shown to improve the prognosis of patients with different types of cancers [1–5]. By blocking the function of key checkpoint factors, ICIs increase the activation of T cells, leading to a more vigorous response against tumors [6]. Monoclonal antibodies approved by the U.S. Food and Drug Administration (FDA) for cancer treatment include the programmed cell death 1 (PD-1) inhibitors nivolumab, pembrolizumab, cemiplimab, and dostarlimab and the programmed death ligand 1 (PD-L1) inhibitors atezolizumab, durvalumab, and avelumab [6]. Another “brake” on immune cells that negatively regulates activation of T-cells (CD4+ and CD8+) is cytotoxic T-lymphocyte-associated protein 4 (CTLA-4); the antibody ipilimumab inhibits CTLA-4 function and is also used to treat cancer. Recently, the FDA has approved relatlimab, a checkpoint inhibitor that targets lymphocyte-activation gene 3 (LAG-3), for the treatment of melanoma. Immune checkpoint inhibitors have transformed dramatically cancer management. Moreover, more patients have become eligible for ICI therapy over the years, including about 43.6% of patients with cancer in 2018 [7]. However, not all cases respond equally to ICI treatment, and different predictors of response have been suggested (e.g., PD-L1 expression, tumor-infiltrating lymphocytes, tumor mutational burden, tumor-associated macrophages) [8, 9].

Although ICIs have important benefits, they also can induce adverse events, called immune-related adverse events (irAEs), which affect different tissues of the body [10]. The irAEs are related to the activation of the immune system by ICIs. These adverse events might deteriorate patients’ quality of life and may not be tolerable to some patients. When patients need to make a decision regarding the initiation of ICI treatment, they must balance these benefits with the harms associated with treatment, taking into consideration risk factors for developing these harms. Furthermore, in this decision process, it is important that patients consider their own values and preferences, including financial concerns.

The internet is a common source of health information for patients [11]. With the increased use of ICIs in cancer patients, it is important to understand the quality and content of educational websites providing information on this therapy for patients. The purpose of this study was to assess the quality and usability of available websites providing information about ICI for cancer patients.

Materials and methods

We performed an environmental scan to systematically find and appraise the quality of currently available websites that provide patients with information regarding ICIs.

Results

Out of 1,083 URLs resulting from our search, we included 37 websites for analysis. After removal of duplicates, of the 874 URLs screened in the first step, most of them (n = 491, 56.2%) were excluded because the reviewers determined that the information provided by these websites was not directed specifically for patients. Sixty-eight (7.8%) websites were excluded because they had information about cancer treatment in general (n = 25) or about immunotherapy in general (n = 43), but not specific information about ICIs. Nine (1%) websites were excluded because they were unrelated to the topic. Fifty-two (5.9%) websites were excluded because they were duplicates. One hundred fifty-six (17.8%) websites were excluded because of format (they were YouTube videos, blog posts, social media posts, or newsletters, etc.). Nine (1%) websites were excluded because they were not created in the United States (S1 Table). The number of URLs identified in each step, the reasons for exclusions, and the final websites included are depicted in the flow diagram shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow diagram of immune checkpoint inhibitors website selection.

Modified from: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

https://doi.org/10.1371/journal.pone.0275676.g001

Website characteristics

The included websites are listed in Table 1. The distribution of domain extensions was as follows: 19 were .com (57.4%), 16 were .org (43.2%), and 2 were .gov (5.4%). For the pages that provided information in PDF format, we evaluated the main website (S2 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General description of websites included for analysis.

https://doi.org/10.1371/journal.pone.0275676.t001

Quality assessment

For each item assessed, the number of websites rated as “yes” is shown in Table 2. Below, we present a summary of the quality evaluation for each domain. S3 Table ranks the websites by overall score.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Quality assessment by domain and item.

https://doi.org/10.1371/journal.pone.0275676.t002

Accuracy.

The information provided by 24 websites (64.9%) was based on guidelines or other scientific publications. In 10 (27%) websites, it was not possible to know the source of the information. Of the 37 websites analyzed, 7 were from pharmaceutical companies manufacturing the immune checkpoint inhibitors (opdivo.com, yervoy.com, bavencio.com, keytruda.com, imfinzi.com, libtayo.com, tecentriq.com). The content provided by the pharmaceutical companies’ websites and four other websites (cancerreaserch.org, gene.com, medicinenet.com, rxlist.com) was based on FDA information.

Completeness and comprehensiveness.

Most of the websites (89.2%) provided a clear definition with a simple explanation of the topic ICI, and 56.8% provided information describing each treatment, how it works, mechanism of action, and contraindications. Most of the websites (81.1%) also provided information about complications. However, less than 20% of the websites had information about the balance between benefits and harms. The cost of the treatment was included in only 8.1% of the websites.

We added specific items to evaluate the completeness and comprehensiveness of the websites’ information about ICIs. We judged that 74.7% of the websites provided information about irAEs. Specifically, most of the websites listed the irAEs (cardiac, dermatological, endocrine, gastrointestinal, hematological, muscular, neurological, ocular, pulmonary, renal, systemic and rheumatic). Less than half (40.5%) of the websites explained risk factors for irAEs. Most of the websites that provided information about the risk factors denoted pre-existing autoimmune disease and patients with organ transplant as the main risk factors. Few websites (n = 5, 13.5%) included information about the management of irAEs of ICIs. Only three websites provided information about vaccine contraindications their adverse events and benefits, or efficacy in patients receiving ICIs.

Technical elements.

Most of the websites (94.6%) provided a clear statement of purpose, declaring that the information was not meant to replace the advice of a health professional. Moreover, most of the websites gave a clear definition of the target audience (91.9%). Thirty websites (81.1%) provided the date that the website was last updated. However, only three websites (8.1%) explained clearly the hierarchy of evidence, meaning that it was not clear what information was obtained from opinions and what information was from higher levels of evidence (e.g., cohort studies, randomized controlled trials, systematic reviews). The majority of the websites (78.4%) complied with advertisement guidelines [17].

Design and aesthetics.

All the websites had: 1) appropriate visual aspects (scroll bars and alignment), 2) acceptable typography, and 3) appropriate grammar. However, only 18 (48.7%) contained relevant graphics and images, and 19 (51.35%) used an appropriate variety of material (i.e., text, graphics, tables, equations, audio and video), appropriate cover images (i.e., to attract attention and clearly portray the purpose of the materials), and appropriate illustrations and media (i.e., no auto-play and minimal use of animation).

Usability.

Most of the websites were judged to be easy to use. Thirty-six (97.3%) websites appeared to be well organized, had an index and a table of contents, made it easy to return to the previous page, and had a help feature and frequently asked questions. All the websites showed appropriate speed, taking less than 5 seconds for the pages to load, including text and images. Thirty-two (86.5%) websites had an internal search engine. In three of the websites, the search engine was enhanced by Google.

Accessibility.

Under this domain, we evaluated the cultural match of the website, the appropriateness of color contrast, the findability of the content, and the availability of the website to people with disabilities. Most of the websites (94.6%) used appropriate language and images and presented examples in realistic and positive ways. We judged that 94.6% of the websites used appropriate color contrast, with font and background color contrasting, and were reader-friendly. We also determined that for most of the websites, the content was easy to find. On the other hand, we judged that only 32.4% of the websites complied with the World Wide Web Consortium 2018 guidelines, which means that the website is available to people with disabilities.

Readability.

The mean SMOG readability score for the 37 websites was 10 (standard deviation ±2.1) with a minimum of 6 and a maximum of 16. Only 2 websites had a readability score of 6, 1 website had a score of 7, and 8 websites had a score of 8. For websites that included more than one page or several PDF documents, we calculated the average score of all pages. Most of the websites (91.9%) used a conventional writing style, with active voice and simple sentences, and 94.6% consistently gave context before presenting new information.

Website scores.

The maximum score that a website could have if all the answers to the items were “yes” was 72. All the websites except four had a final score lower than 50.

User experience.

In S4 Table, we present the results of the user experience. All the websites met the expectations of the person who rated the user experience, and for more than 97% of the websites, the reviewer stated that she would return to the website and would recommend it to others. However, for 27% of the websites, the reviewer perceived that the website did not always use words that could be understood. Furthermore, for 13.5% of the websites, too little information about ICIs was given.

Discussion

To our knowledge, this is the first study to evaluate the content and quality of websites with educational material about ICI treatment. By assessing accuracy, completeness, technical elements, design and aesthetics, readability, usability, and accessibility, and navigation, we identified important differences between websites, allowing us to provide a comprehensive account of the current state of 37 patient-oriented websites on ICI treatment.

Most websites gave the complete definition, mechanism of action, and complications of ICI. More than 80% of the websites included a clear definition of the drugs and expanded on complications. However, only 20% of the websites gave information about the balance between benefits and harms. This is an important issue to address since previous studies have shown that presenting qualitative and quantitative information about benefits and risks leads to more accurate perceptions of drugs’ adverse events and benefits [18]. In a randomized controlled trial that sought to evaluate whether qualitative information, quantitative information, or a combination of the two best communicates benefits and risks, the investigators observed that providing absolute frequencies and percentages may benefit consumers by improving their drug knowledge and changing their perceptions [19]. In the present study, more than 75% of the websites provided information about irAEs. However, less than half of the websites explained risk factors for developing irAEs, and only 13.5% provided information about the management of irAEs. Only three websites gave information about vaccine contraindications, adverse events, and efficacy in patients receiving ICIs. irAEs are adverse events of ICI treatment that occur through immune activation. Almost any tissue of the body can be affected, and some patients bear a greater risk of developing them. A meta-analysis published in 2017 showed that patients with melanoma being treated with ICIs had a higher frequency of gastrointestinal and skin irAEs [20]. Patients with preexisting autoimmune diseases could have a greater risk of irAEs and flares of their autoimmune disease. In a retrospective cohort study, irAEs or autoimmune disease flares occurred in 71% of cancer patients with preexisting autoimmune disorders who were treated with ICIs, but they were easily managed [21]. Similar findings were observed in other studies, with a slightly lower incidence of flares and adverse events [22, 23]. Therefore, it is important for patients to receive clear and detailed information about risks and management of irAEs, which is largely lacking in current ICI websites.

Regarding the technical domain, most of the websites had general disclaimers, identification of the target audience, general disclosures, and contact information. More than 80% of the websites provided the date of last update. On the other hand, less than 10% of websites provided a clear hierarchy of evidence, and just 17% clearly stated the sources of the information. Almost all websites had good visual aspects, with appropriate typography and type of material. Furthermore, all the websites showed appropriate grammar. These findings indicate that the websites were strong overall in various technical aspects.

Concerningly, when we analyzed the readability of the websites, we observed that only two websites had a SMOG readability score of 6, and one website had a score of 7. The National Institutes of Health (NIH) recommend that health material be written at a grade 6–7 reading level. Our results echo those of previously published studies. In a study published in 2016, Hutchinson et al. observed that online educational material pertaining to 9 common internal medicine diagnoses used reading levels significantly above grade 6–7 [24]. Similarly, other studies have shown that the content of online educational material in other topics have on average a higher reading level than the NIH-recommended level [13, 25]. Our findings indicate that more effort should be made to provide patients with less complex information on ICI treatment that is more understandable.

We also evaluated the user experience of a user. For all the websites, the reviewer was satisfied with the experience of using the website. We also noticed that more than 25% of the websites did not always use words that were easily understood. This result is similar to that of our previously published study that evaluated the quality of websites with educational material for patients with rheumatoid arthritis [16]. In that study, more than 35% of the websites were perceived to not always use easily understandable words.

Our study has some limitations. First, we used three search engines, but there are other search engines that might find other websites. However, Google is the most popular search engine in the United States, accounting for more than 85% of all searches done in the country [26]. Another point of concern is that we excluded websites from other countries. There could be well-developed websites with important information about ICI for patients from other countries that patients from the United States can have access and get information from. However, we decided not to include other countries’ websites because the search engines we used account for IP address and would not prioritize the websites in the ranking of search results. Finally, the results of our study search would be difficult to replicate because of the complexity of the algorithms used by search engines to rank what comes up first in a search. Nevertheless, the results of our study give a picture of what is currently on the internet about ICIs for patients.

In conclusion, we have shown the strengths and weaknesses of the quality and content of current websites with information for cancer patients on ICIs, which will help organizations develop websites with detailed, high-quality information, presented in less-complex language. Patients could make better and more informed decisions if all the potential weaknesses found on the websites in our study are taken into consideration and addressed by the website’s developers. Moreover, improving the readability of websites would make them easier to understand and use, especially for people with low health literacy skills. These individuals might feel better informed with a website that follows the standards outlined, which can lead to less uncertainty about their condition and improve their self-efficacy.

Supporting information

Acknowledgments

The authors thanks Gary Mitchel Deyter, technical writer of the Department of Health Services Research at MD Anderson Cancer Center, TX. The manuscript was edited by Sarah Bronson of the Research Medical Library at MD Anderson.