Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development of a web tool to increase research literacy in underserved populations through public library partnerships

  • Melissa A. Simon ,

    Roles Conceptualization, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Catherine A. O’Brian,

    Roles Writing – original draft, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Laura Tom,

    Roles Investigation, Methodology, Project administration, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Q. Eileen Wafford,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Galter Health Sciences Library & Learning Center, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Shenita Mack,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Chicago Public Library, Chicago, Illinois, United States of America

  • Samuel R. Mendez,

    Roles Investigation, Methodology, Project administration, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Magdalena Nava,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Rabih Dahdouh,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Center for Health Equity Transformation and Department of Obstetrics & Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

  • Rachelle Paul-Brutus,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Chicago Department of Public Health, Chicago, Illinois, United States of America

  • Kathryn H. Carpenter,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation University Library, University of Illinois-Chicago, Chicago, Illinois, United States of America

  • Barbara Kern,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation The John Crerar Library, University of Chicago, Chicago, Illinois, United States of America

  • Kristi L. Holmes

    Roles Investigation, Methodology, Project administration, Supervision, Writing – review & editing

    Affiliations Galter Health Sciences Library & Learning Center, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America

Development of a web tool to increase research literacy in underserved populations through public library partnerships

  • Melissa A. Simon, 
  • Catherine A. O’Brian, 
  • Laura Tom, 
  • Q. Eileen Wafford, 
  • Shenita Mack, 
  • Samuel R. Mendez, 
  • Magdalena Nava, 
  • Rabih Dahdouh, 
  • Rachelle Paul-Brutus, 
  • Kathryn H. Carpenter



Inadequate diversity in clinical trials is widely recognized as a significant contributing factor to health disparities experienced by racial/ethnic minorities and other diverse populations in the US. To address this in a scalable way, we sought to develop a web tool that could help enhance underserved minority participation in clinical research.


We used our research literacy support flashcard tool as the initial prototype for human-centered design and usability testing of the web tool Health for All in public library settings. After forming partnerships with leadership from Chicago Public Libraries (CPL), local medical libraries, and the Chicago Department of Public Health, we conducted seven iterative design sessions with focus groups of library patrons and library staff from six CPL branches serving underserved communities followed by two rounds of usability testing and website modification.


Based on the qualitative research findings from Design Sessions 1–7, we enacted the design decision of a website that was a hybrid of fact-filled and vignette (personal stories) paper prototypes divided into 4 modules (trust, diversity, healthy volunteers, pros/cons), each with their own outcome metrics. The website was thus constructed, and navigation issues identified in two rounds of usability testing by library patrons were addressed through further website modification, followed by the launch of a beta version of a hybridized single-scrolling and guided module prototype to allow further development with website analytics.


We report the development of Health for All, a website designed to enhance racial/ethnic minority participation in clinical trials by imparting research literacy, mitigating distrust engendered by longstanding racism and discrimination, and providing connections to clinical trials recruiting participants.


Precision medicine is "an emerging approach for disease treatment and prevention that takes into account individual variability in genes, environment, and lifestyle for each person" [1]. The goal of the practice of precision medicine in the US is to give health care providers the information they need to make customized recommendations for their patients [2]. A significant barrier to the realization of this goal is the underrepresentation of racial/ethnic minorities in clinical research in the US [35]. This lack of parity is the impetus for national initiatives such as the All of Us Program, which aims to increase diversity in clinical trials and biobanking initiatives in order to support more equitable representation in clinical research and bring the benefits of advances in healthcare forged by clinical research equitably to all, including currently underrepresented minority populations [2].

There are myriad barriers to participation in clinical research experienced by underrepresented minorities, such as cost (e.g., lack of health insurance, high out-of-pocket expenses), exclusion criteria (e.g., comorbid conditions, functional status and age), and logistics (e.g., transportation and time commitment or scheduling challenges) [6,7]. A major barrier is the distrust engendered in underrepresented minorities by longstanding experiences of racism and discrimination. This includes the heinous deception and mistreatment of African Americans in the Tuskegee syphilis study and other exploitative studies in the historical record [8]. Another significant barrier is limited research literacy, which includes lack of awareness of opportunities to participate in clinical research and what participation entails, and misperceptions about clinical research [57]. Further, limited research literacy and unfamiliarity with clinical research concepts presents the risk that individuals may make a decision about participating in research without adequately understanding the information presented in the informed consent process [5]. Advancing research literacy, particularly among underrepresented minorities, may avert or mitigate this risk, and at the same time enhance the diversity of participants in clinical research, e.g., by improving trust and empowering individuals to participate in research.

When developing health interventions in community settings, community-based approaches to human-centered design processes have been shown to align interventions such as web tools with the needs and contexts of the community members served and other local stakeholders [9,10]. We sought Chicago Public Library (CPL) partnership for Health for All development because public libraries are widely recognized as valuable and logical settings for community outreach, engagement and education, including in the domain of public health [1113]. We now report on the development of Health for All, a web-based tool adapted from a Research Literacy Support (RLS) tool to the setting of public library branches via community-based, human-centered design and usability testing. Health for All is designed to both enhance research literacy among diverse library patrons and connect them with clinical trials.

Materials and methods

Research literacy support tool

In partnership with the Cancer Disparities Research Network (CDRN) and under the auspices of the NCI-funded Geographic and Biobanking Management Program (G/BMaP), we previously employed a user-centered design approach to develop a flashcard-based Research Literacy Support (RLS) communication tool for use by clinical trial recruiters to advance research health literacy among potential clinical research participants by serving as an interactive guide that enhances communication with the recruiters [5,14]. Development of the RLS tool is published elsewhere [5,14] but briefly, the development process involved three phases. Recognizing the essentiality of investigating the specific needs of intended users at the outset of a user-centered design process [15], a needs assessment and initial prototype creation was accomplished in Phase 1. Upon completion of Phase 1, the RLS tool consisted of a series of 34 cards organized into sections on basic research information, study-specific information, and research participants’ rights in addition to cards that addressed myths and truths about clinical research [14].

Phase 2 of RLS tool development entailed cognitive interviewing [5,16]. Phase 2 was comprised of two iterations of semi-structured interviews with potential research participants representing underrepresented racial/ethnic groups and experienced clinical research recruiters followed by modifications to the tool content and design [5].

In Phase 3 of RLS tool development, a usability survey was administered to potential research participants from underrepresented racial/ethnic groups after they reviewed the RLS flashcard prototype. The usability survey results indicated that the cards were informative, easy to read, and explained research concepts in simple language [5]. The RLS tool was thus found to be useful to both participants and researchers at initiating, guiding, and fostering communication between researchers and participants that may foster minority participation in clinical research [5]. This RLS flashcard-based tool formed the basis of the Health for All web tool to be adapted for public library settings in which to scale and reach a larger audience.

Design thinking model

Our approach to the community-based, human-centered design and usability testing of the web tool Health for All from the flashcard RLS tool was guided by the Stanford model of design thinking. This design thinking framework is termed human-centered design because it centers on empathy [17]. Sometimes referred to as user-centered design, human-centered design integrates a set of practices to understand users in order to build deep empathy with them, generate multiple ideas to help solve their problems, and arrive at a solution via rapid prototyping [18]. The Stanford model of design thinking entails iterative implementation of five stages: empathize, define, ideate, prototype, and test [19]. In this design thinking model, the five stages do not have to follow any specific order and they can often occur in parallel and be repeated iteratively [19]. In our iterative design process, empathize meant understanding the needs and experiences of the end users of the Health for All intervention, who are public library patrons, as well as those of public library staff/managers and our Chicago Public Library executive partners; define entailed understanding the diverse experiences and perspectives of public library patrons and library staff/managers about the intervention to allow human centered refinement of the tool; ideate involved nonjudgmentally generating multiple options to solve problems identified in the define stage; prototype entailed developing concrete representations of solutions identified in the ideate stage in the form of paper and web interface prototypes of the intervention; and test involved evaluating and garnering feedback on intervention prototypes.

Partnership building for community-based development of Health for All

The Health for All tool was developed by the Northwestern University Center for Health Equity Transformation and the Northwestern University Galter Health Sciences Library and Learning Center, in partnership with Chicago Public Library (CPL), the University of Chicago John Crerar Library, the University of Illinois-Chicago Library of the Health Sciences-Chicago, and the Chicago Department of Public Health. Prior to meeting with CPL, we conducted background research on CPL infrastructure, visit statistics, and events, and we created user stories to conceptualize the context in which the tool would be used. We thus homed in on concepts/questions for initial project meetings with CPL related to technical resources, current scope of responsibilities for librarians, and current interactions with other university researchers. In the inaugural meeting, which was attended by representatives of our research team, CPL administration, the Chicago Department of Public Health, and leadership from local medical libraries, discussions centered on the goals and human-centered design approach of the project. Following the meeting, CPL would identify six diverse library branches in which to deploy the design-thinking process to develop Health for All, which would include focus group design sessions, usability testing, and deployment of iterative prototypes. The 6 library branches identified by CPL represented its northern, central and southern districts. Subsequent meetings were held between the research team and branch managers that involved project overviews and descriptions of the human-centered design process, followed by discussions about the needs of the librarians within the partnership.

Environmental scan

Data pertaining to the participating library branches was obtained from CPL and included information about the types of books borrowed from each branch and statistics describing public accessing of various sections of the website of each branch. Observational visits were made to each of the library branches by the research team to note setting characteristics such as noise level and how the library patrons used the space, e.g., browsing books or newspapers, using computers, gathering and conversing with each other, etc. Important to web design methodology, all six branches shared the same computer technology, which was Windows desktop computers with Internet Explorer 11 as the only available web browser.

Paper prototyping of the web interface

Paper prototyping is recognized as important to the efficiency of human-centered web design processes, both in terms of speed and cost containment. Paper prototypes are easily modifiable and thus can be used early in the design process to conduct rapid iterative testing of the terminology, navigation, content, page layout, and functionality of the interface and thereby identify and resolve problems prior to investments in programming [2022]. To design the web tool to be employed in usability testing, we gathered input from focus groups and CPL partners, e.g., some of the resources deployed were recommended by CPL partners, and the structure of guiding questions was informed by focus group input. We conducted seven iterative design sessions with focus groups of library patrons and library staff at the six participating local public library branches (CPL Branches A-F); six design sessions were conducted sequentially at each branch in the order of A to F, and the seventh design session was held at CPL Branch A. A convenience sample of 3–6 patrons and library staff from each branch were recruited through flyers posted at bulletin boards within that branch and on-site by research assistants in the hour prior to the focus group; all persons 18 years of age and older were eligible to participate. Focus group participants recruited among library staff included the branch manager(s). In each session, the agenda included three items. The first item was the storyboarding of a typical library visit; the second item was a facilitated discussion to glean focus group participants’ awareness and opinions regarding clinical trial participation and the underrepresentation of racial/ethnic minorities and other groups in clinical trials; and the third item was eliciting focus group responses to the RLS flashcard tool presented as a paper prototype of a web interface. All sessions were conducted in English, were audio recorded, and lasted approximately one hour. At each session, one research team member served as the moderator, and another took observational notes about participant movement, tone, and gestures throughout the session. Based on the audio recording and observational notes, summaries of each session were written and used to inform iterative refinement of the paper prototype. Data were analyzed by employing qualitative methods as previously described [23] to identify codes and generate prevalent themes. Focus groups sessions were held iteratively until thematic saturation–where no new themes emerged.

Web tool usability testing

The methodology used in the usability testing phase of the study was think-aloud [24] combined with audio recording, screen capture, and observation, as described in detail below. The general need for users to navigate with sufficient ease to find information in an acceptable timeframe guided our web tool usability testing approach, as did recognition that our target audience includes persons of low literacy and limited computer skills, underscoring the importance of simplified navigation strategies to optimize accessibility [25]. Together with a technical consultant, we built a single-page web tool with a long-scrolling layout in Wordpress (Version 1) and an InVision App click-through wireframe of a multi-page prototype with four guided modules, (e.g., trust in clinical trials), for the initial round of usability testing on a web interface at CPL Library Branches A and D. We recruited a convenience sample of usability testing participants by posting flyers at these library branches and by on-site recruitment by research assistants the day of the testing session; all persons 18 years of age and older were eligible to participate. The usability testing script asked participants to give their first impressions on certain sections of the single-page WordPress build, find specific information, give an overall impression of the website, and then make a comparison to the multi-page prototype. We ran these tests with one facilitator asking questions. We used a voice recorder to document the session. We used Panopto screen capture to document on-screen cursor movement. There were two note takers present to document the participants’ body language and tone of voice, any assistance required to complete the assigned usability tasks, and overall routes through the website. The records of the note takers revealed those instances where responses to the facilitator did not align with observed user experiences, e.g., where a user described website navigation as easy but required assistance with this. Thus, we assessed ease of use by both qualitative interviewing and observation. Based on coding and thematic analysis of notes taken and participant responses [23], we made adjustments to the single-page web tool prototype iteratively during the first round of usability testing to improve user experiences, resulting in Version 2 of the web tool. We conducted the second round of usability testing similarly at CPL Library Branches D and G using a multi-page prototype that incorporated the improvements made to the single-page prototype during the first round of usability testing, with iterative improvements made to the WordPress build based on documented user experiences, resulting in Version 3.

Human subjects research

All research reported here involving human participants, i.e., library patrons and staff, was approved by the Institutional Review Board (IRB) of Northwestern University and was conducted according to the principles expressed in the Declaration of Helsinki. All participants provided written informed consent prior to study participation.


Characteristics of the participating CPL library branches and the neighborhoods they serve

The key library characteristics of the six CPL library branches selected by CPL for participation in the human-centered design of Health for All are shown in Table 1 (Branches A-F), which reflects 2017 data. Activity levels varied widely across the six branches, as evident from comparing the size of circulation (i.e., lending activity), which ranged from 15,017 to 129,314, and the number of visitors (range, 33,578–157,976). In addition to these environmental differences, germane to the planned web tool development in the public library setting was computer accessibility. Table 1 shows that the six branches all had >10,000 computer sessions in 2017, indicating the appropriateness of these branches as web tool development settings.

Table 1. Library characteristics at CPL branches serving as settings for human-centered design and usability testing of Health for All1.

Table 2 shows the demographic characteristics of the neighborhoods served by the selected CPL branches (CPL Branches A-F). All six branches served neighborhoods where the majority of residents were racial/ethnic minorities, as reflected in the demographic characteristics of the neighborhoods in Table 2. The racial/ethnic and socioeconomic diversity across the six neighborhoods can also be gleaned from Table 2. Two branches (A, F) representing the Southern and Central districts of Chicago were in neighborhoods that were >90% Black or African American, one branch (E) was in a neighborhood that was >80% Hispanic or Latino, and three branches (B-D) were in neighborhoods which were 45–50% Hispanic or Latino, 29–45% White, and diverse across socioeconomic characteristics, particularly the percentage of residents who were college graduates. Thus, the library characteristics of the selected CPL branches and the racial/ethnic and socioeconomic diversity across the communities they serve both support the appropriateness of these six branches as settings for community-based, human centered design of a web tool intended to serve health disparities populations, and observational visits to the library branches by research team members were confirmatory of this.

Table 2. Demographic characteristics of Chicago neighborhood locations of CPL branches participating in human-centered design and usability testing of Health for All1.

Human-centered design of the web tool Health for All

Phase 4 of development of the web tool Health for All (Fig 1) entailed seven iterative community-based, human-centered design sessions with paper prototyping of the web interface at CPL Branches A-F. Table 3 shows the demographic characteristics of the focus groups participating in iterative human-centered design sessions at the CPL branches. We note that the total number of participants was not affected by cancellation or no-shows in any of the focus groups, because focus group recruitment was primarily conducted on-site shortly before each session began. The initial paper prototype displayed the content of the Research Literacy Support (RLS) flashcard tool [5]; for an example of a flashcard in the tool, see Fig 2.

Fig 2. Flashcard example from the Research Literacy Support (RLS) tool.

Table 3. Demographic characteristics of focus groups participating in human-centered design of the web tool Health for All.

Design Session 1.

There was extensive discussion among focus group participants about barriers to participating in clinical trials faced by African Americans and other minorities and past experiences. Among the focus group participants, both library patrons and library staff expressed that they distrusted clinical research including clinical trials and referenced Henrietta Lacks and Tuskegee. Focus group participants viewed the paper prototype of Health for All as too formal, too impersonal, and too distant. The prototype elicited responses of distrust and skepticism and was considered uninviting. This general response was compounded by the distrust participants expressed about clinical trials and the medical community. Modifications of the tool recommended in the group discussion were to add contact information, incorporate personal stories, and make the tool modular for the users for customizable user experiences. Participants viewed the library setting favorably; they related that the social aspects of the library were a major draw, and that the availability of computers was important to them.

Design Session 2.

The strong distrust of clinical research and trials expressed in Design Session 1 was not expressed by this group. Multiple participants related personal or family experience with clinical trials, and they discussed having seen clinical trial recruitment notices on buses, in mailings from local universities, and on the community board of the library. Provider recommendation of clinical trials was described as the most important consideration in making a decision about whether or not to join clinical trials. Consonant with the level of trust in the medical community expressed by this group, participants described the paper prototype as "informative," "easy to understand," and "just fine." When asked, they stated that they did not find the tool to be “impersonal” or too “formal,” and they noted that they did not want a website that was overly simplistic. Similar to the tool feedback in Design Session 1, participants brought up that people have different learning styles, and they recommended having different options to meet these diverse preferences. In response to this feedback, we refined the prototype as a fact-filled version (Fig 3) and also created a vignette version of the prototype, which featured live-action video-based stories (Fig 4). Regarding the setting, participants shared that they generally come to the library for books and events, and less often for computer access.

Fig 3. Fact-filled version of the Health for All prototype developed after design Session 2.

Fig 4. Vignette version of the Health for All prototype developed after design Session 2.

Design Session 3.

A notable theme in this session was the group’s perception of clinical trials as being pertinent only when a person is ill and a lack of awareness of participation of healthy volunteers in clinical trials. This was consistent with views expressed in Design Sessions 1 and 2, and indicated the importance of addressing this misperception in the content of Health for All. Participants did not distrust clinical trials and recognized diversity in clinical trials as important. Participants were presented fact-filled and vignette versions of the prototype, and both were deemed usable by the group, although the fact-filled version was viewed as better organized. There was much discussion about the target audiences for these prototype versions. The fact-filled version was viewed as more useful for someone seeking information quickly, whereas the vignette version was seen as more suitable for those who hold skepticism or distrust of clinical trials. The group emphasized the need for multiple formats to accommodate personal learning style, range of questions/concerns, education level, and accessibility needs. This raised the question of how to filter audiences to a more personalized experience. With respect to the library setting, multiple participants noted that this library branch was quieter than most and a good venue for study and computer access. Given the very different perspectives of the focus groups participating in Design Sessions 1–3, we decided not to iteratively modify the prototypes based on focus group feedback for the remaining sessions and instead make further modifications at the end of this phase of web tool development.

Design Session 4.

Fear and distrust around clinical trials were discussed, but not necessarily distrust for research or the medical system. The fear and distrust were centered on the risk of taking a medication where the full effects and long-term risks are unknown. The vignette version of the prototype seemed preferred by the group overall. Notably it was viewed as being more persuasive and convincing than the fact-filled version. Further, participants noted that they would want to speak with an informed person before engaging the fact-filled version. However, participants still thought that the information in the fact-based prototype could have more answers to the kinds of questions people would have in an actual clinical trial. Focus group members used the library for diverse purposes, such as access to children’s programming and a quiet place to study; many discussed using the library to look into medication, medication interactions, and new diagnoses.

Design Session 5.

The participants of this focus group expressed that they were very receptive to clinical trials. However, multiple participants noted the need to be within the included age range and to have the studied ailment as limiting enrollment opportunities. They also expressed interest in new clinical trials as they get older and have new diagnoses, e.g., hearing loss. The group was not aware that racial/ethnic minorities are underrepresented in clinical trials. Participants liked both versions of the prototype, and there was not a consensus about which was preferred. One participant preferred the fact-filled version because they enjoy reading whereas another compared it to a user agreement that people never read. The group agreed that the vignette version helped with added context, but one participant said it reminded them of work trainings. Participants in this design session all reported using the library for social/recreational reasons, and their visit frequency ranged from twice a month to every day.

Design Session 6.

A theme of distrust of clinical trials arose in this session. The group consensus was that the African American community, particularly men, don't trust clinical trials out of historical and political concerns. There was also a consensus that people of color are largely excluded from clinical trials, because they are not recruited, and the trials don't provide their communities access. The group felt that the tool would be useful for those who read but would not reach those with other learning styles or short attention spans. Alternative formats suggested included audio (podcasts) and video clips of conversations. A live format was also recommended to sustain interest. Regarding the library setting, participants said that they regularly visited the library to browse/borrow DVDs, books, newspapers, and to access computers.

Design Session 7.

Participants were aware of the need for more diverse clinical trials. Multiple participants pointed out that the underrepresentation of minorities in clinical trials results from multiple barriers in addition to lack of knowledge, e.g., socioeconomic factors, language barriers, and lack of diverse providers were mentioned. Participants felt strongly that personal vignettes and government resources were preferred over other content as trustworthy sources. Due to time constraints, library utilization was not discussed.

New themes did not arise after the sixth design session, suggesting that saturation was reached and supports that the sizes of the individual focus groups and the number of design sessions held were sufficient for completion of this phase of development after seven sessions. The data from Design Sessions 1–7 led to our team’s design decision of a Health for All website that was a hybrid of the fact-filled and vignette paper prototypes divided into 4 modules (trust, diversity, healthy volunteers, pros/cons), each with their own outcome metrics. The Trust module served to address the theme of distrust of clinical trials that was emphasized at some design sessions. The Diversity module was to address the need for increased awareness of the underrepresentation of racial/ethnic minorities in clinical trials. The Healthy Volunteers module addressed the fact that our design sessions showed us that potential clinical trial healthy volunteers would have different informational needs and different concerns than other users. Finally, the Pros/Cons module was chosen because design session participants commonly described someone looking at this website after a conversation with a doctor, or to prepare for a conversation with a doctor about a clinical trial. Next, our technical consultant completed an InVision App click-through wireframe of a “Guided Modules” prototype and a WordPress build of a “Single Scrolling Page Modules” prototype. The latter was a fully functional website, except that it did not include the feedback questions we developed for the end of each module. Initial feedback on these prototypes was provided by leadership from local medical libraries and CPL, CPL library branch managers, and other members of the partnership. There was a positive consensus on the readability and understandability of the text and ease of navigation; minor modifications were made to wording and icon choice in the prototypes as a result of this feedback.

Web tool usability testing

Phase 5 of the development of the web tool Health for All was the usability testing of the website prototype by CPL library patrons (Fig 1); this entailed two rounds of testing and website modification. Usability testing was conducted as one-on-one sessions between each participant and a member of the research team; in addition, another member of the research team took notes at these sessions. The demographic characteristics of the convenience sample of library patrons participating in the first round of usability testing are shown in Table 4. Based on observations that five participants are required per formative test cycle to detect over 80% of issues in usability testing [26], we conducted two iterative cycles of usability testing with totaling 13 (Round 1) and 11 (Round 2) participants.

Table 4. Demographic characteristics of library patrons participating in usability testing of the web tool Health for All.

The usability testing script is provided in Table 5. For the list of tasks performed, see Table 6.

Table 5. Usability testing script for Health for All development.

Table 6. List of tasks in the usability testing phase of Health for All development.

In Round 1 of usability testing, participants were each given ten tasks that required navigation of the single scrolling website to specific content or feedback about understanding of specific content, and then were asked to compare the website with the guided module prototype. When asked, most (12/13) indicated that they found the content in the website easy to understand; it was noted that the content was free of medical jargon and accessible to native English speakers. When asked specifically about the definition of “control group” on the website, 11/13 found it easy to understand. The website was considered a trustworthy source of information by 8/13 participants; two participants noted that they would need contact information to trust the website, and one noted that they would trust it only if it was recommended by their physician. The website was viewed as difficult to navigate by 6/13 participants and easy to navigate by the rest. The key navigation-related usability issues revealed in the first round of testing included: 1) the lack of a home button made navigation to the home page difficult for some, as navigation entailed clicking on the Health for All logo; 2) hyperlinks were not readily visualized by some because they were not sufficiently distinct; 3) font size was too small for some to easily decipher; and 4) there was not a one-click path to view all external resources. The majority (8/13) preferred the guided modules platform, finding it more compact, better organized and easier to quickly navigate; those who preferred the single scrolling platform indicated they found it simpler to navigate. Issues identified in this round of usability testing were remedied by further refinement of the website and guided modules prototypes, resulting in version 2 of these platforms (Fig 5).

Fig 5. Version 2 of the guided module Health for All prototype.

Eleven library patrons participated in Round 2 of usability testing with version 2 of the single scrolling website and the guided module prototype; their demographic characteristics are shown in Table 4. Participants were asked to complete the same ten tasks with the single scrolling website as those employed in Round 1 and to compare the single scrolling website with the guided module prototype. The majority (8/11) found the website content easy to understand, 8/11 considered the definition of “control group” easy to understand, 6/11 viewed the website as a trustworthy source of information, and 6/11 found the website easy to navigate. The guided module prototype was the platform preferred by 4/11 participants, while 6 participants did not express a preference. Minor adjustments were made to version 2 following usability testing, resulting in version 3, which we launched as a beta version of a hybridized single-scrolling and guided module prototype to allow further development with website analytics. The Health for All web tool is available for public access at


In this report, we describe the iterative development of the web tool Health for All through community-based participatory research and human-centered design in public library settings. The inclusion of Black or African American, Hispanic or Latino, and other diverse library patrons as participants in the human-centered design and usability testing phases of tool development was an intentional means of achieving accessibility and engagement among diverse racial/ethnic groups, as the purpose of the tool is to increase diversity in clinical trials by enhancing research literacy among diverse library patrons and simultaneously giving them access to clinical trial enrollment opportunities. Human-centered design of the web tool entailed seven iterative focus group sessions held at six public library branches serving diverse neighborhoods in Chicago. At these sessions, participants discussed how they use the library and how they view clinical research and the underrepresentation of minorities in clinical trials; they also provided feedback on paper prototypes of the Health for All web tool, such as accommodation of different learning styles and the importance of engendering trust. We determined that the seven focus groups held were sufficient to capture major themes, and built the web tool on that basis. Usability testing was conducted in two iterative rounds by the think-aloud method augmented by audio recording, screen capture, and observation; we included >10 participants in each round. After resolving issues detected, we conducted beta testing with website analytics and then launched the web tool at The methods described can guide future intervention development involving adaptation of health equity tools to digital formats in community settings such as public libraries.

Prior research on clinical trials participation shows that effective methods of outreach and recruitment for underrepresented populations include direct mass mailing, personal contact and word of mouth [27]. Community involvement from partners has strong implications for retention in clinical trials among African American/Latino populations. In addition, cultural adaptation of recruitment and informational materials remains an important part of increasing participation [27]. Underlying these approaches, an establishment of trust between potential participants and the medical researchers is paramount. Through our partnerships with public library branches with deep roots in the communities they serve, our outreach methods included both personal contact and word of mouth, and branch librarians played key roles in these recruitment efforts through the strong trust placed in them by underrepresented minorities in the communities they serve. Thus, our successful recruitment efforts support these prior conclusions when engaging with underrepresented populations. Further, our results suggest that the informational setting (vs. a clinical setting) may prove as a safer space to ask questions about clinical trials and thus open the discussion forum.

Importantly, through its online platform and deployment in public libraries, Health for All is well positioned to ultimately be scaled with potential to reach the large numbers of underrepresented racial/ethnic minorities needed to impact the rates of minority enrollment in clinical research. Public libraries serve as the bedrock resource of information access and knowledge in communities across the US. Importantly, public libraries are free, widely available and often located in underserved communities where they serve as an important resource and gathering place; public libraries also provide public access to computers and the internet [11]. Thus, public libraries are valuable partners for developing educational resources aimed at improving public health, especially within medically underserved communities [1113]. Currently, the potential reach through deployment of Health for All in seven CPL branches has been enhanced by the build out of more virtual programming by CPL, including a dedicated health and science category, and the distribution of Health for All bookmarks at CPL checkout counters. Further, the subject matter scope of Health for All has been expanded to include COVID-19, which disproportionately affects racial/ethnic minorities in the US [28].

Based on our experiences described here, we also recognize librarians as key partners in developing tools to advance health equity, such as Health for All. The librarians participating in Health for All development were seen as arbiters of factual information in their communities, and they served the role of guiding library patrons to finding answers to their questions and relevant community resources. This suggests that public librarians are well positioned to serve as trusted partners in discussing clinical trial basics with patrons. Roles played by librarians in the development of Health for All included active recruitment of library patrons to the study as well as participation in focus groups, where they shared the types of health-related questions they often are asked by patrons, their opinions of clinical trials as they relate to their patrons, and other germane information and perspectives.

During the entire design process of developing the Health for All web-based tool, it was important to always include the librarians and library patrons at the center of our approach. They collectively served as our designers by providing real-time feedback and demonstrating immediate effects of any revisions in our prototypes. Whereas other public health tools may draw from a large body of empirical databases and statistics, our process took into consideration the different factors at play in the different library environments, the dynamics between library patrons and librarian and the perceptions and experiences of the intended users.

Strengths of this study include the grounding of the Health for All web tool development (human-centered design and usability testing) in the community [9,10], the iterative [9] and contextually rich approach taken in tool development, and its predication on the Research Literacy Support flashcard tool [5]. In intervention research, favoring scientific rigor and experimental designs can come at a cost to true community-based involvement in intervention design and testing. In this paper, we report development of a web tool through human-centered design, which is inherently a community-engaged approach, in concert with examination of that process through the scientific rigor of implementation science approaches. Importantly, the process we’ve outlined, informed by implementation science, is a promising strategy that optimizes rigor and community participation and can be replicated without specialized equipment or large sample sizes.

A limitation of this study is that detailed demographic data were not captured to determine whether any participants in the human-centered design and usability testing phases were persons with disabilities or members the LGBTQ community. Another limitation regarding diversity is that there was little or no representation of Pacific Islanders or Native Americans among library patron participants, reflecting the demographic makeup of Chicago, which is predominantly Black or African American, Hispanic or Latino, and White. Acknowledging these limitations, we posit that one cannot create a tool that is genuinely optimized for all audiences. Rather, the optimization of Health for All is community based, and can serve as a starting point for subsequent platforms optimized with the inclusion of other underrepresented groups, such as the LGBTQ community or Native Americans.


Inadequate diversity in clinical trials is widely recognized as a significant contributing factor to health disparities experienced by racial/ethnic minorities and other diverse populations in the US. We report here the community-based, human-centered development of Health for All, a web tool designed to improve research literacy, allay distrust of clinical research engendered by systemic racism and discrimination, and facilitate connection to clinical trials in underserved minority communities through engagement in public library settings. We anticipate that future work to investigate how a tool such as Health for All might support more virtual workflows at public libraries will be buttressed by ongoing virtual adaptations to the COVID-19 pandemic.


We thank the staff and patrons at the CPL and branch libraries for their partnership and participation, without whom this project would not be possible.


  1. 1. US National Library of Medicine. What is Precision Medicine? Accessed on March 17, 2020.
  2. 2. National Institutes of Health. All of Us Research Program Overview. Accessed on March 17, 2020.
  3. 3. Clark LT, Watkins L, Pina IL, Elmer M, Akinboboye O, Gorham M, et al. Increasing Diversity in Clinical Trials: Overcoming Critical Barriers. Curr Probl Cardiol. 2019;44(5):148–72. Epub 2018/12/14. pmid:30545650.
  4. 4. Landry LG, Ali N, Williams DR, Rehm HL, Bonham VL. Lack Of Diversity In Genomic Databases Is A Barrier To Translating Precision Medicine Research Into Practice. Health Aff (Millwood). 2018;37(5):780–5. Epub 2018/05/08. pmid:29733732.
  5. 5. Simon MA, Haring R, Rodriguez EM, Gonzalez E, Kaur JS, Kirschner M, et al. Improving Research Literacy in Diverse Minority Populations with a Novel Communication Tool. J Cancer Educ. 2019;34(6):1120–9. Epub 2018/08/26. pmid:30144005; PubMed Central PMCID: PMC6934084.
  6. 6. Ford JG, Howerton MW, Lai GY, Gary TL, Bolen S, Gibbons MC, et al. Barriers to recruiting underrepresented populations to cancer clinical trials: a systematic review. Cancer. 2008;112(2):228–42. Epub 2007/11/17. pmid:18008363.
  7. 7. Hamel LM, Penner LA, Albrecht TL, Heath E, Gwede CK, Eggly S. Barriers to Clinical Trial Enrollment in Racial and Ethnic Minority Patients With Cancer. Cancer Control. 2016;23(4):327–37. Epub 2016/11/15. pmid:27842322; PubMed Central PMCID: PMC5131730.
  8. 8. Scharff DP, Mathews KJ, Jackson P, Hoffsuemmer J, Martin E, Edwards D. More than Tuskegee: understanding mistrust about research participation. J Health Care Poor Underserved. 2010;21(3):879–97. Epub 2010/08/10. pmid:20693733; PubMed Central PMCID: PMC4354806.
  9. 9. Adam M, McMahon SA, Prober C, Barnighausen T. Human-Centered Design of Video-Based Health Education: An Iterative, Collaborative, Community-Based Approach. J Med Internet Res. 2019;21(1):e12128. Epub 2019/01/31. pmid:30698531; PubMed Central PMCID: PMC6372941.
  10. 10. Kia-Keating M, Santacrose DE, Liu SR, Adams J. Using Community-Based Participatory Research and Human-Centered Design to Address Violence-Related Health Disparities Among Latino/a Youth. Fam Community Health. 2017;40(2):160–9. Epub 2017/02/17. pmid:28207679; PubMed Central PMCID: PMC5319705.
  11. 11. Goytia EJ, Rapkin B, Weiss ES, Golub D, Guzman V, O'Connor M, et al. Readiness and capacity of librarians in public libraries to implement a breast cancer outreach and screening campaign in medically underserved communities. Cancer Control. 2005;12 Suppl 2:13–20. Epub 2005/12/06. pmid:16327746.
  12. 12. Whiteman ED, Dupuis R, Morgan AU, D'Alonzo B, Epstein C, Klusaritz H, et al. Public Libraries As Partners for Health. Prev Chronic Dis. 2018;15:E64. Epub 2018/05/29. pmid:29806580; PubMed Central PMCID: PMC5985906.
  13. 13. Philbin MM, Parker CM, Flaherty MG, Hirsch JS. Public Libraries: A Community-Level Resource to Advance Population Health. J Community Health. 2019;44(1):192–9. Epub 2018/07/12. pmid:29995303; PubMed Central PMCID: PMC6329675.
  14. 14. Torres S, de la Riva EE, Tom LS, Clayman ML, Taylor C, Dong X, et al. The Development of a Communication Tool to Facilitate the Cancer Trial Recruitment Process and Increase Research Literacy among Underrepresented Populations. J Cancer Educ. 2015;30(4):792–8. Epub 2015/03/31. pmid:25820604; PubMed Central PMCID: PMC4591093.
  15. 15. Griffin L, Lee D, Jaisle A, Carek P, George T, Laber E, et al. Creating an mHealth App for Colorectal Cancer Screening: User-Centered Design Approach. JMIR Hum Factors. 2019;6(2):e12700. Epub 2019/05/09. pmid:31066688; PubMed Central PMCID: PMC6530259.
  16. 16. Langford AT, Hawley ST, Stableford S, Studts JL, Byrne MM. Development of a Plain Language Decision Support Tool for Cancer Clinical Trials: Blending Health Literacy, Academic Research, and Minority Patient Perspectives. J Cancer Educ. 2019. Epub 2019/02/11. pmid:30739270.
  17. 17. Institute of Design at Stanford. An Introduction to Design Thinking Process Guide. Accessed on December 24, 2020.
  18. 18. Invision. Human Centered Design. Accessed on December 24, 2020.
  19. 19. Deitte LA, Omary RA. The Power of Design Thinking in Medical Education. Acad Radiol. 2019;26(10):1417–20. Epub 2019/03/15. pmid:30867087.
  20. 20. Stevens VJ, Funk KL, Brantley PJ, Erlinger TP, Myers VH, Champagne CM, et al. Design and implementation of an interactive website to support long-term maintenance of weight loss. J Med Internet Res. 2008;10(1):e1. Epub 2008/02/05. pmid:18244892; PubMed Central PMCID: PMC2483846.
  21. 21. Vilardaga R, Rizo J, Zeng E, Kientz JA, Ries R, Otis C, et al. User-Centered Design of Learn to Quit, a Smoking Cessation Smartphone App for People With Serious Mental Illness. JMIR Serious Games. 2018;6(1):e2. Epub 2018/01/18. pmid:29339346; PubMed Central PMCID: PMC5790963.
  22. 22. Xu J, Chen PY, Uglow S, Scott A, Montague E. Design Challenges and Guidelines for Persuasive Technologies that Facilitate Healthy Lifestyles. Comput Technol Appl. 2012;3(2):140–7. Epub 2012/01/01. pmid:24855501; PubMed Central PMCID: PMC4028115.
  23. 23. Simon MA, Tom LS, Leung I, Taylor S, Wong E, Vicencio DP, et al. Chinese Immigrant Women's Attitudes and Beliefs About Family Involvement in Women's Health and Healthcare: A Qualitative Study in Chicago's Chinatown. Health Equity. 2018;2(1):182–92. Epub 2018/10/05. pmid:30283866; PubMed Central PMCID: PMC6110181.
  24. 24. Joe J, Chaudhuri S, Le T, Thompson H, Demiris G. The use of think-aloud and instant data analysis in evaluation research: Exemplar and lessons learned. J Biomed Inform. 2015;56:284–91. Epub 2015/06/15. pmid:26071683; PubMed Central PMCID: PMC4532606.
  25. 25. Moore M, Bias RG, Prentice K, Fletcher R, Vaughn T. Web usability testing with a Hispanic medically underserved population. J Med Libr Assoc. 2009;97(2):114–21. Epub 2009/05/01. pmid:19404502; PubMed Central PMCID: PMC2670225.
  26. 26. Aiyegbusi OL. Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems. Qual Life Res. 2020;29(2):325–33. Epub 2019/11/07. pmid:31691202; PubMed Central PMCID: PMC6994506.
  27. 27. Yancey AK, Ortega AN, Kumanyika SK. Effective recruitment and retention of minority research participants. Annu Rev Public Health. 2006;27:1–28. Epub 2006/03/15. pmid:16533107.
  28. 28. Tai DBG, Shah A, Doubeni CA, Sia IG, Wieland ML. The Disproportionate Impact of COVID-19 on Racial and Ethnic Minorities in the United States. Clin Infect Dis. 2020. Epub 2020/06/21. pmid:32562416.