https://orcid.org/0000-0002-8024-5424
Figures
Abstract
Background
The National Health Interview Survey (NHIS), a nationwide in-person and telephone survey of the civilian non-institutionalized population, is used to measure progress towards Healthy People objectives on cancer screening. Survey reports have been compared to medical record data to assess validity of these questions; however, few studies have assessed reliability, or response consistency over time. We assessed the test-retest reliability of questions on cervical, colorectal, breast and lung cancer screening from the 2021 and 2022 NHIS.
Methods
We surveyed 1,770 adults ages 18 + years with and without prior screening for breast, cervical, colorectal, and lung cancer from four U.S. health systems (three in Washington state and one in Michigan) to examine validity of the NHIS questions. Among people who completed the validity survey, we randomized individuals to receive an additional survey one month later (N = 944) and three months later (N = 822). We calculated Cohen’s Kappa, Gwet’s AC1, and simple concordance to assess reliability.
Results
Among our study sample, 535 (56.7% of those assigned) to one month follow-up and 537 (65.0%) assigned to three month follow-up completed the reliability survey. Among these, 36 (6.6%) and 94 (17.6%) of those assigned to one and three month follow-up respectively were excluded from reliability analyses because they had additional screening tests after completing the validity survey. Cohen’s Kappa showed strong-to-moderate reliability for being up to date with mammography screening: 0.83 at one month and 0.78 at three months; reliability for other screening tests was considerably lower: 0.61 and 0.60 for cervical, 0.57 and 0.62 for colorectal, and 0.72 and 0.58 for lung and one and three months, respectively.
Citation: Kessler LG, Comstock B, Aiello Bowles EJ, Mou J, Nash MG, Bravo P, et al. (2026) Reliability of cancer screening questions from the National Health Interview Survey. PLoS One 21(7): e0352356. https://doi.org/10.1371/journal.pone.0352356
Editor: David T. Zhu, Virginia Commonwealth University School of Medicine, UNITED STATES OF AMERICA
Received: January 12, 2026; Accepted: June 8, 2026; Published: July 1, 2026
Copyright: © 2026 Kessler et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: A final research data set which constitutes the minimal data set required to replicate the reported study findings in their entirety, including all plots, tables, and summary statistics is available. This final data set is structured to maximize future scientific value while protecting patient and health system privacy. The University of Washington Data Coordinating Center has removed and deidentified all 18 HIPAA-specified direct identifiers. Our data-sharing policy strives for the least restrictive plan possible while providing appropriate protection for participant privacy, health system privacy, and scientific integrity. The exception to providing data covers our cognitive data. Because of the difficulty of de-identifying these cognitive data, we have not made these available. This final study data set is accessible via a supervised private data enclave managed by the University of Washington Center for Biomedical Statistics. The data set contains all data collected under the study protocol, pursuant to de-identification, and a data dictionary will accompany the data set. Access will be limited to registered users who submit proposed questions and analysis plans to medstat@uw.edu and sign a data use agreement. “Supervised” indicates that individual requests are reviewed to protect the intellectual property rights of the project investigative team by restricting external development of manuscripts using the study data that substantially overlap with those already developed by study investigators.
Funding: In 2020, the Centers for Disease Control issued a cooperative agreement to the University of Washington (Coop Agreement #5 U48DP006398-02-00) to assess the validity and reliability of the redesigned questions on cervical, colorectal, breast, and lung cancer screening from the 2020-2022 National Health Interview Survey. Henry Ford received funds from ICF International Inc. (Subcontract GS-00F-010CA/HHSN261201700003B/75N91022F00002/TO9). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Cancer was the second leading cause of death, after heart disease, in the United States (U.S.) in 2024 [1,2]. According to modeling done by the NCI Cancer Intervention and Surveillance Modeling Network (CISNET) program, an estimated 5.94 million cancer deaths were averted for breast, cervical, colorectal, lung, and prostate cancers combined between 1975 and 2020 [3]. Cancer screening efforts played an important role in averting some of these deaths. Routine cancer screenings can lead to prevention or earlier detection for some cancers, notably cervical, breast, colorectal, and lung cancers, before symptoms appear and when cancer is often most treatable [4].
For the past four decades, the U.S. Department of Health and Human Services (US DHHS) has developed measurable public health objectives every ten years, known as the Healthy People objectives. In Healthy People 2030, the objectives focus on promoting evidence-based cancer screening and prevention strategies [5]. Determining whether the US population meets these goals is a critical step in developing and modifying interventions to increase the appropriate use of cancer screenings [6]. To measure progress towards the Healthy People objectives, the US DHHS utilizes data from the National Health Interview Survey (NHIS), a nationwide in-person and telephone survey of the civilian non-institutionalized population conducted by the National Center for Health Statistics, Centers for Disease Control and Prevention (CDC), including cancer screening utilization [7–12].
Four prior studies have evaluated the reliability of cancer screening questions (not necessarily NHIS questions) with mixed results [13–16]. Reliability varied across these studies depending on the cancer screening site with studies showing slightly better reliability for questions about breast cancer screening than colorectal cancer screening. Vernon et al recognized that, “Qualitative data from focus groups showed that many adults do not know or recognize the names of CRCS [colorectal cancer screening] tests and that many were unable to distinguish between SIG [sigmoidoscopy] and COL [colonoscopy] or between home-based and office-based FOBT [fecal occult blood testing].” In our cognitive study of cervical cancer screening, we found similar problems with understanding specific cancer screening tests including a lack of understanding of human papilloma virus testing [17]. We are unaware of studies that have evaluated the reliability of cervical or lung cancer screening questions.
Using a modified version of the NHIS Cancer Control Supplement (CCS), we evaluated reliability of cancer screening questions at four separate health care systems that differ in administrative structures and populations serviced. In this paper, we analyze and present data on the reliability of these questions at intervals of 1 and 3 months after their first survey. Along with the validity results of this study, published elsewhere [18], these reliability estimates will help assess whether the data from the NHIS presents an accurate reflection of the true cancer screening behavior of the respondents to the survey.
Materials and methods
The overall study methods, published elsewhere, are summarized here [19]. We randomized 1,770 adults ages 18 + years to complete a web- or phone-based survey on cancer screening using a modified version of the NHIS CCS questions. The study settings included three health systems in Washington state: University of Washington (UW), Kaiser Permanente Washington (KPWA), and MultiCare Health System (MHS). A fourth setting in Michigan, Henry Ford Health (HFH), was later added to achieve further population diversity and to increase recruitment of participants eligible for lung cancer screenings. We obtained IRB approval to begin collection of electronic health record data on March 21, 2022. We recruited participants from June 1, 2022, through March 31, 2024.
Design
The randomized aspect of this study compared phone- vs. web-based survey questions about breast, colorectal, cervical and lung cancer screening to gold-standard data from electronic medical records (EMRs). To test whether the questions elicit reliable answers, we randomized all participants to receive an additional survey at either 1 or 3 months after their initial survey. We chose the 1-month time interval because it was sufficiently short so that we would expect little to no changes in the interim. The 3 month recall period corresponded to the literature in the field [16]. Figure 1 shows that of the 769 validity surveys completed with an initial assignment of web completion, 379 were randomized to 1 month follow up surveying, with 232 completions (59.4%), and an additional 390–3 month follow up surveying with 205 completions (54.1%). For those with initial assignment of phone completion, we had 1001 total interviews; 565 were randomized to 1 month, with 303 completions (69.5%), and an additional 436–3 months with 332 completions (58.8%) (Fig 1).
Ethical approval
Recruitment, consent, and study procedures were reviewed and approved by the University of Washington Institutional Review Board (STUDY 12071). KPWA, and MHS sites ceded human subjects review to the University of Washington. Henry Ford Health obtained IRB approval from Henry Ford Health Institutional Review Board (STUDY 16261). Verbal and digital consent is documented for each participant in our databases.
Setting
The three Washington health systems recruited people eligible for breast, colorectal, cervical, and lung cancer screening, whereas HFH only recruited people eligible for lung cancer screening, those participants eligible for additional cancer screening at the time of the survey were asked to complete those screening questions.
Stratification and Randomization
We extracted data on prior cancer screening from electronic medical records (EMRs). Participants were stratified by race and ethnicity with the goal of recruiting at least 25% Non-Hispanic Black participants and 25% Hispanic participants for the study population [19].
Reliability assessments
We calculated three standard measures of reliability: simple agreement or concordance from the first to second survey (as a percentage), Cohen’s Kappa which adjusts for chance, and Gwet’s AC1, which also does an adjustment for chance [20,21]. Gwet’s AC1 is a statistical measure used to assess the agreement between two or more raters, calculating the probability of agreement between them beyond what would be expected by chance. This is similar to Cohen’s Kappa but with a slightly different interpretation, as it compares observed agreement with an expected disagreement rate rather than an expected agreement rate [22]. We assessed reliability by screening test and by sex. We also performed sensitivity analyses by web or phone survey mode.
We presented the proportion of respondents showing concordance by cancer and test type and assigned follow-up period and calculated an odds ratio comparing agreement between 1- and 3-month follow-up for each cancer and test type.
Results
Table 1 shows the demographic composition of respondents. For these screening tests, women are eligible for all tests, while men are only candidates for colorectal and lung cancer screening, thus our sample included nearly 70% women. The age distribution skewed towards older ages because of eligibility criteria for screening. Our sample contains 59% White persons as we oversampled for participants in other racial categories. Our fourth site (HFH) only selected people whose EMR characterized as likely eligible for lung cancer screening. We did not see any substantial differences in the demographic characteristics of the 1-month vs. the 3-month survey respondents, with the exception that increasing time between validity and reliability surveys appeared to result in a greater decrease in responding among Black participants as compared with White participants, resulting in a higher proportion of White respondents for validity surveys administered at 3 months.
In Table 2, we examine the relationship between the survey responders at 1 and 3 months and whether they self-reported being up to date for screening to determine if there were any notable differences between these groups and screening reports.
In Table 3, we observe generally high measures of test-retest reliability. Survey respondents reported similarly to the initial survey regardless of whether the questions about screening were asked 1 or 3 months later. While concordance was near or above 80% for surveys administered at 1 or 3 months in all cancer types, this was partly due to chance agreement resulting from the high proportion of participants reporting up-to-date screening. As Cohen’s Kappa and Gwet’s AC1 correct for chance agreement, these statistics tended to be lower. Reliability statistics did not seem to be markedly lower for the 3-month period compared to the 1-month period. While there appears to be a notable degradation in agreement for lung cancer, we also note the confidence intervals overlap.
Table 3 presents reliability statistics by follow-up period, cancer type, and sex of the respondent. Generally, there are no consistent or material differences in the reliability statistics by 1- or 3-month response. However, for lung cancer screening, across all measures of reliability, there seems a trend for men to have lower reliability statistics than women, though the confidence intervals of the reliability measures by sex overlap.
In Table 4, we present reliability statistics by assigned mode, either to phone-based or web-based interviews. The confidence intervals of the reliability statistics overlap with no consistent pattern of reliability that favors one mode of survey administration. We present these data by 1-month and 3-month intervals in S1 Table.
Table 5 shows concordance by survey time period and cancer type and screening type. This also shows that agreement does not differ between 1-month and 3-month surveys for any cancer type.
Discussion
According to the CDC, accurate cancer screening data are necessary for effective public health surveillance that informs resource allocation, program evaluation, and policy decisions aimed at cancer control [23].
Few studies report on the reliability of cancer screening self-reports. We add to that literature, and are generally in agreement with prior work such as that by Bradbury et al. and Vernon et al [14,16]. We noted higher reliability between measures at 1 month and limited degradation in respondent recall of prior screening examinations.
Prior work suggests that for health studies, Cohen’s Kappa should have a more rigorous recommendation than previously thought and we use the ranges from McHugh to describe the strength of our findings [24]. For our reliability statistics we report by cancer type. We note that for breast cancer, the reliability statistics are strong (.80−.90) to moderate (.60−.79), whereas for the other three cancers, they tend to fall in the moderate to weak (.40−.59) categories, recognizing that the simple agreement (concordance) statistics are quite strong. Regarding variations between modalities, our findings are consistent with Vernon et al. (2008) [16], who reported only modest mode-of-administration effects on reliability of colorectal screening questions across mail, telephone, and face-to-face administration. Given the NHIS’s accelerating shift toward mixed-mode (telephone and household) data collection (Sample Adult interviews in 2021 were 62.8% telephone-administered), the absence of a meaningful mode effect in our data is reassuring for the comparability of reliability across NHIS administration modes over time.
We also considered how prevalence imbalance may affect agreement statistics by considering the phenomenon, known as the prevalence paradox or kappa paradox [25–27]. The proportion of agreement will tend to approach 100% as the proportion of ratings becomes more unbalanced, i.e., the proportion of ratings approaches 100% in one category and 0% in the other, even in the absence of any association between the two raters. This can be misleading if a high proportion of ratings fall into one category or the other. Cohen’s Kappa accounts for this by comparing the observed rate of agreement to the rate of agreement which would be expected if the two ratings were unrelated. Therefore, the interpretation of Cohen’s Kappa is the same regardless of the prevalence of each rating, ranging from a value of zero indicating no association between reviewers and one indicating perfect agreement. Gwet’s AC compares the rate of agreement to the expected rate of disagreement and will tend to increase when the proportion of ratings approaches 100% in one category, causing the expected rate of disagreement to approach zero, even if there is no association between reviewers. Therefore, the interpretation of Gwet’s AC changes as the prevalence of each rating changes [28].
Limitations
The context for evaluating these questions that are used in the CDC National Center for Health Statistics’ (NCHS) National Health Interview Survey is one where the primary approach has historically been in the household. The COVID pandemic dramatically changed that practice. For example, in 2021, 62.8% of the Sample Adult interviews and 61.4% of the Sample Child interviews were conducted at least partially by telephone [29]. While the intent of the NCHS is to return to household interviewing, it is likely at least a portion of the survey will be done by phone. Thus, there may be some loss of generalizability from our study to the NHIS. However, while our mode mix (telephone and web, no in-person) differs from the historical NHIS mode mix, it is increasingly similar to the current NHIS administration, which is approximately 60% telephone-based.
Likely more important may be our response rates. Whereas NHIS has experienced at least a 50% response rate to its survey requests, our study had only an 8.9% response rate. Our response rates merit careful interpretation. The 8.9% figure cited above refers to recruitment into the initial (validity) survey from health-system membership rolls and is the relevant denominator for evaluating the representativeness of our sample composition. The relevant denominator for evaluating non-response bias in our reliability estimates, however, is the response rate to the randomized follow-up among those who already completed the validity survey: 56.7% at 1 month and 65.0% at 3 months. Thus, our sample may be biased if survey responders differ from non-responders. This potential bias may be further exacerbated by those who agreed to do a second survey, at either 1 or 3 months, where our retention is quite respectable. It is possible, then, that our data and conclusions might not be generalizable to a more representative population
It is quite possible that this sample overrepresents individuals who are more engaged with healthcare and more likely to remember their screening attendance, potentially inflating reliability estimates. That would suggest that our reliability estimates may be an overestimate of what one would find if any study were able to universally reach a population and ask them screening questions at spaced intervals. Such studies are exceedingly uncommon but would help address questions around generalizability of results in the future.
Additionally, the exclusion of participants who obtained screening between surveys may further bias results by restricting the analysis to a more stable subgroup, thereby overstating agreement.
We also note that we performed our surveys starting in 2022 and ending in 2024. This began during the pandemic when there was a well-documented decrease in cancer screening [30–31]. There is no a priori reason to believe that recalling cancer screening would be more or less accurate with the intervening pandemic. It is, in fact, possible that agreement might be higher because going for screening, certainly early in the pandemic, might have been very salient because clinical contact was so proscribed, particularly early in 2020.
Finally, the majority of our sample is derived from Washington state. Because this area of the country is low in minority representation, we compensated by oversampling non-Hispanic Black and Hispanic participants where we had race and ethnicity information available. Even with that approach, it bears repeating that most of our sample comes from one region of the country. We believe that our findings are robust, however, as in other cases of our limitations, our results across sites are quite similar to each other, suggesting a lack of bias in our predominantly northwest dwelling participants. Based on the overall consistency across test types and the various agreement statistics we examined.
Conclusion
In a study of four health systems, we asked respondents to report their cancer screening experience at intervals of 1 or 3 months after an initial survey and found moderate to strong reliability in responses for breast cancer screening. We also saw little to no diminution of recall from 1 to 3 months in reliability statistics except in the case of lung cancer screening. Overall, we conclude that reliability may not be a major threat to the interpretation of results from asking about cancer screening behavior; however, as our response rates for this study were quite low, we would encourage further study of reliability of cancer screening questions to consider if these are the best ways for the US to monitor cancer screening behavior
Supporting information
S1 Table. Reliability statistics by assigned modality and follow-up.
https://doi.org/10.1371/journal.pone.0352356.s001
(DOCX)
Acknowledgments
We would like to acknowledge the assistance of Dr. Rachel Issaka, Dr. Matthew Triplette, and Dr. Cynthia Ko of UW Medicine, and the KPWHRI Survey Research Program. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention or the National Cancer Institute.
References
- 1. Siegel RL, Giaquinto AN, Jemal A. Cancer statistics, 2024. CA A Cancer J Clinicians. 2024;74:12–49.
- 2. National Cancer Institute. Common Cancer Sites - Cancer Stat Facts. National Cancer Institute. https://seer.cancer.gov/statfacts/html/common.html. Accessed 2025 July 30.
- 3. Goddard KAB, Feuer EJ, Mandelblatt JS, Meza R, Holford TR, Jeon J. Estimation of cancer deaths averted from prevention, screening, and treatment efforts, 1975-2020. JAMA Oncology. 2025;11:162–7.
- 4. Loud JT, Murphy J. Cancer Screening and Early Detection in the 21st Century. Semin Oncol Nurs. 2017;33:121–8.
- 5. Office of Disease Prevention and Health Promotion. Cancer Screening Objectives – Healthy People 2030. https://odphp.health.gov/healthypeople/objectives-and-data/browse-objectives/cancer. Accessed 2025 July 30.
- 6. Office of Disease Prevention and Health Promotion. Browse evidence-based resources - Healthy People 2030. Healthy People 2030. https://odphp.health.gov/healthypeople/tools-action/browse-evidence-based-resources
- 7. Centers for Disease Control and Prevention. About NHIS. National Health Interview Survey. https://www.cdc.gov/nchs/nhis/about/index.html. 2025. Accessed 2025 July 30.
- 8. National Health Interview Survey (NHIS) Cancer Control Supplement (CCS). https://healthcaredelivery.cancer.gov/nhis/. Accessed 2025 July 30.
- 9. Breen N, Kessler L. Changes in the use of screening mammography: evidence from the 1987 and 1990 National Health Interview Surveys. Am J Public Health. 1994;84(1):62–7. pmid:8279613
- 10. Brown ML, Potosky AL, Thompson GB, Kessler LG. The knowledge and use of screening tests for colorectal and prostate cancer: data from the 1987 National Health Interview Survey. Prev Med. 1990;19(5):562–74. pmid:2235923
- 11.
Dawson DA, Thompson GB. Breast cancer risk factors and screening: United States, 1987. 1990.
- 12. Hiatt RA, Klabunde C, Breen N, Swan J, Ballard-Barbash R. Cancer screening practices from National Health Interview Surveys: past, present, and future. J Natl Cancer Inst. 2002;94(24):1837–46. pmid:12488477
- 13. Nelson DE, Holtzman D, Bolen J, Stanwyck CA, Mack KA. Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS). Soz Praventivmed. 2001;46 Suppl 1:S3-42. pmid:11851091
- 14. Bradbury BD, Brooks DR, Brawarsky P, Mucci LA. Test-retest reliability of colorectal testing questions on the Massachusetts Behavioral Risk Factor Surveillance System (BRFSS). Prev Med. 2005;41(1):303–11. pmid:15917026
- 15. Rivera S, Vernon SW, Tiro JA, Coan S, Del Junco D, Chan W, et al. Test-retest reliability of self-reported mammography in women veterans. Prev Med. 2006;42(4):320–6. pmid:16442154
- 16. Vernon SW, Tiro JA, Vojvodic RW, Coan S, Diamond PM, Greisinger A, et al. Reliability and validity of a questionnaire to measure colorectal cancer screening behaviors: does mode of survey administration matter?. Cancer Epidemiol Biomarkers Prev. 2008;17(4):758–67. pmid:18381467
- 17. Higashi RT, Tiro JA, Winer RL, Ornelas IJ, Bravo P, Quirk L, et al. Understanding the effect of new U.S. cervical cancer screening guidelines and modalities on patients’ comprehension and reporting of their cervical cancer screening behavior. Prev Med Rep. 2023;32:102169. pmid:36922960
- 18. Kessler LG, Comstock B, Aiello Bowles EJ, Mou J, Nash MG, Bravo P, et al. Measuring the Validity of Survey Questions on Breast, Cervical, Colorectal, and Lung Cancer Screening. Am J Epidemiol. 2026;:kwag040. pmid:41755645
- 19. Kessler LG, Comstock B, Aiello Bowles EJ, Mou J, Nash MG, Bravo P, et al. Protocol to measure validity and reliability of colorectal, breast, cervical and lung cancer screening questions from the 2021 National Health Interview Survey: Methodology and design. PLoS One. 2024;19(3):e0297773. pmid:38437207
- 20. Cohen J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement. 1960;20(1):37–46.
- 21. Gwet K. Kappa statistic is not satisfactory for assessing the extent of agreement between raters. https://www.scirp.org/reference/referencespapers?referenceid=2208179. 2002. Accessed 2025 July 30.
- 22. Vach W, Gerke O. Gwet’s AC1 is not a substitute for Cohen’s kappa - A comparison of basic properties. MethodsX. 2023;10:102212. pmid:37234937
- 23. Ryerson AB, Massetti GM. CDC’s public health surveillance of cancer. Prev Chronic Dis. 2017;14:E39.
- 24. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. pmid:23092060
- 25. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–9. pmid:2348207
- 26. Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551–8. pmid:2189948
- 27. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol. 1993;46(5):423–9. pmid:8501467
- 28. Vach W, Gerke O. Gwet’s AC1 is not a substitute for Cohen’s kappa - A comparison of basic properties. MethodsX. 2023;10:102212. pmid:37234937
- 29. Centers for Disease Control and Prevention, National Center for Health Statistics. National Health Interview Survey (NHIS). https://www.cdc.gov/nchs/hus/sources-definitions/nhis.htm. 2024. Accessed 2025 July 30.
- 30. Mazidimoradi A, Tiznobaik A, Salehiniya H. Impact of the COVID-19 Pandemic on Colorectal Cancer Screening: a Systematic Review. J Gastrointest Cancer. 2022;53(3):730–44. pmid:34406626
- 31. Allahqoli L, Mazidimoradi A, Salehiniya H, Alkatout I. Impact of COVID-19 on cancer screening: a global perspective. Curr Opin Support Palliat Care. 2022;16(3):102–9. pmid:35862881