https://orcid.org/0000-0001-6047-883X
Figures
Abstract
Introduction
Lack of awareness for chronic kidney disease (CKD), including end stage kidney disease (ESKD) and their management options is a major impediment to patients being able to select and use home dialysis therapies. While some instruments have been developed to measure CKD awareness, we lack validated instruments to evaluate patients’ awareness of ESKD and dialysis modalities. This study is part of multipart project for developing and validating an ESKD-centric disease awareness questionnaire.
Methods
A team of specialty renal care experts developed a 45-items questionnaire encompassing the subdomains of General Kidney Knowledge, CKD Knowledge, and ESKD Knowledge. Item reduction analysis—specifically, calculation of item difficulty and item discrimination index scores—was used to items for further review and potential removal.
Results
Index scores were reviewed in conjunction with consideration of theoretical and substantive item content to reduce the number of items in the questionnaire, resulting in a 32-item questionnaire, retaining 5/10 items in the general kidney knowledge subdomain, 14/21 items in the CKD knowledge subdomain, and 13/14 items in the ESKD knowledge subdomain. Retained items ranged from 0.19 to 0.79 on the difficulty index, and from 0.31 to 0.81 on the discrimination index. Scores for percent correct on the reduced questionnaire spanned 0% to 87.5% correct on the full scale, 0% to 100% correct on the General Knowledge subdomain, 0% to 100% on the CKD Knowledge subdomain, and 0% to 92.3% on ESKD Knowledge subdomain.
Conclusions
The questionnaire developed and refined in this study constitutes a patient disease awareness instrument that spans a range of difficulty, and yet shows strong ability to distinguish between patients with varying levels of disease awareness. This study is the first in part of a multistep project to addresses a gap in measuring ESKD specific knowledge. Accurate assessment of patients’ disease awareness through a validated instrument can allow identification of knowledge domains leading to positive impacts on their healthcare decisions and improve targeted patient education efforts.
Citation: Orozco T, Segal E, Hinkamp C, Olaoye O, Shell P, Shukla AM (2022) Development and validation of an end stage kidney disease awareness survey: Item difficulty and discrimination indices. PLoS ONE 17(9): e0269488. https://doi.org/10.1371/journal.pone.0269488
Editor: Abduzhappar Gaipov, Nazarbayev University School of Medicine, KAZAKHSTAN
Received: November 5, 2021; Accepted: May 22, 2022; Published: September 9, 2022
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Shukla AM reports funding from Department of Veterans Affairs, Health Service Research and Development (I01HX002639), Office of Rural Health (16004), and Clinical Science Research and Development (I01CX001661). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Over 97% of the US incident End-Stage Renal Disease (ESKD) patients are managed by some form of dialysis therapy [1]. Considering that home modalities of dialysis provide equivalent survival and trends for better patient-centered outcomes at a significantly lower healthcare cost (relative to in-center dialysis) [2–7], overwhelming use of in-center dialysis for the management of ESKD in the US has been a concern for over two decades. Thus, most stakeholders in ESKD, including the Center for Medicare and Medicaid Services have repeatedly advocated for increasing informed home dialysis utilization for the management ESKD [8, 9].
However, an individual’s dialysis modality is conventionally recommended to be finalized through patient empowerment and shared decision-making. Informed dialysis selection requires individual patients to comprehend the complex medical, social, and financial aspects of their dialysis options and select the modality of dialysis best suited to their life-style [10–13]. Prior studies have shown that providing comprehensive pre-ESKD education (CPE) to patients with advanced chronic kidney disease (CKD) results in significant increase in patient-centered selection and utilization of home dialysis therapies [14–16]. Longitudinal cohort studies have further shown that such CPE further improves multiple pre-ESKD and post-ESKD outcomes including survivals [17–19]. Unfortunately, providing CPE and affecting patient-centered transition to ESKD requires substantial resources that are poorly compensated by the current reimbursement standards. As a result, few nephrology providers are able to provide CPE to all of their advanced CKD patients [20].
Several investigators have developed survey instruments to assess CKD awareness and knowledge with high degree of reliability and validity [21]. Unfortunately, most of these CKD awareness instruments primarily focus on CKD and have limited items to evaluate the comprehension and awareness of ESKD, including various dialysis modalities. Determining the domains of kidney disease knowledge that are predictive of facilitating patients’ ability to reach informed dialysis decision, and specifically home dialysis selection, may allow better targeting of the educational efforts in advanced CKD patients. Assessment of ESKD knowledge specifically is also important because it will allow examination of which facets of patient ESKD knowledge are associated with increased informed dialysis selection and increased home dialysis use.
The overarching purpose of this study is to conduct a multistep validation process for a questionnaire focused on knowledge and awareness of ESKD, then further evaluate if awareness predicts outcomes in patients’ selection of dialysis modality, and which disease awareness domains are associated with those outcomes. The current study focuses on item reduction analysis—specifically, item difficulty and item discrimination—to evaluate the utility of an initial set of items developed for our questionnaire assessing patients’ kidney disease knowledge.
Methods
The data obtained for this study are a part of a prospective observatory cohort study aimed at evaluating the effects of CKD patients’ knowledge and awareness about kidney disease on their HRQoL and clinical outcomes. The study was approved by the Institutional Review Board at University of Florida. In the first phase of the study, 108 consecutive consenting adults, aged 18 years or older, attending the university specialty nephrology clinic were enrolled after obtaining written informed consent. Baseline assessments included questionnaire for self-reported socioeconomic factors, Charlson Comorbidity Index, HRQoL as measured by the Kidney Disease Quality of Life (KDQoL-36) without the dialysis item, the study questionnaire for kidney disease knowledge assessment, and their renal replacement therapy choices if there was an immediate hypothetical need for initiating dialysis therapy. Once the baseline data was collected, participants are followed prospectively at bi-annual intervals through the electronic health record (EHR) surveillance for major cardiovascular and renal outcomes.
To measure kidney disease knowledge, a team of experts from all parts of specialty renal care team created items with a prompt of “what are the critical bits of information that an advanced CKD patient should know?” This team included a renal dietician, renal social worker, pharmacist, dialysis nurse, and renal providers, which included nephrologist and an advanced nurse practitioner specializing in the care of advanced CKD and ESKD. Multiple-choice questions were designed to span a broad range of difficulty. For example, “which one of the following [response options] is one of the most common causes of kidney failure worldwide?” was designed as a relatively easy question, whereas “if you get Medicare and decide to do in-center hemodialysis, when does your Medicare begin?” was designed as a relatively difficult question.
In total, the authors developed a 45-item questionnaire whose items spanned 9 lower-level topics and could be grouped across three broader, higher-level subdomains: general kidney knowledge, CKD knowledge, and ESKD knowledge (see Table 1). All items were multiple-choice, with four to five total response options and one correct option for each item. While the items were multiple choice, they were scored dichotomously as either correct or incorrect, with item score = 1 for a correct response and item score = 0 for an incorrect response. Missing items scores—items where a participant did not answer—were scored as incorrect. Item scores were then summed within each subdomain to form subdomain scores. Item scores for the entire 45-item scale were summed to form a total kidney disease awareness score.
Item reduction analysis was used to refine and reduce the number of items in a kidney disease knowledge questionnaire. The primary item reduction indices used to identify candidate deletion items were item difficulty and item discrimination. These indices were calculated for the total kidney disease awareness score and for the three subdomain scores.
Item difficulty, perhaps better understood as item easiness, is the proportion of correct responses on the item across the total sample. It is calculated as c/n, where c is the number of correct responses for the item, and n is the number of participants (blank responses were scored as incorrect; Fletcher, 2010). Items were categorized into five groups based on item difficulty (or easiness) score, ranging from “too difficult” (0 to .20) to “too easy” (.80 to 1.0) (Table 2). Given that this is a general-purpose scale for assessing a patient’s awareness of their kidney disease state—as opposed to, for example, a scale which is designed to detect expertise, or otherwise be highly sensitive at the lower and upper ends of disease knowledge—thresholds of .20 and .80 were used for item difficulty. In other words, items where less than 20% or more than 80% of the respondents scored correctly, regardless of their total or subdomain kidney disease awareness scores, were identified for further review and potential removal.
Item discrimination is the item’s ability to distinguish between respondents who score high vs respondents who score low on a subdomain or on the total score. Item discrimination is calculated as (u-l)/n, where u is the score correct by respondents who rate in the upper 1/3 of correct scores, l is the score correct by respondents who rate in the lowest 1/3 of the correct scores, and n is the total number of respondents [22]. Following guidelines and examples from Crocker & Algina (1986) and Rizvi et al., (2017) [23, 24], items were categorized based on their level of discriminatory ability (ranging from items with negative or non-discrimination to excellent discrimination; see Table 3) and a threshold of .30 was used to identify items with good discriminatory ability relative to the item’s respective subdomain score or the total score. Specifically, items with a discrimination index of < .30 were flagged for further review and potential removal.
Item standard deviation (SD), reliability index scores, item correlations with total and subdomain scores, and inter-item correlations are provided in supplementary materials. While the current study focused on item reduction and item-level analysis, Cronbach’s alpha was also computed for the questionnaire total and subdomains, and is included with the supplementary item-level analysis tables. R 4.0.2 (R Core Team, 2021) [25] and Rstudio 1.4.1106 (RStudio Team, 2021) [26] were used to analyze the data. R’s psychometric (v2.2; Fletcher, 2010) package was used to calculate item difficulty index, item discrimination index, and discrimination indices, inter-item correlations, and other item-level metrics [22].
Results
Patient characteristics are provided in Table 4. The mean age was 60.3 years with SD = 17.3. Summary statistics showed a majority of participants to be male (55.6%), white (66.7%), with nearly all (88%) having at least a high school diploma/GED level education. Most participants reported their annual household income level as either <$20,000 USD (30.8%) or $50,000-$100,00 USD (30.8%). Most participants reported living in a household with at least one other person (78.7%) and the median household size was 2. Participants showed a mean Charlson Comorbidity Index of 2.7 (SD = 2.3) and a mean health literacy of 6.5 (SD = 1.1) on Rapid Estimate of Adult Literacy in Medicine—Short Form (REALM-SF) score.
Examination of full-scale item difficulty and full-scale item discrimination index scores indicated 13 items that could be removed from the full original scale. There was a near complete overlap between items identified by our thresholds based on index scores calculated for the total scale score, and items identified by thresholds for index scores calculated for the three subdomain scores. In other words, most items flagged for removal by the total score item exam were also flagged for removal by subdomain score item exams.
There was also considerable overlap between items identified by item difficulty thresholds and item discrimination thresholds. For the full scale (using total score as criterion) and for the CKD subdomain, only one additional item—which medications can help lower protein in urine—was flagged by the item discrimination index, which the item difficulty did not already identify as potential removal items. For the general kidney knowledge and ESKD knowledge subdomains, all items below discrimination index score threshold had already been identified for removal by their too high or too low scores on item difficulty index.
One item from the ESKD subscale in particular emerged as warranting conditional consideration, however; this item asked about timing to place a fistula. Although this item was just beyond the a priori threshold for item difficulty index, it was retained for several reasons. First, although it was outside of the bounds of the item difficulty index (scoring as “too difficult” at .19), it was only just outside the bounds of the threshold by .01. Moreover, it scored as “excellent” (.47) in its ability to discriminate amongst high and low scorers on this disease awareness questionnaire. Finally, there was a substantive rationale to retain this item, as it had a companion question—an item about timing to place a peritoneal dialysis catheter—which met the predefined thresholds for both difficulty and discrimination indices. The content of all other removed items were examined as well, but ultimately their scores on difficulty and discrimination indices resulted in the decision to remove them. Thus, while the team used item difficulty and discrimination index scores as the primary guide for item retention vs. removal, item content and subject matter were also important considerations for each item.
Discussion
Prior studies have shown that improving patients’ knowledge and awareness of their health conditions can have direct positive impacts on their healthcare decisions and clinical outcomes [27]. Similar positive associations have also been shown in patients with CKD. Several studies employing kidney disease education have shown improved CKD awareness positively impacts multiple kidney disease related outcomes, including blood pressure control, vascular access creation, and better control of metabolic imbalances [19, 28–31]. However, these educational domains are focused on individual comorbidities, and substantially different than the knowledge base required to make informed dialysis decision to choose a dialysis modality. Thus, the overall goal of this study is first to create a validated ESKD questionnaire capable of identifying critical deficiencies in individuals’ knowledge base essential for forming an informed dialysis decision, and second, to determine whether targeting these domains can lead to greater informed home dialysis utilization. For this initial step of the project, the goal is to evaluate the quality and utility of items for assessing patients’ level of kidney disease knowledge.
To ensure that we target knowledge critical for patients to form informed dialysis choice, the study team created a comprehensive questionnaire with inputs from all members of the multidisciplinary CKD team. This resulted in an extensive 45-item questionnaire, whose items were then divided by the authors into 3 domains of general kidney disease awareness, CKD awareness, and ESKD awareness. The questions spanned the entire spectrum of difficulty, with some questions, i.e.,”If you get Medicare and decide to do in-center hemodialysis, when does your Medicare begin?” scoring as difficult as 0.04 on the calculated difficulty (or rather, easiness) index, whereas, others, i.e.,”which one of the following is one of the most common causes of kidney failure worldwide?” scoring as high as 0.88.
Psychometric examination of this scale—in particular, examination of item discrimination and item difficulty—revealed items which could be removed to produce a better instrument. The resultant scale had a total of 32 items, which we believe will be much less burdensome for patients and participants to complete, while still showing a sufficient, non-restricted disease awareness score range and ability to distinguish between respondents with high vs low kidney disease awareness. The resultant scale scores ranged from 0.0% to 87.5% for the Total scale score, while subdomain scores ranged 0.0% to100.0% for the General and CKD subdomains, and 0.0% to 92.3% for the ESKD subdomain (see Table 5). This shows that the scale has a sensitive and large range with neither ceiling nor floor effects.
While the items in this instrument were written within a structure of 3 subdomains, including general kidney knowledge and CKD knowledge, the ESKD subdomain was of particular interest, as noted above. Thus, the relatively high proportions of ESKD items (number of ESKD items retained relative to number of General and CKD items retained, and relative to the total number of items in the questionnaire) is ideal for our purposes. The ESKD subdomain had only two items identified for removal based on discrimination and difficulty index scores, but one was retained (see explanation for fistula item above), resulting in 13/14 = 92.9% of the ESKD items retained.
These items together constitute an instrument that shows strong ability to distinguish between high and low disease awareness. There is yet much to learn, however, about the impact of kidney disease awareness on patient outcomes. First, it is still unknown whether this instrument would show similarly high ability to discriminate between high and low disease awareness in other samples of kidney disease patients. Furthermore, it is unclear whether disease awareness is related to ability to make informed decisions about disease management—for example, whether it is predictive of patients’ disease management confidence, attitudes, and behaviors. It would be especially useful to know whether kidney disease awareness, especially ESKD awareness, was predictive of dialysis modality (e.g., in-center vs home dialysis) selection and long-term use.
Assuming that a patients’ disease awareness is related to disease management down the line, it is also unknown whether disease awareness can be improved. Providing comprehensive kidney disease education could increase disease awareness, which should be related to greater ability for patients to make the best choices for themselves and their individual lifestyles regarding disease management. This increased capacity for informed decision making could in turn translate to improved disease management choices and behavior, as alluded to previously, which may have potentials to ultimately influence the quality of life. Thus, development of a questionnaire to measure patients’ kidney disease awareness is an early and important step for assessing the effects of disease awareness, and potentially improving not only patients’ disease awareness but disease outcomes.
There are several limitations to the current work on developing an ESKD-focused kidney disease awareness questionnaire. Boateng et al. (2018) outlined nine steps for scale validation and development, and not all steps have been completed here [32]. For example, while items were developed with consideration from multiple experts in specialty renal care, cognitive interviews were not conducted with a sample from the target audience (kidney disease patients). We plan to address as many of these shortcomings as possible, however, in subsequent work. With the items developed for this questionnaire, future directions for this multipart study including further testing and validation of the instrument itself (e.g., factor analysis of the proposed 3 domain structure), as well as testing against other patient and clinical factors. Specifically, we will test the validity and predictive ability of this questionnaire against quality-of-life measures and other hard outcomes (e.g., dialysis choice and dialysis use). We will test whether kidney disease awareness is associated with patient characteristics such as age, education, or comorbidity. Finally, we will test whether kidney disease awareness can be improved via patient education.
Second, the survey items were developed during the period when there was some ongoing debate in the kidney disease community about patient-centered verbiage—for example, the use of “kidney” instead of “renal” or “nephro-.” As such, items were developed during this time to assess patient comfort with such verbiage. Some such items with older terminology were retained in the reduced questionnaire, but for future implementation of the survey, we will replace “renal” and “nephro-”with “kidney” where possible and appropriate.
Finally, while a 32-item questionnaire would be less burdensome for patients than a 45-item questionnaire, the length of this measure may yet be a barrier to its widespread use. Many routinely used instruments in the kidney disease community, i.e., KD-QOL (36 items) and KiKS survey (28 items) have significant item length as well. We anticipate that our future work (such as factor analysis) may result in further refinement and reduction of items, or result in the construction of short forms for use when the patient has other questionnaires to complete, reserving the full scale for research or clinical situations which require fewer questionnaires. Moreover, further validation of the questionnaire may support administration of items belonging to a specific domain of interest for the practitioner. For example, the clinician may wish to focus on, and thus only administer, the ESKD domain of the questionnaire, which at 13-items may improve its feasibility in routine clinical use. In the meantime, it may be useful to nevertheless provide the current questionnaire for practitioners and researchers who wish to assess and improve patients’ kidney disease awareness.
To summarize, previous research has shown that improving patients’ disease awareness can have positive impacts on healthcare decisions and clinical outcomes. Effectively studying this in kidney disease, and more specifically in ESKD requires an instrument that can accurately assess ESKD specific knowledge. This study seeks to address this existing gap. Developing a targeted questionnaire to assess ESKD knowledge can accurately define the deficiencies in patients’ ESKD awareness. It can further allow assessment of the efficacy of measures aimed to improve these deficiencies and facilitate better targeting of patient education efforts in this area to address the long-term nephrology goals for increased informed home dialysis use.
Supporting information
S1 Appendix. Inter-item correlations original.
https://doi.org/10.1371/journal.pone.0269488.s001
(XLSX)
S2 File. ESKD awareness questionnaire.original.FULL.
https://doi.org/10.1371/journal.pone.0269488.s003
(DOCX)
S3 File. ESKD awareness questionnaire.original.Reduced.
https://doi.org/10.1371/journal.pone.0269488.s004
(DOCX)
References
- 1. Saran R, Robinson B, Abbott KC, Agodoa LYC, Bragg-Gresham J, Balkrishnan R, et al. US Renal Data System 2018 Annual Data Report: Epidemiology of Kidney Disease in the United States. American Journal of Kidney Diseases. 2019;73(3):A7–A8. pmid:30798791
- 2. R BFC, Michael W, Scott WK, Garth M, Narine S-D, Robert RQ, et al. Effect of Frequent Nocturnal Hemodialysis vs Conventional Hemodialysis on Left Ventricular Mass and Quality of Life: A Randomized Controlled Trial. JAMA. 2017;298(11):1291–9. pmid:17878421
- 3. Zazzeroni L, Pasquinelli G, Nanni E, Cremonini V, Rubbi I. Comparison of Quality of Life in Patients Undergoing Hemodialysis and Peritoneal Dialysis: a Systematic Review and Meta-Analysis. Kidney & blood pressure research. 2017;42(4):717–27. Epub 2017/10/20. pmid:29049991.
- 4. Lee MB, Bargman JM. Survival by Dialysis Modality—Who Cares? Clinical Journal of the American Society of Nephrology. 2016;11(6):1083–7. pmid:26912541
- 5. Fadem SZ, Walker DR, Abbott G, Friedman AL, Goldman R, Sexton S, et al. Satisfaction with Renal Replacement Therapy and Education: The American Association of Kidney Patients Survey. Clinical Journal of the American Society of Nephrology. 2011;6(3):605–12. pmid:21330485
- 6. Juergensen E, Wuerth D, Finkelstein SH, Juergensen PH, Bekui A, Finkelstein FO. Hemodialysis and Peritoneal Dialysis: Patients’ Assessment of Their Satisfaction with Therapy and the Impact of the Therapy on Their Lives. Clinical Journal of the American Society of Nephrology. 2006;1(6):1191–6. pmid:17699347
- 7. Liu FX, Walton SM, Leipold R, Isbell D, Golper TA. Financial Implications to Medicare from Changing the Dialysis Modality Mix under the Bundled Prospective Payment System. Peritoneal Dialysis International. 2014;34(7):749–57. pmid:25292402
- 8. Golper TA. A View of the Bundle from a Home Dialysis Perspective. Clinical Journal of the American Society of Nephrology. 2018;13(3):471. pmid:29255064
- 9.
House TW. Executive Order on Advancing American Kidney Health https://www.whitehouse.gov/presidential-actions/executive-order-advancing-american-kidney-health/2019, July 10 [2019, August 23]. Available from: https://www.whitehouse.gov/presidential-actions/executive-order-advancing-american-kidney-health/.
- 10. GALLA JH. Clinical Practice Guideline on Shared Decision-Making in the Appropriate Initiation of and Withdrawal from Dialysis. Journal of the American Society of Nephrology. 2000;11(7):1340–2.
- 11. Barry MJ, Edgman-Levitan S. Shared Decision Making—The Pinnacle of Patient-Centered Care. New England Journal of Medicine. 2012;366(9):780–1. pmid:22375967.
- 12.
ASN Releases Benchmarks for Home Dialysis Training. Kidney news online. 2017.
- 13. Covic A, Bammens B, Lobbedez T, Segall L, Heimbürger O, van Biesen W, et al. Educating end-stage renal disease patients on dialysis modality selection. NDT Plus. 2010;3(3):225–33. pmid:28657058
- 14. Shukla AM, Easom A, Singh M, Pandey R, Rotaru D, Wen X, et al. Effects of a Comprehensive Predialysis Education Program on the Home Dialysis Therapies: A Retrospective Cohort Study. Perit Dial Int. 2017;37(5):542–7. Epub 2017/05/27. pmid:28546368.
- 15. Shukla AM, Hinkamp C, Segal E, Ozrazgat Baslanti T, Martinez T, Thomas M, et al. What do the US advanced kidney disease patients want? Comprehensive pre-ESRD Patient Education (CPE) and choice of dialysis modality. PLOS ONE. 2019;14(4):e0215091. pmid:30964936
- 16. Easom AM, Shukla AM, Rotaru D, Ounpraseuth S, Shah SV, Arthur JM, et al. Home run—results of a chronic kidney disease Telemedicine Patient Education Study. Clinical Kidney Journal. 2019. pmid:33123362
- 17. Devins GM, Mendelssohn DC, Barré PE, Taub K, Binik YM. Predialysis psychoeducational intervention extends survival in CKD: a 20-year follow-up. Am J Kidney Dis. 2005;46(6):1088–98. Epub 2005/11/29. pmid:16310575.
- 18. Mason J, Khunti K, Stone M, Farooqi A, Carr S. Educational Interventions in Kidney Disease Care: A Systematic Review of Randomized Trials. American Journal of Kidney Diseases. 51(6):933–51. pmid:18440681
- 19. Kurella Tamura M, Li S, Chen SC, Cavanaugh KL, Whaley-Connell AT, McCullough PA, et al. Educational programs improve the preparation for dialysis and survival of patients with chronic kidney disease. Kidney Int. 2014;85(3):686–92. Epub 2013/09/27. pmid:24067435.
- 20. Shukla AM, Bozorgmehri S, Ruchi R, Mohandas R, Hale-Gallardo JL, Ozrazgat-Baslanti T, et al. Utilization of CMS pre-ESRD Kidney Disease Education services and its associations with the home dialysis therapies. Perit Dial Int. 2020:896860820975586. Epub 2020/12/02. pmid:33258420.
- 21. Wright JA, Wallston KA, Elasy TA, Ikizler TA, Cavanaugh KL. Development and Results of a Kidney Disease Knowledge Survey Given to Patients With CKD. American journal of kidney diseases: the official journal of the National Kidney Foundation. 2011;57(3):387–95. PMC3053083. pmid:21168943
- 22.
Fletcher TD. CRAN—Package psychometric. https://CRAN.R-project.org/package=psychometric: Comprehensive R Archive Network (CRAN); 2010 08-08-2010.
- 23.
Crocker L, Algina J. Introduction to Classical and Modern Test Theory.: Holt, Rinehart and Winston, 6277 Sea Harbor Drive, Orlando, FL 32887 ($44.75). 1986.
- 24. Rizvi WM, Saeed A, Usman H, Tahira Z, Rubab B, Sadaf K, et al. Difficulty Index, Discrimination Index and Distractor Efficiency in Multiple Choice Questions. https://wwwapimsnet/indexphp/apims. 2018;13(4). https://www.apims.net/index.php/apims/article/view/9.
- 25.
R: The R Project for Statistical Computing 2021. Available from: https://www.r-project.org/.
- 26.
RStudio | Open source & professional software for data science teams 2021. Available from: https://rstudio.com/.
- 27. Gruman J, Rovner MH, French ME, Jeffress D, Sofaer S, Shaller D, et al. From patient education to patient engagement: implications for the field of patient education. Patient Educ Couns. 2010;78(3):350–6. Epub 2010/03/06. pmid:20202780.
- 28. Peng S, He J, Huang J, Tan J, Liu M, Liu X, et al. A chronic kidney disease patient awareness questionnaire: Development and validation. PLOS ONE. 2019;14(5):e0216391. pmid:31050683
- 29. Gheewala PA, Peterson GM, Zaidi STR, Jose MD, Castelino RL. Public knowledge of chronic kidney disease evaluated using a validated questionnaire: a cross-sectional study. BMC Public Health. 2018;18(1):371. pmid:29554891
- 30. Devins GM, Binik YM, Mandin H, Letourneau PK, Hollomby DJ, Barre PE, et al. The kidney disease questionnaire: A test for measuring patient knowledge about end-stage renal disease. Journal of Clinical Epidemiology. 1990;43(3):297–307. pmid:2313319
- 31. Mason J, Khunti K, Stone M, Farooqi A, Carr S. Educational Interventions in Kidney Disease Care: A Systematic Review of Randomized Trials. American Journal of Kidney Diseases. 2008;51(6):933–51. pmid:18440681
- 32. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Health. 2018;6:149. Epub 2018/06/27. pmid:29942800; PubMed Central PMCID: PMC6004510.