Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development and validation of a data quality index for forensic documentation of sexual and gender-based violence in Kenya

  • Rose McKeon Olson ,

    Roles Formal analysis, Methodology, Validation, Writing – original draft, Writing – review & editing

    rolson@bwh.harvard.edu

    Affiliations Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, United States of America, Harvard Medical School, Boston, Massachusetts, United States of America

  • Wendy Macias-Konstantopoulos,

    Roles Data curation, Methodology, Resources, Validation, Writing – review & editing

    Affiliations Harvard Medical School, Boston, Massachusetts, United States of America, Department of Emergency Medicine, Center for Social Justice and Health Equity, Massachusetts General Hospital, Boston, Massachusetts, United States of America

  • Roseline Muchai,

    Roles Conceptualization, Methodology, Project administration, Supervision, Writing – review & editing

    Affiliation Physicians for Human Rights, Boston, Massachusetts, United States of America

  • Katy Johnson,

    Roles Conceptualization, Project administration, Resources, Writing – review & editing

    Affiliation Physicians for Human Rights, Boston, Massachusetts, United States of America

  • Ranit Mishori,

    Roles Conceptualization, Supervision, Writing – review & editing

    Affiliations Physicians for Human Rights, Boston, Massachusetts, United States of America, Georgetown University School of Medicine, Washington, DC, United States of America

  • Brett Nelson

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing – review & editing

    Affiliations Harvard Medical School, Boston, Massachusetts, United States of America, Divisions of Global Health and Neonatology, Department of Pediatrics, Massachusetts General Hospital, Boston, Massachusetts, United States of America

Abstract

Introduction

High-quality forensic documentation can improve justice outcomes for survivors of sexual and gender-based violence, but there are limited tools to assess documentation data quality. This study aimed to develop and validate a data quality assessment index to objectively assess clinician documentation across the 26 key elements of the standardized forensic evidence forms used in Kenya.

Methods

Informed by prior quality assessment tools, an initial draft of the index was developed. Feedback from Kenya- and U.S.-based clinicians and human rights experts was solicited and incorporated into the draft index in an iterative fashion. Two raters independently employed the finalized Physicians for Human Rights Data Quality Index to assess and score the quality of documentation across 31 clinician-completed forms. Inter-rater reliability was determined using Cohen kappa (к) coefficients.

Results

The Index was found to have substantial overall reliability. Of the 26 documentation items, the Index had a perfect (к = 1.0) and almost perfect (к = 0.81–0.99) level of inter-rater agreement across 17 (65.4%) and 5 (19.2%) items, respectively. On a low-to-high documentation quality scale of 0 to 2, the majority of items (n = 19, 73.1%) had a mean documentation quality score >1.5–2.

Conclusion

Quality assurance of forensic documentation is an essential component of post-sexual assault care. To our knowledge, this is the first validated quality-assessment tool in the peer-reviewed literature for sexual assault documentation and may be a promising strategy to enhance the quality of sexual assault documentation in other settings, locally, regionally, and internationally.

Introduction

Sexual and gender-based violence (SGBV) is a serious issue that affects millions worldwide, impacting people of all genders, ages, and sexual orientation. Sexual violence includes any sexual act or attempted act where consent is not obtained or freely given, often through use of violence and coercion [1]. The World Health Organization (WHO) estimates that approximately 30% of women worldwide have experienced physical and/or sexual violence by an intimate partner or non-partner in their lifetime; significant numbers of men and boys also experience sexual violence [23]. Rates of sexual assault are similar in Kenya [46], where intimate partner violence has been named one of the top ten leading risk factors driving combined death and disability [7]. Sexual violence is also a major contributor to a broad range of physical, psychological, social, legal, and economic consequences that adversely affect survivors, families, communities, and society at large [4, 5].

Survivors of sexual assault deserve timely and high-quality forensic examination, evidence collection, and documentation as part of comprehensive care for survivors. High-quality documentation of the clinical exam after sexual assault has been shown to increase trial, prosecution, and conviction rates of perpetrators [811]. A South African study analyzed the association of sexual assault injury documentation and legal outcomes, and found that conviction was more likely when cases had documented injuries, whether nongenital or ano-genital injuries [12]. Furthermore, an evaluation conducted in Kenya found that the relative amount of medical evidence that appeared in the Post-Rape Care (PRC) Form legal record was associated with an increased likelihood of an adjudication outcome favoring the survivor [9]. In addition to legal justice outcomes, timely evidence collection may have other positive effects, such as enhancing survivor agency, and empowering and validating the experience of survivors.

While high-quality documentation by health care professionals can improve justice for survivors, methodology to grade the quality of SGBV documentation is lacking. There are no published validated tools on quality assessment of sexual assault documentation. One non-peer reviewed index developed in South Africa found wide variability in data quality of post-rape documentation forms, depending on profession and location of data collection [13]. While problematic, this finding is a natural consequence of the wide variability in the quality, components, and professional training level of post-sexual assault evidence collection. In Kenya and in many other under-resourced contexts, there have been reports of low-quality medico-legal documentation after sexual assault [14, 15]. For example, in several contexts the sexual assault exam is heavily based on a hymen examination, which is not an accurate or reliable indicator of sexual assault [16]. These findings suggest that data quality assessments are underutilized, and their appropriate use may strengthen medico-legal evidence and thereby increase trial, prosecution, and conviction rates of perpetrators of sexual violence [10.]

The international nonprofit organization Physicians for Human Rights (PHR) partners with medical, legal, and law enforcement professionals in Kenya, the Democratic Republic of Congo, and beyond to improve the medico-legal response to sexual violence and bolster accountability for associated crimes. PHR focuses on improving the quality and availability of forensic evidence through research, tools, and innovations. Working in close collaboration with multisectoral partners, PHR developed MediCapt, a mobile application that enables clinicians to securely document evidence of sexual violence and safely transmit and store the protected information. As part of an evaluation of the MediCapt project in Kenya, PHR worked with external evaluators to explore the option of a data quality index to eventually compare the quality of standard government-issued paper-based PRC forms with PHR’s mobile MediCapt application.

The aim of this study was to develop and evaluate the PHR Data Quality Index to objectively assess the data quality and inter-rate reliability of forensic evidence documentation of sexual assault by clinicians in Kenya.

Materials and methods

The Kenyan government’s PRC Form is a two-page, triplicate form used by clinicians in Kenya to document survivor-reported sexual assault events and includes the physical examination, psychological assessment, and clinical management by the clinician. The form is divided into two sections: Part A, the description of the incident, the physical examination findings, and the documentation of the clinical management and forensic evidence; and Part B, the psychological assessment.

Informed by prior unpublished quality assessment tools [13, 17], an initial draft index was developed to objectively assess key elements of the PRC Form. We defined data quality within its six well-established dimensions: accuracy, completeness, consistency, timeliness, validity, and uniqueness [18]. The index was designed to specifically target the key components of the Kenyan government PRC Form and included a weighted scoring for each item assessed based on the quality of the data documented in the two-page paper form.

This draft index was subsequently shared in an interactive and iterative process with experienced Kenya- and U.S.-based human rights experts and clinicians. Their feedback informed a revision of the index, which was then shared with an eight-member group of Kenyan clinical, legal, and law enforcement professionals, as well as members of the PHR network in Kenya with long involvement in the care of survivors of SGBV. During a semi-structured videoconference, these professionals provided additional feedback on elements of the PRC Form most critical for documentation and prosecution of SGBV. The index, including item scoring weights, were revised accordingly.

Using the finalized PHR Data Quality Index, two reviewers (RO, BDN) independently scored each of 31 completed post-rape forms for the Index’s 26 quality-metric items. All items were scored on a scale of 0 (no data or low quality), 1 (moderate quality), or 2 (high quality), with the exception of item Part B, the psychological assessment, where a scale of 0, 2, and 4 was used to place greater weight on this large component of the PRC Form and to allow for a more granular quality assessment of the psychological narrative. During initial scoring attempts, lack of clarity on some Index items was discussed by the researchers and addressed by adding a scoring guide to each of the Index items. Independent scoring was then repeated using this finalized Index.

Level of inter-rater reliability was determined with SPSS 26.0 (IBM, Armonk, NY, USA) using Cohen kappa coefficients. These coefficients were interpreted according to the following definitions: poor agreement (0.00), slight agreement (0.01–0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), almost perfect agreement (0.81–0.99), and perfect agreement (1.00) [19].

This study was approved by the Georgetown University institutional review board in the United States. (Protocols 2016–0661 and 2016–1404) and the Egerton University institutional review board in Kenya (Approval Number EUREC/APP/099/2020).

Results

Inter-rater reliability of the PHR Data Quality Index

The finalized Data Quality Index (Table 1) includes 26 data quality items and is presented below.

thumbnail
Table 1. PHR data quality index for assessing quality of sexual violence documentation.

https://doi.org/10.1371/journal.pone.0262297.t001

The overall kappa score for the PHR Data Quality Index was 0.77, corresponding to a substantial level of agreement (Table 2). In six of the seven multi-item Index categories, independent scoring for at least half of the category items had kappa scores of 1.00, indicating a high inter-rater reliability. All items within four Index categories (demographics, management, laboratory samples, and psychological assessment) had perfect levels of agreement across the two independent raters, with itemized kappa scores of 1.00.

thumbnail
Table 2. Inter-rater reliability results for each index item and for the PHR Data Quality Index overall.

https://doi.org/10.1371/journal.pone.0262297.t002

Items with a perfect agreement score (kappa 1.00) included information such as the dates of the incident, exam, and form completion; survivor name, date of birth, and contact information; perpetrator information, including body marks; information regarding care management, referrals, and laboratory studies; police officer date and signatures; and legibility. Most common errors in completing the form were reflected in items with lower agreement scores, including chief complaint, circumstances surrounding the incident, summary body map statement, and examining officer date and signature. Table 3 provides a summary of the Index’s item-by-item inter-rater reliability by level of agreement.

thumbnail
Table 3. Summary of inter-rater reliability by level of agreement.

https://doi.org/10.1371/journal.pone.0262297.t003

Applying the PHR Data Quality Index to assess quality of forensic documentation

To understand which of the Index’s 26 items may be more challenging for quality documentation, a mean data quality score was determined for each Index item. With a maximum data quality score of 2, the large majority (n = 19, 73.1%) of the 26 Index items received a mean data quality score of ≥1.5–2, indicating high-quality documentation by clinicians (Table 4). For the purposes of comparison, Item #24, which is typically scored out of 4, had an adjusted score of 1.81 when adjusted to a two-point scale. Four (15.4%) Index items received a mean rater score of >1.0–1.5, indicating moderate-quality documentation. These items included data on orphans and vulnerable children (OVC) status (mean = 1.48, к = 1.00), chief complaints (mean = 1.48, к = 0.60), circumstances surrounding the incident (mean = 1.32, к = 0.63), and summary statements of genital exams (mean 1.44, к = 0.96). Three (11.5%) items with low-quality mean scores ≤1.0 for documentation included date of last consensual intercourse (mean = 0.84, к = 1.00), police officer signature and date (mean = 0.13, к = 1.00), and document signed by the examining officer within 48 hours of patient visit (mean = 0.13, к = 1.00).

thumbnail
Table 4. Summary of mean rater data quality scores for item data reported on the PRC Form.

https://doi.org/10.1371/journal.pone.0262297.t004

Discussion

High-quality forensic documentation can facilitate increased investigation, prosecution, and conviction rates for survivors of sexual violence [810], yet no validated, published tools are available to assess the quality of documentation after sexual assault. To our knowledge, this is the first peer-reviewed study to develop a validated quality-assessment tool for sexual assault documentation. The PHR Data Quality Index had substantial inter-rater agreement, suggesting it is a valid tool to grade quality of sexual assault documentation and guide targeted interventions to improve data quality and the overall response to sexual violence. Additionally, the Index could be used more broadly to accelerate Sustainable Development Goal 5.2, to end all forms of violence against women and girls [20].

There was perfect inter-rater agreement for many categorical and nominal variables on the sexual assault documentation forms, such as survivor demographics. There were lower agreement rates for more subjective items, such as chief complaint and circumstances surrounding the sexual assault incident. However, not all subjective variables had poor agreement; in fact, scoring of the psychological assessment and legibility both had perfect agreement. This suggests that more subjective variables with lower inter-rater agreement may have the capacity to improve rating scoring either through adjustment of the variables or improved user guidance. As this quality assessment tool is implemented in the Kenyan context, researchers will continue to evaluate how lower-scoring measures can be optimized for improved agreement. The PHR team plans to utilize the validated Index to compare quality assessments between post-sexual assault paper-based forms to those collected via the MediCapt app, a digital form platform.

This present Index was developed for Kenyan medical professionals; however, it highlights the need to develop similar validated data quality indices for sexual assault documentation in other parts of the world. Stakeholders in the assessment of sexual assault in other contexts may review the validated tool to assess its applicability to other widely used forms and their unique environment and consider adaptation for implementation in their health care facilities. Drawing from sexual assault research such as the present study, sexual assault experts should identify global, standardized measures for high-quality sexual assault documentation and develop a validated global standard for quality assessment of sexual assault documentation that could be adapted to local needs and forms.

The present PHR Data Quality Index could be adapted for use in multiple contexts, such as future sexual violence research, health professional training, program evaluation, and targeted quality improvement post-training interventions. Research may be performed to test its validation in other contexts and to identify which documentation measures could be enhanced or added. The Index could be used for health professional education, including undergraduate, graduate, and continuing education to improve sexual assault examination and documentation. Additionally, sexual assault programs may choose to use the Index to assess the quality of the current sexual assault documentation practices, and target weaknesses and thereby enhance quality of documentation and, consequently, care for survivors.

The study has multiple strengths. The PHR Data Quality Index included feedback from several global and local sexual violence clinicians and human rights experts as well as, most importantly, experienced Kenyan health care professionals who use the sexual assault forms in the field. The Index went through several iterations of review by multidisciplinary professionals before the final Index was determined. Inter-rater reliability testing of the Index showed substantial agreement overall.

There are limitations to this study, including the use of the kappa coefficient. While it is commonly used in statistics, some researchers argue it may be too lenient for health-related studies [21]. To address this intrinsic limitation of the kappa statistic, we included percent agreement alongside kappa coefficients, as suggested by several health services researchers [21]. An additional limitation of the study is its external validity, as it was developed using Kenyan post-sexual assault forms and may not be generalizable to other contexts and geographies. Future validation studies should include indices specific to the sexual assault forms from other geographies. Lastly, it is important to consider that the PHR Data Quality Index does not, in its current form, make a broad-scope assessment of external data accuracy.

Conclusion

This study reports the development of a novel data quality index for sexual assault documentation. The index had substantial reliability, making it the first published validated quality-assessment tool for sexual assault documentation. The high inter-rater reliability suggests that the Index may be a promising strategy to enhance the quality of sexual assault documentation in other countries, with the goal of improving health care and justice for survivors.

References

  1. 1. Garcia-Moreno C, Jansen HA, Ellsberg M, Heise L, Watts CH. Prevalence of intimate partner violence: Findings from the WHO multi-country study on women’s health and domestic violence. The Lancet. 2006;368(9543): 1260–1269.
  2. 2. García-Moreno C, Pallitto C, Devries K, Stöckl H, Watts C, Abrahams N. Global and regional estimates of violence against women: Prevalence and health effects of intimate partner violence and non-partner sexual violence. World Health Organization; 2013. ISBN: 9789241564625.
  3. 3. Borumandnia N, Khadembashi N, Tabatabaei M, Alavi Majd H. The prevalence rate of sexual violence worldwide: a trend analysis. BMC Public Health. 2020;20(1): 1835. pmid:33256669
  4. 4. United Nations Children’s Fund Kenya Country Office DoVP, National Center for Injury Prevention and Control, U.S. Centers for Disease Control and Prevention, and the Kenya National Bureau of Statistics. Violence against children in Kenya: Findings from a 2010 national survey. Summary report on the prevalence of sexual, physical and emotional violence, context of sexual violence, and health and behavioral consequences of violence experienced in childhood; 2012.
  5. 5. Kabiru CW, Mumah JN, Maina BW, Abuya, BA. Violence victimisation and aspirations–expectations disjunction among adolescent girls in urban Kenya. International Journal of Adolescence and Youth. 2017;23(3): 281–290.
  6. 6. 2014 Kenya Demographic and Health Survey. Kenya National Bureau of Statistics, Kenya Ministry of Health, the National AIDS Control Council (NACC), the National Council for Population and Development (NCPD), and the Kenya Medical Research Institute (KEMRI). Kenya National Bureau of Statistics; 2014. https://dhsprogram.com/pubs/pdf/fr308/fr308.pdf.
  7. 7. Kenya Institute for Health Metrics and Evaluation. IHME; 2015 [cited 2021 Feb 3]. http://www.healthdata.org/kenya.
  8. 8. McGregor MJ, Du Mont J, Myhr TL. Sexual assault forensic medical examination: Is evidence related to successful prosecution? Ann Emerg Med. 2002;39(6): 639–647. pmid:12023707
  9. 9. Gray-Eurom K, Seaberg DC, Wears RL. The prosecution of sexual assault cases: Correlation with forensic evidence. Ann Emerg Med. 2002;39(1): 39–46. pmid:11782729
  10. 10. Kjærulff MLBG, Bonde U, Astrup BS. The significance of the forensic clinical examination on the judicial assessment of rape complaints—developments and trends. Forensic Sci Int. 2019;297: 90–99. pmid:30797159
  11. 11. Tamamyan H, Armas-Cardona G. Deepening and expanding the cross-sector network response to sexual violence in the Democratic Republic of Congo and Kenya: A project to increase justice for women and girls survivors of sexual violence. UN Women; 2019. https://untf.unwomen.org/en/learning-hub/evaluations/2019/02/final-evaluation-deepening-and-expanding-the-cross-sector-network-response-to-sexual-violence.
  12. 12. Jewkes R, Christofides N, Vetten L, Jina R, Sigsworth R, Loots L. Medico-legal findings, legal case progression, and outcomes in South African rape cases: Retrospective review. PLOS Med. 2009;6(10): e1000164. pmid:19823567
  13. 13. Mathews MA. Development of a quality index tool to assess the completion of j88 forms for rape survivors in South Africa. Thesis, University of the Witwatersrand, Johannesburg. 2016. http://hdl.handle.net/10539/22538.
  14. 14. Ajema C, Mukoma W, Kilonzo N, Bwire B, Otwombe K. Challenges experienced by service providers in the delivery of medico-legal services to survivors of sexual violence in Kenya. J Forensic Leg Med. 2011;18(4): 162–166. pmid:21550565
  15. 15. Wangamati CK, Combs Thorsen V, Gele AA, Sundby J. Postrape care services to minors in Kenya: Are the services healing or hurting survivors? Int J Womens Health. 2016;8: 249–259. pmid:27445506
  16. 16. Olson RM, García-Moreno C. Virginity testing: A systematic review. Reprod Health. 2017;14. pmid:28521813
  17. 17. Mishori R, Anastario M, Naimer K, et al. mJustice: Preliminary Development of a Mobile App for Medical-Forensic Documentation of Sexual Violence in Low-Resource Environments and Conflict Zones. Glob Health Sci Pract. 2017;5(1): 138–151. pmid:28351881
  18. 18. Defining data quality dimensions. DAMA United Kingdom; 2013. http://www.dama-nl.org/wp-content/uploads/2020/09/DDQ-Dimensions-of-Data-Quality-Research-Paper-version-1.2-d.d.-3-Sept-2020.pdf.
  19. 19. Viera AJ, Garrett JM. Understanding interobserver agreement: The kappa statistic. Fam Med 2005;37(5): 360–3. pmid:15883903
  20. 20. García-Moreno C, Amin A. The sustainable development goals, violence and women’s and children’s health. Bulletin of the World Health Organization. 2016;94(5): 396–397. pmid:27147771
  21. 21. McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. 2012;22(3): 276–282. pmid:23092060