Introduction

Concerns regarding surgical treatment quality for melanoma have been raised in Canada and other jurisdictions [1–3]. A population-based study in the Canadian province of Ontario suggests significant variations exist in melanoma surgical management, potentially leading to inadequate care for some, with greater likelihood of inadequate treatment with advanced age, female sex, and in certain jurisdictions [4]. At the same time, stage and outcome have been reported to vary by patient factors such as age, sex and area-level socioeconomic status, suggesting a need to optimize the quality of care between groups to ensure optimal outcomes [2, 3].

Surgery and pathology quality improvement is essential. High quality surgery and pathology reporting are necessary to guide stage-based treatment decisions for optimal melanoma care, including choice of adjuvant systemic therapy that can improve cancer outcomes, including survival [5–8]. Improving the quality of care can also reduce waste, and improve the cost-effectiveness of care for melanoma [9]. Notably, formally developed quality indictors validated through expert consensus exist to assess melanoma care quality, though they have not been explored as broadly as for other cancers [10].

Continuously measuring and reporting on quality indicators for melanoma and other cancers is crucial, as it may contribute to reducing inappropriate practice variation, and improving outcomes [11]. However, collecting and compiling the information required to do so can be resource intensive, time consuming and costly. We thus set out to explore the utility and feasibility of linked routinely collected health data to capture key elements of the quality of surgical management and pathologic reporting in a single payer, universal health setting. To do so, we set out to describe melanoma quality of care in Ontario, using a well-characterized population-based sample with detailed pathology report information [3]. Linked administrative data could provide a means to efficiently and continuously measure variation in quality of care, to improve the quality of care for those diagnosed with melanoma.

Methods

This pilot study utilized a previously developed retrospective population-based cohort. A 65% random sample of all invasive melanoma cases diagnosed in Ontario between January 1, 2007 and December 31, 2012 that were recorded in the Ontario Cancer Registry (OCR) was used for this study. The random sample was a convenience sample based on power requirements for the parent study investigating melanoma treatment outcomes according to stage in Ontario (CIHR MOP 137022) [3]. This source was chosen for the pilot study because completely abstracted population-based pathology data for later years was not available, and we wished to assess feasibility using the most comprehensive population-level data currently available. The OCR is administered by Ontario Health Cancer Care Ontario (CCO), the provincial cancer agency associated with Ontario’s single payer universal health system. The random sample thus provides insights into the management of the complete population of people diagnosed with melanoma over the study period. This study was approved by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (EPID-425-13). This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.

In the case of multiple primaries, details from the earliest melanoma were used. Patients whose first diagnosis was pure in situ disease were excluded based on concerns that OCR in situ data was incomplete. Patients without a pathology report from CCO and with core/FNA only biopsies were excluded. Out-of-province residents treated in Ontario and Ontario residents without continuous provincial medical coverage in a 5-year look back from diagnosis were excluded as were those under 20 years of age. OCR data was utilized to identify cases of melanoma. OCR demonstrates a high level of population coverage, including for melanoma [12, 13]. Available pathology reports for all patients were provided from CCO and abstracted according to a standardized algorithm developed at the Queen’s University Division of Cancer Care and Epidemiology, and deterministically linked to each patient’s OCR record according to their group ID, a unique identifier. Reliability testing indicated 97% complete agreement between all three abstractors and a clinician experienced in melanoma for primary variables, including stage-defining items. M-category data was supplemented by stage information provided by regional cancer centers.

Rurality was measured via the Rurality Index for Ontario (RIO). The RIO score is based on a 0–100 scale, with higher scores indicating a greater degree of rurality [14]. Comorbidity was measured using the Elixhauser comorbidity index with 5-year lookback from diagnosis, based on Canadian Institute for Health Information Discharge Abstract Database and Same Day Surgery data [15].

To pilot the use of linked administrative data to measure quality indicators, we sought out a comprehensive source that provided a representative selection of quality metrics deemed important by many clinicians for measuring melanoma surgery and pathology quality. We also determined a-priori that these metrics should be contemporaneous to the administrative data utilized for this pilot project. Sources were sought out based on literature searches and expert knowledge of the study team. One key study was identified, that identified quality indicators via a comprehensive, iterative consensus-based process involving thirteen experts, based on review of the literature and consensus guidelines [10]. Face, construct and predictive validity were considered in their process. These quality indicators were supplemented by non-overlapping quality metrics identified by review of consensus-based contemporaneous provincial guidelines for the province of Ontario [16].

Quality indicators fell into three broad categories: (1) Pathology reporting of disease, (2) Management of the sentinel node, and (3) Management of the nodal basin. These quality indicators were reviewed and evaluated for their feasibility of assessment with the administrative data linked to the available pathology report data. Features of the administrative data required for measuring the quality indicator, and a description of the type of data affecting measurement feasibility was recorded. Where relevant, the type of missing data affecting the feasibility of measurement with administrative data was catalogued (clinical, pathologic or structural, where ‘structural’ refers to details of the organization or structure of the health care system).

Results

Study cohort

We identified a cohort of 7,654 adult patients with invasive melanoma, provincial health coverage and linked pathology records from 8,877 linkable patients with 9,462 pathology records (S1 Appendix, Table 1). The most common reasons for exclusion were missing pathology records at first diagnosis (n = 438), in situ melanoma as first diagnosis (n = 386) and lapses in provincial health coverage (n = 289). The latest primary site pathology reports for each patient were reviewed. Of these, 5,146 (67.2%) described an excision, 1,649 (21.5%) described a biopsy, 77 (1.0%) described an amputation. The procedure type was missing or not reported for 782 (10.2%) of cases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient and disease characteristics of first primary melanoma (N = 7,654).

https://doi.org/10.1371/journal.pone.0263713.t001

Quality indicator measurement feasibility with linked administrative data

There were 25 non-overlapping consensus-based quality indicators identified (Table 2). Ten of 25 (40%) candidate quality indicators were feasible to be collected with the administrative data. Many indicators (8/25) could not be measured due to unavailable clinical information in administrative data sources (e.g. width of clinical margins) (Table 2). Lack of sufficient pathology information (6/25) or health structure information (1/25) were less common reasons for unfeasibility.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Established surgical and pathology melanoma quality indicators and feasibility of collection with administrative data.

https://doi.org/10.1371/journal.pone.0263713.t002

Quality indicator measurement

The quality indicators that were feasible for measurement with administrative data linkage, and the relevant patient numbers meeting them are provided in Table 3. Additional details are provided in the subsequent paragraphs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Achievement of quality indicators (QIs) first primary melanoma (N = 7,654).

https://doi.org/10.1371/journal.pone.0263713.t003

Pathology reporting

QI 1: Completeness of primary site pathology report.

3,802 (49.7%) patients reported variables required by the quality indicator (Table 3), with 3,933 (51.4%) complete if Clark level was excluded, and 4,271 (55.8%) if Clark level and mitosis were excluded. High levels of reporting were noted for Breslow thickness and Clark level of invasion at 99.1% and 94.9%, respectively. Other variables are summarized in Table 4 and range down to 2.6% for in transit metastasis documentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Reporting of pathologic variables of first primary melanoma (N = 7,654).

https://doi.org/10.1371/journal.pone.0263713.t004

Discussion

In this population-based sample, we evaluated the feasibility of using linked administrative data sources to measure the quality of melanoma surgery and pathology reporting on the basis of consensus-derived quality indicators and provincial evidence-based guidelines. We found that a minority of quality indicators could be measured with the available linked data sources (10/25). Missing clinical information was the most common reason quality indicators could not be measured (8/25). In our pilot cohort, we identified variation in surgery and pathology reporting according to quality indicators relevant to practice at that time (2007–2012). These findings provide a strong impetus to develop ways to continuously measure and improve a comprehensive suite of quality indicators for melanoma in our region, and beyond.

We note that since the time of our pilot cohort, some metrics may have improved due to adoption of synoptic reporting in the province [17, 18]. Ulceration, when present, draws into question the depth of the primary and could not be determined in 517 (6.8%) of patients. Only 4,987 (65.2%) patients reported any information on the presence of satellites or microsatellites. Re-evaluation of these indicators will provide insights into the effectiveness of synoptic reporting in ensuring the quality and completeness of pathology reporting in the setting of a universal single-payer health system.

For other metrics, it is not as certain that improvements have been made since the time of our pilot cohort. For example, SLNB and the beneficial role of lymphoscintigraphy for melanoma have been well documented in the literature for almost 20 years [19–22]. The final analysis of the first Multicenter Selective Lymphadenectomy Trial (MSLT-1) only became available in 2014 confirming the benefits of SLNB; however, reports on the benefits of SLNB in this trial span the era under study [23–25]. By considering whether the surgeon seen in consultation had experience performing SLNB based on previous billing code use, we saw that only 2,726 (66.0%) of Stage IB-II and T1b-T4a patients eligible for a sentinel node biopsy were seen by such a surgeon, and less for high-risk <1mm patients (647). Nevertheless, we hypothesize that SLNB referral has increased due to availability of new effective adjuvant targeted and immune-based therapies where pathologically confirmed lymph node involvement is required for funding. The additional impact of MSLT-II and the German Dermatologic Cooperative Oncology Group (DeCOG) SLNB trial on SLNB rates are less clear. These studies suggest that completion dissection increases regional disease control and provide prognostic information for SLNB positive patients, without increasing melanoma-specific survival (MSLT-II) or distant metastasis-free survival (DeCOG) [26, 27]. Whether this evidence supporting SLNB alone without completion dissection for SLNB positive patients would increase use of SLNB by possibly removing hesitance to use SLNB is unclear. This requires further confirmatory study.

Stage-based treatment cannot be executed without an adequate summary of the extent of disease. Of all 4,132 potential SLNB patients (Stage IB-II and T1b-T4a), only 638 underwent a SLNB, a procedure known to be a powerful prognosticator. Patient choice may play a role, but the possibility of inadequate access is concerning. We observed that patients seen by a SLNB surgeon, and those subsequently receiving SLNB tended to be younger, and have lower comorbidity in univariate analysis. Whether these differences reflect patient-centered decision making cannot be determined based on the available data. We also observed that eligible patients living in rural areas were less likely to see a SLNB surgeon. Again it is unclear whether this reflects patient choice, poor geographic access or both.

Furthermore, the number of lymph nodes dissected for non-SLNB procedures failed to meet established quality metrics for between 18% (inguinal basin) to 33% (cervical basin) of cases. The question of adequacy of resection is not novel to this study [21–24]. Spillane et al in a retrospective review of the number of nodes taken at the Melanoma Institute Australia (MIA) demonstrated that over 90% of cases had greater or equal to 7 and 10 nodes in the specimen for the groin and axilla respectively. For the neck they subdivided results on the basis of levels taken: for those cases where 3 or less levels were excised greater than 90% of patients had 6 or more nodes taken and for neck dissections of four or more levels greater than 90% of patients has 20 or more nodes taken [22]. There were similar findings in Rossi et al’s large patient series from nine Italian Melanoma Intergroup institutions [28]. If we consider the MIA as a center of expertise and directly compare provincial data on the number of nodes resected, outcomes fall below the proposed 90% threshold number of nodes taken for inguinal and axilla basins. Unlike the MIA data, our cervical basin data is not stratified by levels taken, though only 67% had at least 15 nodes resected. We note that others have found similar under-performance for LND/CLND benchmarks in large population data. For example Bilimoria et al found achievement of the LND/CLND consensus-based benchmarks were low in the United States National Cancer Database [10]. Adequacy of surgical management of these basins bears further scrutiny for these populations.

There is no prospective data for melanoma linking these numbers to recurrence-free survival, but the numbers themselves can be used as a proxy for thoroughness of treatment and pathologic evaluation. Regular evaluation of lymph node dissection quality remains important in the MSLT-II era. This study showed no melanoma-specific-survival advantage for patients who went on to regional node resection following positive sentinel node biopsy compared to nodal observation with ultrasonography [26]. Importantly, SLNB positive patients now managed with ultrasound follow-up rather than CLND will commonly develop lymph node recurrence (26% by five years in MSLT-II), and for these patients, the quality of salvage lymph node dissection bears importance. Measurement of appropriate imaging follow-up for SLNB positive patients without CLND will also be an important quality indicator going forward.

To support a program of quality improvement for melanoma pathology and surgery, continuous measurement and reporting of a comprehensive suite of melanoma quality indictors is required. Though the available quality metrics showed important gaps in quality of care requiring attention, we importantly found that only a minority of quality metrics could be measured with meticulously collected linked administrative data sources. How can this be addressed? With the rise of electronic synoptic reporting, some information sources are being more efficiently and completely collected in our jurisdiction and many others. However, other important variables require attention. For example, clinical margin width is only collected systematically in the individual patient chart. Manual abstraction for all patients is time- and resource-intensive. Other alternatives are routine sampling of a small, representative subset of patients from each institution on an ongoing basis. A sampling approach has been used in radiation oncology quality measurement, though it still required a substantial amount of time and coordination [29]. Artificial intelligence (AI) with natural language processing may provide a solution for routinely collected pathology and surgery variables [30]. This requires a comprehensive data infrastructure and human capacity to ensure the training of algorithms to ensure complete and accurate data capture.

There are limitations to our study. Some metrics that are relevant to the clinical community may not have been included in the recommended consensus-based quality indicators we evaluated. For example, the false negative rate after SLNB or SLNB positivity rate. We note that the former requires multiple data sources and extended follow-up, though is technically feasible with appropriate data linkage. The latter is also feasible, without extended follow-up. We also note that recommendations on the optimal number of lymph nodes to be dissected varies to some degree between studies, though the purpose of our present investigation was to explore feasibility and utility of using data linkage for quality measurement, rather than to determine the optimal metric [28, 31].

The pilot cohort under study provided a unique opportunity to study melanoma quality of care at the population level, as we had access to detailed pathologic data linked to administrative sources on a random sample of all cases in the Canadian province of Ontario. However, the sample was historic, covering 2007 to 2012. This was the pre-MSLT-II era, and the use of completion dissection has now changed. However, as noted, it is possible that more patients will require salvage dissection due to nodal recurrence due to use of nodal observation rather than completion dissection, emphasizing the need to ensure the quality of nodal evaluation and follow-up for melanoma. Though historic, our pilot sample provides far more comprehensive pathology data than available in a typical cancer registry. Nonetheless, only 10/25 (40%) of quality indicators were feasibly measured with the linked administrative data. Six of the indicators not measurable with administrative data related to margins as we did not have access to surgical reports documenting the clinical excision margin (5/6 metrics), or pathology report information for excision samples with no residual melanoma that would have confirmed negative margins in many cases (1/6 metrics). Our findings emphasize the need to improve processes for cancer patient data reporting, abstraction and linkage, to ensure that a comprehensive suite of quality indicators can be collected on a continual basis for patients with melanoma, and other cancers.

Conclusion

In this-population-based pilot study of a universal health care system, we found that there was limited feasibility of using linked administrative data sources for measuring quality indicators. For the 40% of indicators that were feasibly measured, evidence suggests opportunities for quality improvement in surgical and pathological quality of care. Completeness of pathologic reporting was sub-optimal in this historic cohort and this may be improved by synoptic reporting. Consultations with surgeons offering SLNB occurred in only two thirds of eligible cases, and far less received SLNB, limiting staging and prognostic information relevant for adjuvant therapy. Nearly one quarter of patients had less than the optimal number of nodes in their pathology report in non-SLNB dissection. We recommend development of robust data platforms to support continuous re-evaluation of melanoma quality indicators in a contemporary sample of patients, with the goal of optimizing quality of care for melanoma patients on an ongoing basis.

Supporting information

Acknowledgments

Parts of this material are based on data and/or information compiled and provided by CIHI. However, the analyses, conclusions, opinions and statements expressed in the material are those of the author(s), and not necessarily those of CIHI. Parts of this material are based on data and information provided by Ontario Health Cancer Care Ontario (CCO). The opinions, results, view, and conclusions reported in this paper are those of the authors and do not necessarily reflect those of CCO. No endorsement by CCO is intended or should be inferred. These data were linked using unique encoded identifiers and analyzed at ICES.