Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A population-based study of administrative data linkage to measure melanoma surgical and pathology quality

  • Douglas R. McKay,

    Roles Conceptualization, Funding acquisition, Methodology, Resources, Validation, Writing – review & editing

    Affiliation Division of Plastic Surgery, Department of Surgery, Queen’s University, Kingston, Ontario, Canada

  • Paul Nguyen,

    Roles Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Writing – review & editing

    Affiliation ICES at Queen’s University, Kingston, Ontario, Canada

  • Ami Wang,

    Roles Conceptualization, Funding acquisition, Methodology, Validation, Writing – original draft, Writing – review & editing

    Affiliation Department of Pathology and Molecular Medicine, Queen’s University, Kingston, Ontario, Canada

  • Timothy P. Hanna

    Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing

    Affiliations ICES at Queen’s University, Kingston, Ontario, Canada, Department of Oncology, Queen’s University, Kingston, Ontario, Canada, Division of Cancer Care and Epidemiology, Cancer Research Institute at Queen’s University, Kingston, Ontario, Canada



Continuous quality improvement is important for cancer systems. However, collecting and compiling quality indicator data can be time-consuming and resource-intensive. Here we explore the utility and feasibility of linked routinely collected health data to capture key elements of quality of care for melanoma in a single-payer, universal health care setting.


This pilot study utilized a retrospective population-based cohort from a previously developed linked administrative data set, with a 65% random sample of all invasive cutaneous melanoma cases diagnosed 2007–2012 in the province of Ontario. Data from the Ontario Cancer Registry was utilized, supplemented with linked pathology report data from Cancer Care Ontario, and other linked administrative data describing health care utilization. Quality indicators identified through provincial guidelines and international consensus were evaluated for potential collection with administrative data and measured where possible.


A total of 7,654 cases of melanoma were evaluated. Ten of 25 (40%) candidate quality indicators were feasible to be collected with the available administrative data. Many indicators (8/25) could not be measured due to unavailable clinical information (e.g. width of clinical margins). Insufficient pathology information (6/25) or health structure information (1/25) were less common reasons. Reporting of recommended variables in pathology reports varied from 65.2% (satellitosis) to 99.6% (body location). For stage IB-II or T1b-T4a melanoma patients where SLNB should be discussed, approximately two-thirds met with a surgeon experienced in SLNB. Of patients undergoing full lymph node dissection, 76.2% had adequate evaluation of the basin.


We found that use of linked administrative data sources is feasible for measurement of melanoma quality in some cases. In those cases, findings suggest opportunities for quality improvement. Consultation with surgeons offering SLNB was limited, and pathology report completeness was sub-optimal, but was prior to routine synoptic reporting. However, to measure more quality indicators, text-based data sources will require alternative approaches to manual collection such as natural language processing or standardized collection. We recommend development of robust data platforms to support continuous re-evaluation of melanoma quality indicators, with the goal of optimizing quality of care for melanoma patients on an ongoing basis.


Concerns regarding surgical treatment quality for melanoma have been raised in Canada and other jurisdictions [13]. A population-based study in the Canadian province of Ontario suggests significant variations exist in melanoma surgical management, potentially leading to inadequate care for some, with greater likelihood of inadequate treatment with advanced age, female sex, and in certain jurisdictions [4]. At the same time, stage and outcome have been reported to vary by patient factors such as age, sex and area-level socioeconomic status, suggesting a need to optimize the quality of care between groups to ensure optimal outcomes [2, 3].

Surgery and pathology quality improvement is essential. High quality surgery and pathology reporting are necessary to guide stage-based treatment decisions for optimal melanoma care, including choice of adjuvant systemic therapy that can improve cancer outcomes, including survival [58]. Improving the quality of care can also reduce waste, and improve the cost-effectiveness of care for melanoma [9]. Notably, formally developed quality indictors validated through expert consensus exist to assess melanoma care quality, though they have not been explored as broadly as for other cancers [10].

Continuously measuring and reporting on quality indicators for melanoma and other cancers is crucial, as it may contribute to reducing inappropriate practice variation, and improving outcomes [11]. However, collecting and compiling the information required to do so can be resource intensive, time consuming and costly. We thus set out to explore the utility and feasibility of linked routinely collected health data to capture key elements of the quality of surgical management and pathologic reporting in a single payer, universal health setting. To do so, we set out to describe melanoma quality of care in Ontario, using a well-characterized population-based sample with detailed pathology report information [3]. Linked administrative data could provide a means to efficiently and continuously measure variation in quality of care, to improve the quality of care for those diagnosed with melanoma.


This pilot study utilized a previously developed retrospective population-based cohort. A 65% random sample of all invasive melanoma cases diagnosed in Ontario between January 1, 2007 and December 31, 2012 that were recorded in the Ontario Cancer Registry (OCR) was used for this study. The random sample was a convenience sample based on power requirements for the parent study investigating melanoma treatment outcomes according to stage in Ontario (CIHR MOP 137022) [3]. This source was chosen for the pilot study because completely abstracted population-based pathology data for later years was not available, and we wished to assess feasibility using the most comprehensive population-level data currently available. The OCR is administered by Ontario Health Cancer Care Ontario (CCO), the provincial cancer agency associated with Ontario’s single payer universal health system. The random sample thus provides insights into the management of the complete population of people diagnosed with melanoma over the study period. This study was approved by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (EPID-425-13). This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.

In the case of multiple primaries, details from the earliest melanoma were used. Patients whose first diagnosis was pure in situ disease were excluded based on concerns that OCR in situ data was incomplete. Patients without a pathology report from CCO and with core/FNA only biopsies were excluded. Out-of-province residents treated in Ontario and Ontario residents without continuous provincial medical coverage in a 5-year look back from diagnosis were excluded as were those under 20 years of age. OCR data was utilized to identify cases of melanoma. OCR demonstrates a high level of population coverage, including for melanoma [12, 13]. Available pathology reports for all patients were provided from CCO and abstracted according to a standardized algorithm developed at the Queen’s University Division of Cancer Care and Epidemiology, and deterministically linked to each patient’s OCR record according to their group ID, a unique identifier. Reliability testing indicated 97% complete agreement between all three abstractors and a clinician experienced in melanoma for primary variables, including stage-defining items. M-category data was supplemented by stage information provided by regional cancer centers.

Rurality was measured via the Rurality Index for Ontario (RIO). The RIO score is based on a 0–100 scale, with higher scores indicating a greater degree of rurality [14]. Comorbidity was measured using the Elixhauser comorbidity index with 5-year lookback from diagnosis, based on Canadian Institute for Health Information Discharge Abstract Database and Same Day Surgery data [15].

To pilot the use of linked administrative data to measure quality indicators, we sought out a comprehensive source that provided a representative selection of quality metrics deemed important by many clinicians for measuring melanoma surgery and pathology quality. We also determined a-priori that these metrics should be contemporaneous to the administrative data utilized for this pilot project. Sources were sought out based on literature searches and expert knowledge of the study team. One key study was identified, that identified quality indicators via a comprehensive, iterative consensus-based process involving thirteen experts, based on review of the literature and consensus guidelines [10]. Face, construct and predictive validity were considered in their process. These quality indicators were supplemented by non-overlapping quality metrics identified by review of consensus-based contemporaneous provincial guidelines for the province of Ontario [16].

Quality indicators fell into three broad categories: (1) Pathology reporting of disease, (2) Management of the sentinel node, and (3) Management of the nodal basin. These quality indicators were reviewed and evaluated for their feasibility of assessment with the administrative data linked to the available pathology report data. Features of the administrative data required for measuring the quality indicator, and a description of the type of data affecting measurement feasibility was recorded. Where relevant, the type of missing data affecting the feasibility of measurement with administrative data was catalogued (clinical, pathologic or structural, where ‘structural’ refers to details of the organization or structure of the health care system).


Study cohort

We identified a cohort of 7,654 adult patients with invasive melanoma, provincial health coverage and linked pathology records from 8,877 linkable patients with 9,462 pathology records (S1 Appendix, Table 1). The most common reasons for exclusion were missing pathology records at first diagnosis (n = 438), in situ melanoma as first diagnosis (n = 386) and lapses in provincial health coverage (n = 289). The latest primary site pathology reports for each patient were reviewed. Of these, 5,146 (67.2%) described an excision, 1,649 (21.5%) described a biopsy, 77 (1.0%) described an amputation. The procedure type was missing or not reported for 782 (10.2%) of cases.

Table 1. Patient and disease characteristics of first primary melanoma (N = 7,654).

Quality indicator measurement feasibility with linked administrative data

There were 25 non-overlapping consensus-based quality indicators identified (Table 2). Ten of 25 (40%) candidate quality indicators were feasible to be collected with the administrative data. Many indicators (8/25) could not be measured due to unavailable clinical information in administrative data sources (e.g. width of clinical margins) (Table 2). Lack of sufficient pathology information (6/25) or health structure information (1/25) were less common reasons for unfeasibility.

Table 2. Established surgical and pathology melanoma quality indicators and feasibility of collection with administrative data.

Quality indicator measurement

The quality indicators that were feasible for measurement with administrative data linkage, and the relevant patient numbers meeting them are provided in Table 3. Additional details are provided in the subsequent paragraphs.

Table 3. Achievement of quality indicators (QIs) first primary melanoma (N = 7,654).

Pathology reporting

QI 1: Completeness of primary site pathology report.

3,802 (49.7%) patients reported variables required by the quality indicator (Table 3), with 3,933 (51.4%) complete if Clark level was excluded, and 4,271 (55.8%) if Clark level and mitosis were excluded. High levels of reporting were noted for Breslow thickness and Clark level of invasion at 99.1% and 94.9%, respectively. Other variables are summarized in Table 4 and range down to 2.6% for in transit metastasis documentation.

Table 4. Reporting of pathologic variables of first primary melanoma (N = 7,654).

Management of the sentinel node quality indicators

Amongst the 7,654 patients, 1,012 (13.2%) patients had sentinel lymph node biopsy (SLNB) procedures, within 6 months of diagnosis.

QI 2: Appropriate certification of surgeons performing SLNB or LND.

There were 1,189 patients undergoing SLNB or LND in the study cohort. All operating surgeons were confirmed to be certified by the College of Physicians and Surgeons of Ontario.

QI 3: Referral for SLNB discussion Stage IB-II.

Stage IB-II melanoma patients were included in this analysis as were T1b-T4a patients. All patients with documented distant metastases (i.e., M1) were excluded as were those with dermal deposits as the appropriate role of SLNB is not well defined. Amongst the first primary of the 7,654 patients, 4,132 (54.0%) satisfied these conditions. Of these, 2,726 (66.0%) consulted with a surgeon who had previously performed SLNB or lymphoscintigraphy plus another lymph node procedure on the basis of their previous OHIP billings (S2 Appendix) within 6 months of their diagnostic date. Compared to those who consulted with a SLNB surgeon, the 1,406 (34.0%) who had not consulted were older (median [IQR]: 67 [54–79] vs. 62 [51–74], p<0.001), more often resided in rural areas (based on RIO scores) (13.8% vs. 8.8%, p<0.001) and had a history of higher comorbidities (≥2 comorbidities: 13.8% vs. 10.5%, p = 0.002).

Of the 2,726 who saw these surgeons, 638 (23.4%) had a SLNB with 270 having positive nodes. Compared those who had SLNB, the 2,088 (76.6%) patients who saw these surgeons and did not have SLNB were older (median [IQR]: 63 [51–75] vs. 61 [50–72], p<0.001) with no significant differences for residency in rural areas (8.3% vs. 10.3%, p = 0.273) and history of greater comorbidities (≥2 comorbidities: 11.1% vs. 8.6%, p = 0.078).

QI 4: Referral for SLNB discussion for high risk <1.0 mm.

1,548 (20.2%) patients with melanomas <1.0 mm in thickness had high risk features: 498 (32.2%) were young (≤39 years), 1,157 (74.7%) with ≥1 mm2 mitotic rate, 77 (5.0%) with ulceration, and 23 (1.5%) with a positive deep margin from a shave biopsy. 647 (41.8%) consulted with a SLNB surgeon within 6 months of diagnosis. 61 of 647 had a SLNB procedure with 25 having positive nodes found.

QI 5: Use of lymphoscintigraphy.

Of the 1,012 patients of any stage who had a SLNB, 473 (46.7%) had at least one positive node and 103 (10.2%) had more than one nodal drainage basin dissected. Lymphoscintigraphy was used in 909 (89.8%). Lymphoscintigraphy procedures have increased compared to the earliest years in the cohort (82.4% 2007, up to 90.3% in 2012).

QI 6a: Lymph node reporting for SLNB.

Almost all of the 1,012 patients had underwent SLNB as their first procedure (>1006). Of these patients, 473 (47%) had at least 1 positive node found, with the majority having one (343) or two (101) positive nodes. One-hundred percent reported on the number of lymph nodes examined and the number of lymph nodes found to contain metastases.

Management of the lymph node basin quality indicators

QI 6b: Lymph node reporting for non-sentinel lymph node (NSLN) dissection.

Of the 190 patients whose first procedure within 6 months of diagnosis was NSLN, 144 (78.7%) had at least 1 positive node found. The majority (124) of these patients had 1–6 positive nodes extracted from NSLN. One-hundred percent (190) reported on the number of lymph nodes examined and the number of lymph nodes found to contain metastases.

QI 7: Cervical LND or CLND node count.

69 patients underwent NSLN or regional nodal recurrence resection of the neck. Of these, 45 (65.2%) had only a NSLN clearance of the neck and 24 (34.8%) had a regional nodal recurrence resected (plus NSLN clearance in <6 cases). 46 (66.7%) of 69 patients undergoing neck node surgery had at least 15 regional lymph nodes removed and assessed. Of those 24 patients presenting with regional neck nodal recurrence, >18 had at least 15 lymph nodes assessed.

QI 8: Axillary LND or CLND node count.

130 patients had procedures in the axilla. This group includes NSLN and the resection of regional nodal recurrences. Of the 130 patients, 101 (77.7%) had at least 10 regional lymph nodes removed and assessed. 84 only underwent NSLN and 46 underwent a resection for regional recurrence (plus NSLN clearance in <6 cases). For the 84 NSLN patients, 71 (84.5%) had at least 10 regional lymph nodes assessed; in the case of regional recurrence, 30 (65.2%) had at least 10 nodes assessed.

QI 9: Inguinal LND or CLND node count.

74 patients had either only NSLN (46) or recurrence dissections (28, with <6 also having NSLN clearance). Of the 74 patients, 61 (82.4%) had at least 5 regional lymph nodes assessed. For the 46 NSLN patients, >40 had at least 5 nodes assessed.

QI 10: Radiologic staging for clinical inguinal/femoral adenopathy.

53 patients had a NSLN procedure in the absence of a prior SLNB or regional nodal resection at the inguinal site. Abdominal and pelvic CT scans were identified using OHIP billing codes (S2 Appendix); 38 (71.7%) patients had a pelvic CT scan within the 6 months prior to the nodal surgery date.


In this population-based sample, we evaluated the feasibility of using linked administrative data sources to measure the quality of melanoma surgery and pathology reporting on the basis of consensus-derived quality indicators and provincial evidence-based guidelines. We found that a minority of quality indicators could be measured with the available linked data sources (10/25). Missing clinical information was the most common reason quality indicators could not be measured (8/25). In our pilot cohort, we identified variation in surgery and pathology reporting according to quality indicators relevant to practice at that time (2007–2012). These findings provide a strong impetus to develop ways to continuously measure and improve a comprehensive suite of quality indicators for melanoma in our region, and beyond.

We note that since the time of our pilot cohort, some metrics may have improved due to adoption of synoptic reporting in the province [17, 18]. Ulceration, when present, draws into question the depth of the primary and could not be determined in 517 (6.8%) of patients. Only 4,987 (65.2%) patients reported any information on the presence of satellites or microsatellites. Re-evaluation of these indicators will provide insights into the effectiveness of synoptic reporting in ensuring the quality and completeness of pathology reporting in the setting of a universal single-payer health system.

For other metrics, it is not as certain that improvements have been made since the time of our pilot cohort. For example, SLNB and the beneficial role of lymphoscintigraphy for melanoma have been well documented in the literature for almost 20 years [1922]. The final analysis of the first Multicenter Selective Lymphadenectomy Trial (MSLT-1) only became available in 2014 confirming the benefits of SLNB; however, reports on the benefits of SLNB in this trial span the era under study [2325]. By considering whether the surgeon seen in consultation had experience performing SLNB based on previous billing code use, we saw that only 2,726 (66.0%) of Stage IB-II and T1b-T4a patients eligible for a sentinel node biopsy were seen by such a surgeon, and less for high-risk <1mm patients (647). Nevertheless, we hypothesize that SLNB referral has increased due to availability of new effective adjuvant targeted and immune-based therapies where pathologically confirmed lymph node involvement is required for funding. The additional impact of MSLT-II and the German Dermatologic Cooperative Oncology Group (DeCOG) SLNB trial on SLNB rates are less clear. These studies suggest that completion dissection increases regional disease control and provide prognostic information for SLNB positive patients, without increasing melanoma-specific survival (MSLT-II) or distant metastasis-free survival (DeCOG) [26, 27]. Whether this evidence supporting SLNB alone without completion dissection for SLNB positive patients would increase use of SLNB by possibly removing hesitance to use SLNB is unclear. This requires further confirmatory study.

Stage-based treatment cannot be executed without an adequate summary of the extent of disease. Of all 4,132 potential SLNB patients (Stage IB-II and T1b-T4a), only 638 underwent a SLNB, a procedure known to be a powerful prognosticator. Patient choice may play a role, but the possibility of inadequate access is concerning. We observed that patients seen by a SLNB surgeon, and those subsequently receiving SLNB tended to be younger, and have lower comorbidity in univariate analysis. Whether these differences reflect patient-centered decision making cannot be determined based on the available data. We also observed that eligible patients living in rural areas were less likely to see a SLNB surgeon. Again it is unclear whether this reflects patient choice, poor geographic access or both.

Furthermore, the number of lymph nodes dissected for non-SLNB procedures failed to meet established quality metrics for between 18% (inguinal basin) to 33% (cervical basin) of cases. The question of adequacy of resection is not novel to this study [2124]. Spillane et al in a retrospective review of the number of nodes taken at the Melanoma Institute Australia (MIA) demonstrated that over 90% of cases had greater or equal to 7 and 10 nodes in the specimen for the groin and axilla respectively. For the neck they subdivided results on the basis of levels taken: for those cases where 3 or less levels were excised greater than 90% of patients had 6 or more nodes taken and for neck dissections of four or more levels greater than 90% of patients has 20 or more nodes taken [22]. There were similar findings in Rossi et al’s large patient series from nine Italian Melanoma Intergroup institutions [28]. If we consider the MIA as a center of expertise and directly compare provincial data on the number of nodes resected, outcomes fall below the proposed 90% threshold number of nodes taken for inguinal and axilla basins. Unlike the MIA data, our cervical basin data is not stratified by levels taken, though only 67% had at least 15 nodes resected. We note that others have found similar under-performance for LND/CLND benchmarks in large population data. For example Bilimoria et al found achievement of the LND/CLND consensus-based benchmarks were low in the United States National Cancer Database [10]. Adequacy of surgical management of these basins bears further scrutiny for these populations.

There is no prospective data for melanoma linking these numbers to recurrence-free survival, but the numbers themselves can be used as a proxy for thoroughness of treatment and pathologic evaluation. Regular evaluation of lymph node dissection quality remains important in the MSLT-II era. This study showed no melanoma-specific-survival advantage for patients who went on to regional node resection following positive sentinel node biopsy compared to nodal observation with ultrasonography [26]. Importantly, SLNB positive patients now managed with ultrasound follow-up rather than CLND will commonly develop lymph node recurrence (26% by five years in MSLT-II), and for these patients, the quality of salvage lymph node dissection bears importance. Measurement of appropriate imaging follow-up for SLNB positive patients without CLND will also be an important quality indicator going forward.

To support a program of quality improvement for melanoma pathology and surgery, continuous measurement and reporting of a comprehensive suite of melanoma quality indictors is required. Though the available quality metrics showed important gaps in quality of care requiring attention, we importantly found that only a minority of quality metrics could be measured with meticulously collected linked administrative data sources. How can this be addressed? With the rise of electronic synoptic reporting, some information sources are being more efficiently and completely collected in our jurisdiction and many others. However, other important variables require attention. For example, clinical margin width is only collected systematically in the individual patient chart. Manual abstraction for all patients is time- and resource-intensive. Other alternatives are routine sampling of a small, representative subset of patients from each institution on an ongoing basis. A sampling approach has been used in radiation oncology quality measurement, though it still required a substantial amount of time and coordination [29]. Artificial intelligence (AI) with natural language processing may provide a solution for routinely collected pathology and surgery variables [30]. This requires a comprehensive data infrastructure and human capacity to ensure the training of algorithms to ensure complete and accurate data capture.

There are limitations to our study. Some metrics that are relevant to the clinical community may not have been included in the recommended consensus-based quality indicators we evaluated. For example, the false negative rate after SLNB or SLNB positivity rate. We note that the former requires multiple data sources and extended follow-up, though is technically feasible with appropriate data linkage. The latter is also feasible, without extended follow-up. We also note that recommendations on the optimal number of lymph nodes to be dissected varies to some degree between studies, though the purpose of our present investigation was to explore feasibility and utility of using data linkage for quality measurement, rather than to determine the optimal metric [28, 31].

The pilot cohort under study provided a unique opportunity to study melanoma quality of care at the population level, as we had access to detailed pathologic data linked to administrative sources on a random sample of all cases in the Canadian province of Ontario. However, the sample was historic, covering 2007 to 2012. This was the pre-MSLT-II era, and the use of completion dissection has now changed. However, as noted, it is possible that more patients will require salvage dissection due to nodal recurrence due to use of nodal observation rather than completion dissection, emphasizing the need to ensure the quality of nodal evaluation and follow-up for melanoma. Though historic, our pilot sample provides far more comprehensive pathology data than available in a typical cancer registry. Nonetheless, only 10/25 (40%) of quality indicators were feasibly measured with the linked administrative data. Six of the indicators not measurable with administrative data related to margins as we did not have access to surgical reports documenting the clinical excision margin (5/6 metrics), or pathology report information for excision samples with no residual melanoma that would have confirmed negative margins in many cases (1/6 metrics). Our findings emphasize the need to improve processes for cancer patient data reporting, abstraction and linkage, to ensure that a comprehensive suite of quality indicators can be collected on a continual basis for patients with melanoma, and other cancers.


In this-population-based pilot study of a universal health care system, we found that there was limited feasibility of using linked administrative data sources for measuring quality indicators. For the 40% of indicators that were feasibly measured, evidence suggests opportunities for quality improvement in surgical and pathological quality of care. Completeness of pathologic reporting was sub-optimal in this historic cohort and this may be improved by synoptic reporting. Consultations with surgeons offering SLNB occurred in only two thirds of eligible cases, and far less received SLNB, limiting staging and prognostic information relevant for adjuvant therapy. Nearly one quarter of patients had less than the optimal number of nodes in their pathology report in non-SLNB dissection. We recommend development of robust data platforms to support continuous re-evaluation of melanoma quality indicators in a contemporary sample of patients, with the goal of optimizing quality of care for melanoma patients on an ongoing basis.

Supporting information

S1 Appendix. Identification of invasive cutaneous melanoma patients and their pathology records in Ontario from January 1, 2007 to December 31, 2012.


S2 Appendix. Ontario Health Insurance Plan (OHIP) billing codes.



Parts of this material are based on data and/or information compiled and provided by CIHI. However, the analyses, conclusions, opinions and statements expressed in the material are those of the author(s), and not necessarily those of CIHI. Parts of this material are based on data and information provided by Ontario Health Cancer Care Ontario (CCO). The opinions, results, view, and conclusions reported in this paper are those of the authors and do not necessarily reflect those of CCO. No endorsement by CCO is intended or should be inferred. These data were linked using unique encoded identifiers and analyzed at ICES.


  1. 1. Clegg LX, Reichman ME, Miller BA, Hankey BF, Singh GK, et al. (2009) Impact of socioeconomic status on cancer incidence and stage at diagnosis: selected findings from the surveillance, epidemiology, and end results: National Longitudinal Mortality Study. Cancer Causes Control 20: 417–435. pmid:19002764
  2. 2. Nennecke A, Geiss K, Hentschel S, Vettorazzi E, Jansen L, et al. (2014) Survival of cancer patients in urban and rural areas of Germany—a comparison. Cancer Epidemiol 38: 259–265. pmid:24680643
  3. 3. Mavor ME, Richardson H, Miao Q, Asai Y, Hanna TP (2018) Disparities in diagnosis of advanced melanoma: a population-based cohort study. CMAJ Open 6: E502–E512. pmid:30381323
  4. 4. Look Hong NJ, Cheng SY, Baxter NN, Wright FC (2018) Melanoma patterns of care in Ontario: A call for a strategic alignment of multidisciplinary care. J Surg Oncol 117: 597–617. pmid:29228470
  5. 5. Pasquali S, Sommariva A, Spillane AJ, Bilimoria KY, Rossi CR (2017) Measuring the quality of melanoma surgery—Highlighting issues with standardization and quality assurance of care in surgical oncology. Eur J Surg Oncol 43: 561–571. pmid:27422583
  6. 6. Eggermont AMM, Chiarion-Sileni V, Grob JJ, Dummer R, Wolchok JD, et al. (2019) Adjuvant ipilimumab versus placebo after complete resection of stage III melanoma: long-term follow-up results of the European Organisation for Research and Treatment of Cancer 18071 double-blind phase 3 randomised trial. Eur J Cancer 119: 1–10. pmid:31400634
  7. 7. Weber J, Mandala M, Del Vecchio M, Gogas HJ, Arance AM, et al. (2017) Adjuvant Nivolumab versus Ipilimumab in Resected Stage III or IV Melanoma. N Engl J Med 377: 1824–1835. pmid:28891423
  8. 8. Long GV, Hauschild A, Santinami M, Atkinson V, Mandala M, et al. (2017) Adjuvant Dabrafenib plus Trametinib in Stage III BRAF-Mutated Melanoma. N Engl J Med 377: 1813–1823. pmid:28891408
  9. 9. Parvez E, Dumitra T, Panagiotoglou D, Meterissian SH, Dumitra S (2020) Cost-effectiveness of nodal observation versus completion lymphadenectomy in patients with melanoma and sentinel lymph node metastases. J Clin Onc 38: e22076.
  10. 10. Bilimoria KY, Raval MV, Bentrem DJ, Wayne JD, Balch CM, et al. (2009) National assessment of melanoma care using formally developed quality indicators. J Clin Oncol 27: 5445–5451. pmid:19826131
  11. 11. Kielhorn BA, Jantz JB, Kosten MS, Phillips SS, Khandavalli SC, et al. (2019) Quality improvement in melanoma care: Multidisciplinary quality program development and comparison of care before and after implementation. Am J Surg 217: 527–531. pmid:30366595
  12. 12. Tran JM, Schwartz R, Fung K, Rochon P, Chan AW (2016) Comprehensive capture of cutaneous melanoma by the Ontario Cancer Registry: validation study using community pathology reports. Cancer Causes Control 27: 137–142. pmid:26537120
  13. 13. Robles SC, Marrett LD, Clarke EA, Risch HA (1988) An application of capture-recapture methods to the estimation of completeness of cancer registration. J Clin Epidemiol 41: 495–501. pmid:3367181
  14. 14. Kralj B (2009) Measuring Rurality—RIO2008 BASIC: Methodology and Results. OMA Economics Department. Accessed 12 May 2017.
  15. 15. Elixhauser A, Steiner C, Harris DR, Coffey RM (1998) Comorbidity measures for use with administrative data. Med Care 36: 8–27. pmid:9431328
  16. 16. Wright F, Spithoff K, Easson A, Murray C, Toye J, et al. (2010) Primary Excision Margins and Sentinel Lymph Node Biopsy in Clinically Node-Negative Cutaneous Melanoma of the Trunk or Extremities. Program in Evidence-based Care Evidence-Based Series No.: 8–2: Cancer Care Ontario.
  17. 17. Renshaw AA, Mena-Allauca M, Gould EW, Sirintrapun SJ (2018) Synoptic Reporting: Evidence-Based Review and Future Directions. JCO Clin Cancer Inform 2: 1–9. pmid:30652566
  18. 18. Karim RZ, van den Berg KS, Colman MH, McCarthy SW, Thompson JF, et al. (2008) The advantage of using a synoptic pathology report format for cutaneous melanoma. Histopathology 52: 130–138. pmid:18184262
  19. 19. Morton DL, Thompson JF, Essner R, Elashoff R, Stern SL, et al. (1999) Validation of the accuracy of intraoperative lymphatic mapping and sentinel lymphadenectomy for early-stage melanoma: a multicenter trial. Multicenter Selective Lymphadenectomy Trial Group. Ann Surg 230: 453–463; discussion 463–455.
  20. 20. Morton DL, Cochran AJ, Thompson JF, Elashoff R, Essner R, et al. (2005) Sentinel node biopsy for early-stage melanoma: accuracy and morbidity in MSLT-I, an international multicenter trial. Ann Surg 242: 302–311; discussion 311–303. pmid:16135917
  21. 21. Leong SP (2004) Sentinel lymph node mapping and selective lymphadenectomy: the standard of care for melanoma. Curr Treat Options Oncol 5: 185–194. pmid:15115647
  22. 22. Reintgen DS, Jakub JW, Pendas S, Swor G, Giuliano R, et al. (2004) The staging of malignant melanoma and the Florida Melanoma Trial. Ann Surg Oncol 11: 186S–191S. pmid:15023749
  23. 23. Durham AB, Wong SL (2014) Sentinel lymph node biopsy in melanoma: final results of MSLT-I. Future Oncol 10: 1121–1123. pmid:24947251
  24. 24. Morton DL, Thompson JF, Cochran AJ, Mozzillo N, Elashoff R, et al. (2006) Sentinel-node biopsy or nodal observation in melanoma. N Engl J Med 355: 1307–1317. pmid:17005948
  25. 25. Morton DL (2012) Overview and update of the phase III Multicenter Selective Lymphadenectomy Trials (MSLT-I and MSLT-II) in melanoma. Clin Exp Metastasis 29: 699–706. pmid:22729520
  26. 26. Faries MB, Thompson JF, Cochran AJ, Andtbacka RH, Mozzillo N, et al. (2017) Completion Dissection or Observation for Sentinel-Node Metastasis in Melanoma. N Engl J Med 376: 2211–2222. pmid:28591523
  27. 27. Leiter U, Stadler R, Mauch C, Hohenberger W, Brockmeyer NH, et al. (2019) Final Analysis of DeCOG-SLT Trial: No Survival Benefit for Complete Lymph Node Dissection in Patients With Melanoma With Positive Sentinel Node. J Clin Oncol 37: 3000–3008. pmid:31557067
  28. 28. Rossi CR, Mozzillo N, Maurichi A, Pasquali S, Macripo G, et al. (2014) Number of excised lymph nodes as a quality assurance measure for lymphadenectomy in melanoma. JAMA Surg 149: 700–706. pmid:24804856
  29. 29. Brundage M, Danielson B, Pearcey R, Bass B, Pickles T, et al. (2013) A criterion-based audit of the technical quality of external beam radiotherapy for prostate cancer. Radiother Oncol 107: 339–345. pmid:23830469
  30. 30. Alawad M, Gao S, Qiu JX, Yoon HJ, Blair Christian J, et al. (2020) Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks. J Am Med Inform Assoc 27: 89–98. pmid:31710668
  31. 31. Sommariva A, Clemente C, Rossi CR (2015) Standardization and quality control of surgical treatment of cutaneous melanoma: looking for consensus of the Italian melanoma intergroup. Eur J Surg Oncol 41: 148–156. pmid:25108812