Data sources

We began searching for Canadian historical infectious disease notification data in 2000. We focused on sub-annual and sub-national data collected through the surveillance programs described in Section A in S1 Appendix, particularly incidence over entire provinces and territories, since finer spatial resolution data were rare. We compiled provincial and territorial population data [12–14] to compute comparable incidence rates per 100,000. We acquired data in any of three formats: (1) paper hard copies, (2) digitally produced PDF files, or (3) spreadsheets (including CSV files). We scanned all of the hard copies. We found that Optical Character Recognition (OCR) was unable to convert scans into digital spreadsheets with sufficient accuracy, so we entered the data manually (see the Data entry section). We used PDF extraction tools [15] to avoid manual entry of digitally-produced PDF pages.

Data entry

We manually entered the information in scans into replica Excel spreadsheets (i.e., digital spreadsheets in which the layout of each spreadsheet matches the original), to facilitate comparing reproductions with their sources (Fig 1). The Ontario Ministry of Health data were entered before our comprehensive effort and were only partially entered as replicas (they are the sole exception). Reading scans and interpreting handwritten sources (Fig 2) was often a slow, and occasionally error-prone, process. Where available in these sources, we also entered reported annual and national data to support later validation against marginal totals (see the Quality control section). When we encountered unclear numbers in the source material, we recorded our initial guesses using a predetermined format described in Section E in S1 Appendix, so that they could be processed systematically. These guesses were revisited and refined as needed when validation procedures, which compared sums of incidence values with marginal totals reported in both CANDID sources and the PHAC portal [10], revealed discrepancies. Scripted data preparation pipelines (see the Data preparation section) facilitated updates throughout this process. We provide details on our data entry process in Section B in S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example of a typewritten source document prepared using a typewriter.

The top panel shows a scan of part of this source document, and the bottom panel shows our replica in Microsoft Excel of this same part.

https://doi.org/10.1371/journal.pgph.0005550.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Examples of handwritten data.

Most handwritten hard copies, such as the 1939 erysipelas and gonococcal data from the Ontario Ministry of Health (A), are easy to read. Others, like the 1955 poliomyelitis data from Statistics Canada (B), are difficult to read, posing challenges for digitization.

https://doi.org/10.1371/journal.pgph.0005550.g002

Data preparation

We developed open-source [16] pipelines that convert replica spreadsheets (Fig 1) into CSV files, using common variables to combine data from different sources (details in Sections D-G in S1 Appendix). The focal variable is the number of new cases of a specific disease reported in a specific location over a specific time period. CANDID consists of three CSV files, ranging from a minimally processed file offering maximum flexibility in data preparation to a heavily processed file prioritizing convenience; each CSV file corresponds to a stage in the data processing pipeline (Fig A in S1 Appendix).

The unharmonized file preserves the raw, digitized data, giving researchers the freedom to apply their own data preparation methods. Descriptors in the unharmonized file use original names (e.g., “infantile paralysis” for poliomyelitis before 1924) to minimize historical information loss [17]. The harmonized file removes low quality data, aggregates some municipal data to provincial levels, and adds harmonized location and disease descriptors that simplify the combination of data from different sources (e.g., poliomyelitis whenever infantile paralysis is reported). Harmonized CSV files are convenient for querying diseases, provinces, and time periods, but they cannot be used directly for analysis due to overlapping incidence values. In data science terms, the harmonized data are not normalized [18,19]. For example, the harmonized data include total polio incidence alongside separate values for polio with and without paralysis (see Section G in S1 Appendix for more examples). Such overlapping data are useful for quality control (which we explain in the Quality control section), but data analysis requires removing overlaps to prevent double-counting cases. The normalized file does not contain overlapping incidence values, enabling aggregation without double-counting. When deciding which overlapping records to remove, we generally retain the finest resolution. Continuing our example, we would remove total polio incidence and retain separate records for polio with and without paralysis (see Section G in S1 Appendix for details on removal criteria). For convenience when computing incidence rates, we joined provincial population sizes to the normalized data. All figures in this paper are based on the normalized file. Researchers who wish to make different harmonization or normalization choices can use the upstream files (see Section G in S1 Appendix for details).

Data provenance

Each record can be traced back to the relevant original scan, replica spreadsheet, and/or script used to produce it, using information in the CSV files (Section H in S1 Appendix). We follow research data management practices by distributing DataCite [20] (version 4.3) metadata with each dataset in the archive. These metadata will make it easier to deposit future versions of CANDID into a research data repository, which we plan to do.

Quality control

We compared sums of incidence values with marginal totals reported both in CANDID sources and in the PHAC portal [10]. Discrepancies suggest possible data-entry or scripting errors. We investigated discrepancies and fixed those that appeared to be due to digitization error. These investigations were simplified using our open data provenance tools (see the Data provenance section) and digitized data source replicas (Fig 1). We provide full detail on quality control in Section I in S1 Appendix.

Examples

Poliomyelitis cases peaked at the same time each year across all provinces.

Poliomyelitis incidence was strongly seasonal (upper panel, Fig 7), with national peaks consistently occurring between week 31 and 40 each year from 1933 to 1963 after which the cycles disappeared. These annual cycles were synchronous across provinces, with peak incidence occurring around the same time in each (provincial panels, Fig 7). All national peaks fell between August and October, and most provincial peaks followed this pattern, with only a few outlying province-year combinations (Fig I in S1 Appendix). Identifying such spatial synchrony requires incidence data at sub-annual and sub-national scales.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Weekly poliomyelitis incidence from 1933 to 1968 (square root scale).

The vertical lines do not indicate the start of each year but mark the week of peak national incidence (top panel) in years with more than 20 cases. Provincial peaks closely align with these national peaks, indicating strong spatial synchrony. This pattern could not have been detected with annual or national data. The introduction of two important vaccination programmes are shown as blue vertical dashed lines. We plot incidence rates per 100,000 to make incidence comparable across provinces and territories, and with other studies. Methods for producing this plot are described in Section J in S1 Appendix.

https://doi.org/10.1371/journal.pgph.0005550.g007

Regional differences in whooping cough incidence.

Aggregating provincial whooping cough data to the national level (Fig 8) reveals a pattern consistent with previous analyses that lacked provincial data [22]. One feature of this pattern is an apparent resurgence of whooping cough in the 1990s (highlighted in Fig 8). We find that this much-discussed resurgence (e.g., [22]) was not uniformly expressed across the country (Fig 8 bottom six panels), and is clearly apparent only in the territories, the prairies, and Québec. Throughout the 1990s, yearly cases per 100,000 peaked at 21 in Ontario, but in the territories the peak was 293. The original study [22] did not have the sub-national data required to explore these regional differences.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Regional [23] differences in average annual whooping cough incidence in Canada over twelve decades (square root scale).

The national data (top panel) are very similar to the first figure in a review based on different data sources [22]. The red region (1990–1999) highlights the first resurgence in national whooping cough incidence since widespread vaccination began in 1943. This plot shows that not all regions peaked in the 1990s, a pattern that could not have been detected with the national data used in the original study. Methods for producing this plot are described in Section K in S1 Appendix.

https://doi.org/10.1371/journal.pgph.0005550.g008

Future directions

In this initial release of CANDID, we focused on sub-annual and sub-national disease incidence, with plans to expand further. First, we will include data stratified by age, sex, and municipality for available time ranges, provinces, and diseases. Age data, for example, can be critical for estimating the impact of vaccination programmes in childhood diseases (e.g., [32]). Second, we will extend the dataset’s time range, disease coverage, and geographic detail as finer-scale or corrected data become available. Third, we will curate population-level information useful for epidemiological analyses, such as birth rates, mortality, vaccination, and school-term dates. Beyond expansion, we will address open questions in Canadian infectious disease history. For example, we will test whether regional differences in vaccination programmes could explain spatial variation in the size of the 1990s resurgence of whooping cough (Fig 8). Finally, we will explore AI-powered optical character recognition, using our archive as a training dataset to enhance its efficiency and accuracy.

Logistical challenges

The process of assembling CANDID highlighted logistical obstacles that are likely to arise in other countries seeking to build comparable historical surveillance datasets. The first challenge was locating and obtaining source material. Historical records were dispersed among federal and provincial publications issued under evolving titles and formats, with many available only as hard copies in archives or libraries. Coordinated searches, correspondence with data stewards, and formal data requests to multiple agencies and libraries were required. These challenges are described in more detail in Section A in S1 Appendix.

A second challenge was manual data entry. Optical character recognition could not accurately extract numerical tables from scans, necessitating full manual transcription. Coordinating several data enterers required standardized templates, shared conventions for unclear entries, and frequent communication to ensure consistency across spreadsheets. Looking ahead, advances in artificial intelligence may mitigate some of these difficulties, as noted in our future plans (see the Future directions section), although many questions remain.

A third challenge was harmonizing information drawn from sources that differed widely in format and structure. Converting these diverse formats into consistent, machine-readable CSV files and resolving overlaps across locations, time scales, disease hierarchies, and data sources required careful sequencing of processing steps and version control across multiple repositories. These challenges are detailed in Sections D-G in S1 Appendix.

Finally, quality control involved iterative cross-checks across diseases, jurisdictions, and time scales. Addressing discrepancies meant revisiting earlier stages of data entry and processing, emphasizing the importance of reproducible pipelines. These challenges illustrate the logistical scale of constructing a comprehensive historical surveillance archive. By documenting our workflows and tools openly, we have worked for CANDID to serve as both a resource and a practical starting point for researchers undertaking similar digitization efforts elsewhere.

Limitations

Under-reporting is a known limitation of surveillance data [31,33]. Correcting for it requires disease- and context-specific methods that rely on supplementary data (e.g., serological surveys, case-fatality ratios, demography) and modelling [34]. Given the number of diseases and years covered, such corrections are beyond our scope. Our contribution is to make data available in a form that supports future work, including efforts to address under-reporting. As one step in that direction, we distinguish true zero case counts from data that were not reported, based on information available in the original sources (see Section E in S1 Appendix).

In addition to under-reporting, two other factors complicate the interpretation of these incidence data: evolving sub-disease hierarchies and inconsistent time scales. First, changes in how diseases are classified and reported over time affect comparability, as the level of aggregation can vary year to year (e.g., the normalized dataset includes 37 meningitis sub-diseases, with 1–15 reported in any given year; Figs C–F in S1 Appendix). Second, variation in reporting frequency (e.g., weekly, monthly, quarterly; Figs 4, 5, and 6) hinders the construction of evenly spaced time series, which are often needed for modelling and visualization.

Historical gaps in CANDID persist due to surveillance program changes (e.g., chickenpox was not notifiable from 1959–1985) and incomplete source coverage, particularly outside Québec, Ontario, and Saskatchewan before 1924. These gaps range from missing weeks (e.g., lost book pages) to illegible handwritten records (see the Quality control section). Although transcription and coding errors cannot be ruled out, we compared reported subtotals with their marginal totals and corrected all identified discrepancies for our primary example diseases (whooping cough, poliomyelitis). By releasing open data pipelines, we aim to enable similar checks across all diseases and to support collective efforts to improve the completeness and reliability of CANDID over time (details in Section I in S1 Appendix).

Conclusion

More than a century of infectious disease surveillance in Canada has produced a valuable record of epidemic patterns that has been largely unexploited, but can now be easily accessed. Comprehensive sub-annual and sub-national Canadian infectious disease incidence data have previously been unavailable. Similar data from other countries have been critical to establishing the foundations of epidemiological modelling and continue to push the field forward (e.g., [1–4,35–42]).

CANDID makes it possible to study variation in disease incidence within years and across provinces, with applications ranging from infectious disease research to broader interdisciplinary work. For example, it can be used to assess how incidence in Canada relates to socio-economic factors such as urbanization and wealth inequality. The dataset also supports public health planning by situating recent outbreaks and epidemics within their historical context.

In principle, it should be straightforward to keep the archive up to date, but doing so will require the cooperation of provincial and territorial public health agencies, which have released less data publicly since strictly digital data collection began in the 1990s. We contacted all these agencies but were able to obtain recent weekly data from only two provinces. While we recognize that agencies may face practical constraints, such as the staff time required to prepare and maintain public releases, we encourage Canadian governments to support routine public access to weekly, aggregated counts of infectious disease notifications.