4.1 Diagnostic performance.

Fig 3 provides an overview of the proposed study design for the experiment on the diagnostic performance. Generally, this experiment consists of five consecutive steps, with the second step offering two methods to validate diagnostic performance. The first method involves verifying egg counts by reviewing and counting all eggs within the captured FOVs. The second method entails spiking a minimum number of eggs into randomly selected stool samples to achieve counts indicating an MHI infection. In the first step of the experiment, fresh stool samples will be collected from SAC at the schools. In the second step, the consistency of the stool samples will be scored based on the Bristol Stool Chart [38]. Subsequently, sample will be homogenised, and one KK thick smear per sample will be prepared in one of the two following ways for the two validation methods described above. For FOV-based validation (step 2A), samples will be processed as recommended by WHO. For egg spiking-based validation (step 2B), the cone of stool (after removing the KK template) will be spiked with purified eggs to artificially increase the egg counts to at least an MHI infection (Ascaris: >209 eggs; Trichuris: >42 eggs; hookworms: >84 eggs). This step 2B will only be done in a subset of the samples (never on samples that are processed through 2A) and is introduced to ensure that sufficient cases of MHI infections for each of the STHs are obtained (see also section 5). The selection of the samples to be spiked will be done through a randomization process. In the third step and following a smear clearing time of 30 min, the smears will be randomly allocated to be analysed by either KK1.0 (even participant ID) or KK2.0 (odd participant IDs). This randomization process is required to avoid systematic bias due to hookworm egg degradation over time [13, 14, 39]. In the fourth step, egg counts will be recorded for each helminth species (Ascaris, Trichuris, hookworm and S. mansoni), separately. Thereafter (step 5), KK thick smears will be stored at 4°C to be used in the context of the experiment on reproducibility/repeatability (see section 4.2). In Ethiopia, sample processing (from step 2 onwards) will be conducted in the Neglected Tropical Disease Laboratory of Jimma University (a fully equipped laboratory setting), while in Uganda all steps will be conducted on-site (M&E setting).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Overview of the study design for the experiment on diagnostic performance.

FOV: field-of-view, KK: Kato-Katz. Figure created using BioRender.com.

https://doi.org/10.1371/journal.pone.0309816.g003

In absence of a gold standard, it will be important to define the ground truth for each slide separately, to test the hypotheses (H1.1 – H.1.4). For the slides that were not spiked, all FOVs that were captured through the AI-DP will be manually annotated by one trained laboratory technician. A second trained laboratory technician will then verify the annotations. In case of disagreement, a third trained laboratory technician will make the final call. For the spiked samples, the ground truth (samples being classified as MHI infection) is already established through the process of spiking.

4.2 Repeatability and reproducibility.

In this experiment, we will be evaluating the two parameters repeatability and reproducibility. Repeatability refers to the variability in test results (Ascaris, Trichuris and S. mansoni) when the same KK thick smear is examined by the same operator (e.g. scanner of AI-DP/microscopist), so called intra-annotator agreement, while reproducibility refers to the variability in test results when the same slide is examined by a different operator (e.g. scanner of AI-DP/microscopist), so called inter-annotator agreement (see also Fig 4 for graphic definition of both repeatability and reproducibility). For KK2.0, we will focus on the scanning process (step 3 in Fig 2) and the AI verification process (step 5 in Fig 2). For KK1.0, we will focus on the egg counting process only (see also Fig 4). Generally, we hypothesise that both the repeatability and reproducibility of KK2.0 is at least 99%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Overview of the study design for the experiment on the repeatability and reproducibility.

Figure created using BioRender.com.

https://doi.org/10.1371/journal.pone.0309816.g004

Fig 4 provides an overview of the proposed study design for the experiment on the repeatability and reproducibility. For this experiment, we will use a subset of the KK thick smears prepared during the experiment on the diagnostic performance. The subset will comprise two slide boxes, each containing 45 KK thick smears. To ensure we assess the repeatability and reproducibility across different egg counts, we will randomly select 30 negative KK thick smears, 30 smears with a total egg count for any helminths (Ascaris, Trichuris and S. mansoni) between 1 and 100, and 30 smears with a total egg count greater than 100, resulting in a total of 90 smears. We will ensure that at least 50% of the KK thick smears in each box contain eggs from at least two different helminth species.

For the repeatability and reproducibility of the scanning process (KK2.0), all KK thick smears in slide box #1 (green) and box #2 (blue) will undergo two rounds of scanning. To ensure the entire sample is scanned, boundaries will be set larger than the smear for every scan, limiting interference and error caused by human error. The repeatability and reproducibility of the scanner process will be based on the final test results generated by the complete scanning process, which includes slide loading, boundary setting, device calibration, automatic focus setting, scan algorithms, AI detections and egg grouping algorithms. For the repeatability and reproducibility of the result verification process, the AI results of the unique scans of scanner #1 (red frame) will be verified by at least two different microscopists. For KK1.0, the examination of the KK thick smears will be conducted using the same flow as applied for the AI verification process. We will ensure that the same microscopists examine the same slides for both KK1.0 and do the AI verification for KK2.0.

4.3 Time-to-result.

During the experiments on the diagnostic performance (section 4.1), and repeatability and reproducibility (section 4.2), four different steps of the KK2.0 procedure will be timed. The four steps involve (i) participant registration (step 1 in Fig 1), (ii) the scanning process (step 3 in Fig 1), (iii) the AI process (step 4 in Fig 1), and (iv) the verification process (step 5 in Fig 1). The time required for each of these steps to be completed will be recorded by the AI-DP. The total time-to-result will be defined as the sum of the durations needed for the individual steps. Furthermore, in the Ugandan field setting, the time for setting up the AI-DP system at the different field locations will be recorded. The time-to-result for KK1.0 will not be measured in the present study. This has been intensively researched elsewhere as part of four clinical trials, each trial conducted in a different country [29, 34]. We will use these data as a comparator for KK2.0.

4.4 Cost-efficiency.

For this experiment, we built up on two general frameworks that were previously developed to support cost-efficient study design choices for large-scale STH deworming programs, including epidemiological surveys to reduce/stop large-scale deworming programs and to declare STH eliminated as a public health problem [40, 41], and to monitor the therapeutic drug efficacy [22]. Generally, these frameworks consist of three consecutive steps. In the first step, an in-depth analysis of the operational costs to process one stool sample is conducted for each diagnostic tool. In the second step, simulation studies are performed to determine the probability of making the reliable program decision. In the third step, the outcome of the cost assessment is integrated into the simulation study to estimate the total survey costs and determined the most cost-efficient study design. For the in-depth analysis of the operational costs to process one sample, we will both conduct an itemized cost assessment and determine the salary costs, which will be a function of the time-to-result (see section 4.3). For the simulation, we will deploy simulation frameworks previously published by both Kazienga et al. (2023) and Coffeng et al. (2023) [22, 40]. Both frameworks account for different sources of variation in egg counts, including (i) variability in mean egg intensity between schools; (ii) inter-individual variability in mean egg intensity due to variation in infection levels between individuals, where the level of aggregation is a linear function of the school-level mean egg intensity; (iii) day-to-day variability in mean egg intensity within an individual due to heterogeneous egg excretion over time; (iv) variability in egg counts between repeated aliquots of a stool sample due to the aggregated distribution of eggs in stool; (v) inter-individual variability in the effect of drug administration. Through the outputs of the experiment on both the diagnostic performance (section 4.1), and the repeatability and reproducibility (section 4.2), we will be able to further customize the simulation work to KK2.0 (e.g., additional variation in test results due to AI verification process and imperfect egg recovery).

4.5 Usability.

We define usability as the degree to which the KK2.0 can be used easily, efficiently, and with satisfaction/low user burden by the stakeholders [42]. For this experiment, KK2.0 naïve participants (having no previous exposure or experience with the system) will receive practical training in the use of the KK2.0 system, which includes three steps, namely the set-up, the scanning, and the AI verification process.

The practical training consists of an initial demonstration of this three-step process and a walk-through of system user manuals. Afterwards, the participants will be invited to two natural use environments, either to a laboratory setting, or a field setting. The participants will be organized into four groups per setting, each consisting of two participants per group, resulting in a total of 16 participants. This grouping reflects a planned real-life group setup, wherein the involvement of two laboratory technicians is expected to carry out the tasks. The group will be asked to perform the set-up as a team. The two following steps, the scanning and verifying AI, will be performed individually. For this, participants will be asked to each process 6 slides with KK2.0. Each slide will be processed in following order, whereby the participant’s effort will be increased: (1) the results are available soon after scanning is complete (e.g., KK2.2 results), (2) the user must perform the simple verification procedure before the results are available, (3) the user must perform the complete verification procedure before the results are available. During the three-step task performance, participants will verbalize their experiences and detect weak points in their interaction with the scanner (i.e., think–aloud protocol [43]). The whole session will be video-recorded, and data will be generated by verbatim transcriptions, and an observation checklist for collecting comparative metrics (e.g., task completion time and both error and success rates). Following, a semi-structured interview will be implemented to capture the ease-of-use/ease-of-learning, efficiency, and satisfaction/low user burden, as well as potentially missed barriers and facilitators during the task completion process. The interviews will be conducted by one investigator and structured around four sections: the background of the participant; the training; the KK2.0; the context. The data will be audio-recorded and transcribed verbatim.

5.1 Diagnostic performance, repeatability, and reproducibility.

Generally, we opted to conduct a series of simulation studies over the standard sample size methodologies, as this approach allowed us (i) to better capture the variation in test results that are otherwise difficult to account for (e.g., clinical sensitivity of KK1.0 increases as a function of egg numbers in a slide), and (ii) to ensure that the sample size calculation and the final interpretation of the field data are both based on the same statistical approach (e.g., the relative position of confidence intervals (CI) to predefined set of values; see also Fig 5). In brief, each of these simulation studies consists of a series of in-silico experiments that are iterated under different conditions (e.g., different sample sizes). Based on this iterative process, we determined the lowest sample size that allowed for confirming the hypothesis in at least 80% of the iterations (= power).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Overview of the different outcome scenarios based on a random sample and its corresponding CI.

This figure illustrates the different outcome scenarios around the difference in performance between KK2.0 and KK1.0 based on the CI. The green lines represent the scenarios where there is evidence of non-inferiority, while the lines in orange illustrate the scenarios where there is no evidence of non-inferiority. In this example we set the level of equivalence at -5 percent difference between (KK2.0 –KK1.0), a negative value indicating that KK1.0 is better.

https://doi.org/10.1371/journal.pone.0309816.g005

5.1.1 Diagnostic performance. For the clinical sensitivity to detect low intensity (H1.1) and MHI infections (H1.2), we accounted for (i) a varying clinical sensitivity as a function of the number of eggs in a slide; (ii) a proportion of the eggs in a slide being missed, (iii) correlation between test results of KK1.0 and KK2.0 on the same slide, (iii) and helminth specific FEC thresholds defining low intensity and MHI infections (Table 5). In this simulation, we assumed that the clinical sensitivity of KK2.0 is equal to that of KK1.0 and an equivalence level of 5-point percent. In other words, the lower limit of the CI around the difference (KK2.0 –KK1.0) should be at least -5% (see also Fig 5). As we will draw conclusions on three different STHs at the same time and because we are testing for non-inferiority, we set the level of significance at 0.05/3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. The FEC thresholds defining low intensity and MHI STH infections.

This table summarizes the WHO FEC (in EPG) thresholds to classify the intensity of STH infections into low, moderate and heavy [44].

https://doi.org/10.1371/journal.pone.0309816.t005

Based on these assumptions, the required number of KK thick smears representing low intensity infections based on the ground truth is 125 for Ascaris, 180 for Trichuris and 140 for hookworms. The required number of KK thick smears representing MHI infections based on the ground truth, is 110 for Ascaris and 145 for Trichuris. For hookworms, the required sample size exceeded 350, which revealed to be beyond the capacity of this project.

For the clinical specificity to detect any intensity (H1.3) and MHI infections (H1.4), we used another data generation process (based on binary test results (positive/negative) instead of egg counts). Because of this, the required sample size is the same for each of the different STHs. In this simulation study, we also (i) accounted for correlation between test results of KK1.0 and KK2.0 on the same KK thick smear, (ii) assumed an equal clinical specificity for both diagnostic tools, an equivalence level of 5-point percent, and (iii) set the level of significance at 0.05/3. Based on these assumptions, the required number of KK thick smears representing no infections based on the ground truth is 225 for Ascaris, Trichuris and hookworms each. Consequently, the required number of KK thick smears representing low intensity infections based on the ground truth is also 165 for each of the three STHs separately. It is important to note that smears can be used for multiple hypotheses. For example, when a KK thick smear represents low Ascaris and heavy Trichuris intensity infections and is negative for hookworms, it can be used for hypotheses H1.1 (Ascaris), H1.2. (Trichuris), H1.3. (hookworms) and H1.4 (Ascaris). Based on the available prevalence data, it is anticipated that we need to recruit 600 children at each site.

5.1.2 Repeatability and reproducibility. To verify whether the repeatability and reproducibility for the scanner set-up (H2.1), the AI verification process (H2.2), and the complete KK2.0 (H2.3), is at least 99%, we conducted a simulation study where we determined the number of KK thick smears that resulted in a lower limit of the CI that is at least 95% in 80% (= power) of the iterations when the true underlying probability of success equals 99%. Given that we are testing both repeatability and reproducibility at same time for each process, and that we are testing for non-inferiority, we set the level of significance at 0.05/2. Based on these assumptions the, required KK thick smears that need to be re-processed equals 90 for each of the three hypotheses.

5.2 Usability.

In this experiment, we will include 16 participants to receive (i) practical training and engage in the three-step process (ii–iv) and usability testing. A group size of 3–20 participants is considered valid in such problem discovery scenarios, with 5–10 participants being a sensible baseline range [45]. The group size should typically be increased along with the study’s complexity and the criticality of its context. Since the study will take place in two different settings, either in a well-equipped laboratory or field setting, we considered 8 participants per setting, resulting in a total of 16 participants (4 groups of 2 participants per setting).

6.1 Diagnostic performance.

6.1.1 Primary outcomes. We will draw contingency tables representing the test results of both KK1.0 and KK2.0 for each type of ground truth (no, low intensity and MHI infections) and STH species (Ascaris, Trichuris and hookworms). From these tables, both the clinical sensitivity and specificity, and the corresponding 95% CI (Wald) will be calculated for each test and STH separately. Subsequently, we will also calculate the 90% CI around the difference in performance (KK2.0-KK1.0) accounting for multiple hypotheses testing. Given that test results are paired (same smears are processed by KK1.0 and KK2.0, we will use the formulae described by Newcombe for paired data [46]. We will conclude that the clinical sensitivity or specificity of KK2.0 for a particular STH is non-inferior if the lower limit of the adjusted 90% CI does not include the -5-point percent.

6.1.2 Secondary outcomes. We will draw contingency tables representing the test results of both KK1.0 and KK2.0 for each type of ground truth for S. mansoni infections. From these tables, both the clinical sensitivity and specificity, and the corresponding 95% CI will be calculated (S1.1).

To determine the detection limit (the lowest number of eggs that yields a positive test result in 95% of the cases) of KK1.0 and KK2.0 for STH and S. mansoni (S1.2), logistic regression models accounting for repeated measures will be built for each helminth species separately using the ‘mixed_model’ function in R. The test result (positive or negative) will be used as dependent variable while ‘test’ (2 levels: ‘KK1.0’, ‘KK2.0’), log transformed egg counts based on ground truth at first examination, Bristol stool scale and all two-way interactions will be used as predicting variables. From these models, we will predict the probability having a positive test result and the corresponding 95% prediction interval for each integer value of ground truth egg counts between 1 and 100 using the ‘marginal_coefs’ function in R. We will define the detection limit as that range of egg counts for which the 95% prediction intervals include 0.95. We will explore the egg recovery rate (= observed egg counts / ground truth egg counts) of KK1.0 and KK2.0 when compared to the ground truth for Ascaris, Trichuris, hookworms and S. mansoni (S1.3). These analyses will only be conducted on KK thick smears representing low intensity infections. Finally, we will draw contingency tables representing the test results of KK2.0 for each type of ground truth (negative, low intensity and MHI infections) for each helminth species and AI verification process (simplified AI verification (limited selection of AI objects presented for verification) vs. no AI verification), separately. From these tables, both the clinical sensitivity and specificity, and the corresponding 95% CI will be calculated for each helminth species and type of AI-verification process (S1.4).

6.2 Repeatability and reproducibility.

6.2.1 Primary outcomes. The egg counts on the same smear will be considered not repeatable/reproducible in one of the following three scenarios of discrepancy: (i) there is a difference in presence/absence, (ii) the difference in egg counts exceeds 10 eggs for slides with egg counts ≤100 eggs, (ii) the difference in egg counts exceeds 20% eggs for slides with egg counts >100 eggs. These criteria are developed by the Swiss Tropical Institute of Tropical and Public Health (Speich et al., 2015), and are currently the standard way of quality control of egg counts in clinical trials [25, 26].

To determine the repeatability (proportion of cases for which a repeated test result by the same operator/scan met the aforementioned criteria) and reproducibility (proportion of cases for which a repeated test result by a different operator/scan met the aforementioned criteria) of the scanning process (P2.1) and AI-verification (P2.2), we will draw contingency tables representing the repeated test results of KK2.0 on the same KK thick smears by the same operator / scanner (repeatability) or different operator / scanner (reproducibility) for each of the different steps of the KK2.0. From these tables, both the repeatability and reproducibility, and the corresponding 90% CI (Wald) accounting for multiple hypotheses testing will be calculated for the scanning process, AI-verification and complete KK2.0, separately. We will conclude that the reproducibility/repeatability of these steps are at least 99% if the adjusted 90% CI does not include 95%.

6.2.2 Secondary outcomes. We will explore the agreement in repeated egg counts by using a Bland-Altmann plot for the scanning process, AI-verification, the complete KK2.0 and KK1.0 for each of the three helminths, separately (S2.1). In addition, we will repeat the analysis of repeatability and reproducibility for both a simplified AI-result verification process (limited selection of AI objects presented for verification) and where AI-result verification is omitted (S2.2).

6.3 Time-to-result.

We will determine the mean (and corresponding 95% confidence intervals) time-to-result (P3.1) and the time for participant registration using EDC tools (S3.1). In addition, we will also explore the correlation between time-to-result and Ascaris, Trichuris and S. mansoni egg counts recorded by KK2.0 (S3.2) based on the Spearman’s coefficient. Finally, we will repeat the analysis to determine the time-to-result of our AI-DP when the AI verification process is simplified (limited selection of AI objects presented for verification) and where AI-result verification is omitted (S3.3.)

6.4 Cost-efficiency.

We refer the reader to section 4.4 for more details.

6.5 Usability.

To achieve a thorough comprehension of the training and scanner usability, we will employ data triangulation as a method for analysing and incorporating multiple data sources. The approach to qualitative data analysis will combine inductive and deductive elements, using the determinants of usability: ease-of-use; efficiency; satisfaction/low user burden. Analytical categories will be developed from the initial research questions and emerge during the analysis process. Using NVivo (Version 14, 2020, Lumivero), identified categories will be operationalized as codes in a flexible coding scheme. The content of the codes will be discussed extensively between independent coders, and subsequently used to identify pain points and to explore improvements. The quantitative data obtained through the observational checklists will be analyzed through basic descriptive statistics.