Study design and sample

We conducted a cross-sectional exploratory survey administered via targeted email to Ecolectivos investigators, as well as external evaluators familiar with implementation research who were not participating in the Ecolectivos trial (hereafter referred to as “external evaluators”). Between February 2022 and January 2023, surveys were sent out to 60 researchers across 20 academic institutions, comprised of 46 external evaluators and 14 Ecolectivos investigators. To improve the response rate, three reminders were sent during this timeframe. All survey instruments were administered in English.

The survey included questions related to the PRECIS-2 domains for the evaluation of the original Ecolectivos study protocol. The original PRECIS-2 paper [7], an educational Powerpoint presentation, and a paper using the PRECIS-2 tool for a global air pollution trial was provided so that reviewers could understand our intended approach [1]. We asked reviewers to rank their perceived degree of pragmatism in the PRECIS-2 nine domains of trial design and conduct, using a five-point Likert scale: very explanatory (1), explanatory (2), equally explanatory/pragmatic (3), pragmatic (4), or very pragmatic (5). We also asked them to provide short answers explaining their score choices. We also asked participants two questions, “on a scale of 1–10 (1 = low, 10 = high), how knowledgeable are you about the discipline of Implementation and Dissemination Science?” and “on a scale of 1–10 (1 = low, 10 = high), how much experience have you had working with randomized controlled trials?”

The nine domains of PRECIS-2 tool described briefly are: 1) eligibility (trial participants similar to those who would receive the intervention if it was usual care); 2) recruitment (effort needed to recruit participants beyond what would be used in usual care settings); 3) setting (setting similar to usual care setting); 4) organization (resources, provider expertise, and organization of care delivery in intervention different from usual care); 5) flexibility in delivery (flexibility in how the intervention is delivered versus flexibility in usual care); 6) flexibility in adherence (flexibility in how participants are monitored to adhere to the intervention versus usual care); 7) follow-up (follow-up of participants in trial similar to typical follow-up in usual care); 8) primary outcome (trial’s primary outcome directly relevant to participants); and 9) analysis (all data included in the analysis of the primary outcome). The developers of the PRECIS-2 tool found the tool to have good face validity, discriminant validity (able to determine pragmatism across a broad spectrum of trial protocols) and interrater reliability (intraclass correlation coefficient > 0.65) [30].

All participants provided written informed consent to participate. They entered their responses into an anonymous, online data collection tool, which was stored on a password protected, secure cloud-based server. We did not collect personal identifying information; instead we collected gender, race, ethnicity, and age as categorical demographic characteristics. We did not collect data on educational degree (all possessed a doctorate degree (e.g., PhD or MD)) or specific institution.

Ethics approval and consent to participate

All participants provided written informed consent to participate. Responses did not include identifying information about the respondent. The study protocol has been reviewed and approved by institutional review boards (IRBs) at the Universidad del Valle de Guatemala (245-05-2021) and Emory University (#00002412), with Reliance Agreements the University of Georgia (#00005189) and the University of California at San Francisco (21–35706). The study has been registered with ClinicalTrials.gov (Identifier NCT05130632).

Data analysis

We summarized PRECIS-2 Likert scores using medians (interquartile ranges (IQR), representing the 25^th and 75^th percentile scores) for each domain. We evaluated the distributions of the data to assess whether assumptions of normality were met. Since the data were not normally distributed, we tested the differences in scores between the Ecolectivos investigators and the external evaluators using the nonparametric Mann Whitney-U test. We used Excel and SPSS version 29.0 for data analyses. Given the nature of this small study, we did not conduct a power analysis for the quantitative analysis.

The purpose of the qualitative analysis was to describe themes that drove the PRECIS-2 domain scores. The short responses supporting why each respondent chose their score for each of the nine domains were analyzed thematically and grouped to illustrate common, recurrent patterns in the qualitative data that enriched the quantitative findings [31]. Themes were identified and exemplars were then extracted to illustrate the theme and integrated into the results. Using online surveys to retrieve descriptive qualitative findings has been found to provide valid information for researchers [32].

1. Eligibility criteria: To what extent are the participants in the trial similar to those who would receive this intervention if it was part of usual care?

The median response for eligibility among all respondents was 4 (out of a range of 5), or rather pragmatic (IQR 3, 5). Narrative feedback highlighted that the protocol’s broad inclusion criteria was the top rationale for receiving a primarily pragmatic ranking. One external reviewer who viewed the trial as “rather pragmatic” stated, “as community level workshops have no inclusion or exclusion criteria, the intervention itself is quite pragmatic. Exclusion of pregnant women or tobacco-using…. for the urine and personal air pollution for the [biomonitoring women] samples (as opposed to lack of criteria for questionnaires to workshop attendees) is why I selected "rather" instead of "very" pragmatic.” Several respondents noted that the intervention would likely be reflective of actual community conditions since there are minimal exclusion criteria. However, although the eligibility criteria make the sample likely representative of the Jalapa, Guatemala population, evaluators disagreed about whether this was a limitation or benefit to overall pragmatism. While Ecolectivos members stated the trial was more explanatory (median, 2 [IQR 2,3]), external evaluators scored the trial as more pragmatic (median, 4 [IQR 4,5]) (p-value, < 0.001), showing a divergence of opinions. One Ecolectivos co-investigator stated, “For the main trial, there are a fair number of eligibility criteria, and the willingness to attend 12 working group sessions may exclude a lot of people.”

2. Recruitment path: How much extra effort is made to recruit participants over and above what would be used in the usual care setting to engage with patients?

The median response for recruitment among all respondents was 3 (IQR 2, 4), or equally pragmatic/explanatory. With a relatively balanced perception, evaluators commented on how recruitment meets participants “where they are.” Leadership from community members through an established Community Advisory Board (CAB) was also referenced as a pragmatic recruitment feature. One evaluator stated, “the recruitment path, while explanatory in justifying word-of-mouth and informational meetings, is also pragmatic in that it places leadership into community members/stakeholders via CAB input. This represents a pragmatic solution to reaching a dispersed population. I find the approach to be just as pragmatic as it is explanatory for this setting.” The random selection of households for the community-level baseline survey, with village size, distance between villages, and proximity to main roads taken into consideration for stratified sampling, is cited as an explanatory moderating factor. Notably, evaluator commentary spanned several aspects of the recruitment process; for example, recruitment for the recruitment for baseline community survey, which required fieldworkers canvasing neighborhoods door-to-door to administer surveys using google survey maps and census data to randomly select households, was deemed as more explanatory: “For the formative phase baseline assessment, the door-to-door approach requires extra effort and would not typically be part of usual conditions”, while recruitment for the biomonitoring randomized controlled trial among women of reproductive age was viewed as more pragmatic, given the methods of recruitment, “however, for the main trial, recruitment seems to be within usual effort for engagement with this type of project, given the community-based approach, village-level randomization, and widespread existing use of biomass fuel for cooking and burning trash/plastic as a typical means of disposal. Most communities already hold regular meetings or events, so it is expected that approaching people for surveys [through village meetings] would not be too extreme of an effort for recruitment.”

3. Setting: How different are the settings of the trial from the usual care setting?

The median response for setting among all respondents was 4 (IQR 3, 5), or rather pragmatic. Most of the evaluators noted that the intervention will target rural Xinca indigenous communities in a specified region where there is limited waste disposal infrastructure or recycling programs, requiring community-based strategies for plastic risk mitigation. These rural indigenous communities experience similar circumstances as other communities in Guatemala that rely on burning waste for disposal. Evaluators stated that the study setting itself represents the intended audience where outcomes would be utilized, supporting external validity and pragmatic design. One respondent noted that “[t]he setting in the Jalapa Municipality of Guatemala across eight intervention and eight control villages for the main trial seems to be very pragmatic. This region relies heavily on biomass fuel for cooking and burning trash/plastic as a usual means of disposal. The villages will capture a range of settings and natural conditions, rather than just a single, specialized site. The in-home assessments of exposures and biomarkers of exposure are also all occurring in very real-world conditions.” Of note, one evaluator pointed out that the longstanding relationship between researchers and the community may create results that deviate from “usual” community intervention settings, “as with any RCT, it is rather site-specific, and it’s a single site study in a sense. It’s also a study with a long relationship with the investigator team, so there may be more compliance to the intervention, higher acceptability than in other real-world settings.”

4. Organization: How different are the resources, provider expertise, and the organization of care delivery in the intervention arm of the trial from those available in usual care?

The median response for organization among all respondents was 3 (IQR 2, 4) or equally pragmatic/explanatory. Evaluator feedback highlighted several organizational aspects of the intervention, including its feasibility, resources required, and involvement of community members. The intervention’s combination of scientific and community expertise generated a near split in opinion between pragmatic vs explanatory features. Although the study involves external expertise and resources to deliver the intervention, it involves community members in developing interventions, which adds pragmatism. Overall, this was assessed as a balanced approach. One evaluator scored this domain as “rather explanatory” and commented on the complexity of the organizational challenges leading to a more explanatory assessment, given that external resources are required: “[T]he trial requires the efforts of many fieldworkers and external investigators to pull off the intervention, none of which are part of what would normally happen.” Discussion around more explanatory features of the trial also included that training and resources for the 12-week educational sessions are intensive, requiring specific expertise and external resources, and “the intervention is not "standard" care or part of existing community-based services.” The involvement of fieldworkers to improve adherence is also noted by some as shifting their trial rating towards explanatory. On the other hand, since the intervention takes place within the community setting, other evaluators found it practical for the targeted population, as evidenced by features such as use of community centers for workshops, and the role of key informants and village champions to support community conditions for behavior change. Several evaluators pointed out the likely scalability of the intervention, which may be delivered similarly under real-world conditions, increasing its pragmatic usefulness.

5. Flexibility of experimental intervention–delivery: How different is the flexibility in how the intervention is delivered and the flexibility anticipated in usual care?

The median response for flexibility of the experimental intervention in delivery among all respondents was 4 (IQR 3, 5), or rather pragmatic. Several respondents noted the formative phase as a strength, with one stating: “[T]he formative [phase] followed by the RCT will allow for a balance between the pragmatism of the intervention (and allowing the community to drive its development) while providing scientific evidence for the benefit of the selected intervention in the later stages.” In provided rationale, evaluators noted that the protocol is adaptable and responsive to community input, allowing participants to choose intervention approaches that appeal to them and address their specific barriers: “It is very clearly mentioned as to what modules are fixed and which are flexible. It serves the purpose of the project as well as needs of the community for the skill to be learned. It’s a win-win situation.” Since participants play an active role in generating intervention ideas, this was seen to favor pragmatic design and support external validity, or “the details of implementation and intervention are flexible to community members’ needs and decision-making. This creates an intervention that should proxy what happens under usual circumstances.” The multi-prong approach with evidence-based strategies, the formation and structure of community working groups, and community education sessions are noted to be externally driven and less flexible. Evaluators identified features that seemingly strike a balance between a pragmatic and a structured, more explanatory delivery plan.

6. Flexibility of experimental intervention–adherence: How different is flexibility in how participants are monitored and encouraged to adhere to the intervention from the flexibility anticipated in usual care?

The median response for flexibility of the experimental intervention in adherence among all respondents was 3 (IQR 3, 4), or equally pragmatic/explanatory. Evaluators noted that there are established groups in place, such as village champions and trained fieldworkers “available to resolve bottlenecks”, which are intended to increase adherence. However, several respondents felt that details about specific measures of adherence, such as “how many times study staff will reach out to participants to get them to come to the sessions” or “field workers will check on participants, but there’s no way to know what the participants are doing when not being watched” would have further clarified the pragmatism of protocol design. With the last four weeks of the educational intervention being participant-directed, evaluators noted this factor as likely to increase flexibility and community buy-in, with one stating “the fact that the participants get a chance to provide their input already increases their buy-in which will promote adherence to the study. While it is good to have strict protocols, it is important that you are incorporating room for flexibility with the modules.”

7. Follow-up: How different is the intensity of measurement and follow-up of participants in the trial from the typical follow-up in usual care?

The median response for follow-up among all respondents was 2 (IQR 1, 4), or rather explanatory. Respondents focused attention on the follow-up of the biomonitoring group of 400 women who would be visited three times in their homes to assess exposure to air pollution. Evaluators noted that the regular follow-up surveys and biomarker assessments require trained data collectors, described by most (52%) to be very or rather explanatory. One respondent stated, “this trial requires regular assessments of 200 intervention and 200 control participants across 16 villages for monitoring exposure and urinary biomarkers of exposure at baseline, 4 months, and 12 months. There is also scheduled visits for plastic waste collections at baseline and 4 months. This is fairly rigorous for measurement of trial outcomes and requires extensive data collection.” However, two respondents thought that the trial was pragmatic. One stated that the trial was “very pragmatic” since a “12-month post-intervention measure could be insufficient” if adherence to the intervention geared at “not burning plastic” is not followed. Another evaluator stated that the trial was “rather pragmatic” because follow-up after the educational intervention was infrequent, and exhibited “minimal follow-up visits, data collection, and dissemination performed within the community group setting. This represents close to usual follow-up without any demanding expectations for participants.”

8. Primary outcome: To what extent is the trial’s primary outcome directly relevant to participants?

The median response for the primary outcome among all respondents was 3 (IQR 2, 5), or equally pragmatic/explanatory. The biomarker sampling procedures, which measure exposure to air pollutants, including pollutants from plastic waste burning, were by some, deemed as tangible features of trial outcomes, and “directly relate to participants’ exposure burden.” However, it was pointed out that participants may not find these data immediately relevant a priori if they are disconnected from the health effects associated with household air pollution. Others felt that the importance of intervention strategies and their direct relevance to participants were pragmatic in that the “educational intervention would likely hold significance to participants” and “villagers expressed their desire to manage plastic in ways other than burning”. Feedback indicated that certain outcomes may be less relevant to participants, but they are important for evaluating the effectiveness of the intervention and research. One respondent summarized, “while the primary outcomes are important to participants’ health, I don’t think these outcomes will be what would entice participants to make changes or continue those changes outside of the study. No one is going to think "I am going to recycle this plastic instead of burn it because it will keep my bisphenol levels stable." People make changes because of the importance for their everyday life circumstances (finances, ease, and so on). I think the secondary outcomes [health-related quality of life and general self-efficacy] are more explanatory than the primary ones [air pollution exposure and urinary biomarkers of exposure].”

9. Primary analysis: To what extent are all data included in the analysis of the primary outcome?

The median response for the primary analysis among all respondents was 4 (IQR 2, 5), or rather pragmatic. The intention-to-treat analysis of quantitative biomarkers, which “does not have many exclusions of data for primary analysis,” led some respondents to comment that the study was more explanatory than pragmatic. A strength of the study was in the “mixed methods approach that will combine data from multiple sources to evaluate the intervention.” Overall, the feedback suggests that the data analysis plan, including the handling of missing data, appears to be pragmatic and well-considered. Many stated that the intention to include all available data, even for participants who may drop out or become pregnant during the study, reinforces the perception of a more pragmatic approach. This was well summarized in one statement, “there is no systematic exclusion of data detailed in the protocol; the intervention protocol details how any missing data will be addressed during statistical evaluation of all data collected. Data analysis is well defined and pragmatic.”

Limitations and strengths

Although we had a lower participation rate, our response rate was higher than many online surveys. The consequent small sample size (n = 25) can affect overall generalizability. Further, we had greater participation from Ecolectivos investigators (64%) than from external reviewers (35%), which is not surprising; thus, response bias may be a concern for interpretation of findings. Because we found a strong consensus on the pragmatic and explanatory features of the study across these two groups, this enables a broadened perspective on whether the implementation protocol will serve its intended purpose with consideration toward external validity. Of note is that although the median self-rated knowledge and experience with implementation science was higher among external evaluators (7/10) than among Ecolectivos investigators (5/10), results were similar. Other limitations were the requirement to be an English speaker, which meant that some members of our Guatemalan team could not evaluate the study, and that the online survey only included short-answer responses, which limited the richness of the data for qualitative analysis. Study strengths include the incorporation of assessments by implementation science experts external to the Ecolectivos research collaborative, which may serve to offset biases had the protocol been evaluated only by Ecolectivos investigators, and the uniqueness of the topic in the implementation science intervention space. Our findings contribute to the growing body of literature tying community environmental health interventions to the rigor of implementation science.