2.1. Identifying the review questions

We developed two key research questions to address our research aims:

1. When evidence-based practice is implemented for conservation and environmental management, what are the a) human and b) environmental outcomes?
2. Among these documents reporting these outcomes, what are the costs and ROI or VOI of evidence-based practice?

We use a definition of EBP based on Sackett et al. [3] and the Sicily statement on evidence-based practice [4]: EBP involves making decisions and carrying out actions based on the best available, current, valid, and relevant evidence. Decisions and actions are also shaped by the subjects to which they apply, informed by the tacit and explicit knowledge of those implementing them, and within the context of available resources.

Our research question formulation was guided by item 4 in the PRISMA-ScR scoping review extension checklist (see S3 Appendix) [42]. Through the PCC framework, we identified the main elements (Population or participants/Concept/Context), which helped to conceptualize the review questions and the study objectives. The JBI recommends the use of the PCC [44] framework to identify the main concepts for the primary review questions and to inform the search strategy. S1 Appendix includes the PCC framework along with the identified keywords for developing search terms.

2.2. Identifying potentially relevant documents

We will conduct systematic searches in Web of Science [Core Collection], CAB Abstracts, Scopus, GreenFILE, Engineering Village (Inspec and Compendex), ProQuest Dissertations & Theses, and EBSCOhost Business Source Ultimate to gain broad coverage across fields of conservation and environmental management and to avoid missing documents and thus an unrepresentative sample of the literature [45]. We will also search the general web-based search engine Google Scholar and use the R package gsscraper to bulk download the first 400 relevant references found because previous work has shown that this search engine can be useful for finding both academic and grey literature [46]. Furthermore, we will also query the Google search engine and other specific grey literature repositories including British Ecological Society’s Applied Ecology Resources and Overton, downloading the first 400 results from each as well. We will add further search engines and repositories for accessing grey literature during this review and document any searches accordingly. For example, EThOS (a British Library database for UK theses and dissertations) was unavailable at the time of writing due to a cyber-attack.

Table 1 provides examples of preliminary database searches conducted including details of the date ranges searched and the search fields used with search terms shown in S2 Appendix. The Web of Science Core Collection accessed via Imperial College London’s institutional access includes the following editions: Science Citation Index Expanded (SCI-EXPANDED)--1970-present, Science Citation Index Expanded (SCI-EXPANDED)--1970-present, Social Sciences Citation Index (SSCI)--1970-present, Social Sciences Citation Index (SSCI)--1970-present, Arts & Humanities Citation Index (AHCI)--1975-present, Arts & Humanities Citation Index (AHCI)--1975-present, Conference Proceedings Citation Index – Science (CPCI-S)--1990-present, Conference Proceedings Citation Index – Science (CPCI-S)--1990-present, Conference Proceedings Citation Index – Social Science & Humanities (CPCI-SSH)--1990-present, Conference Proceedings Citation Index – Social Science & Humanities (CPCI-SSH)--1990-present, Emerging Sources Citation Index (ESCI)--2015-present, Emerging Sources Citation Index (ESCI)--2015-present.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Preliminary database search results and parameters for each bibliographic platform or database. Search terms are reported in S2 Appendix. We will update searches after publication of this protocol – date range searched is currently: 1^st Jan 1992 to 15^th Jan 2025.

https://doi.org/10.1371/journal.pone.0326521.t001

We will search for documents from 1992 to 2025 without any geographical restriction and document the exact search dates in the final review. We chose 1992 as the earliest publication year because this is when the terms evidence-based medicine (EBM) and evidence-based practice (EBP) were first introduced in the medical literature, widely regarded as the first discipline to promote EBP [1].

We will use the R package litsearchr [47] to identify potentially useful additional keywords based on those found in the documents during the preliminary literature search – the first reviewer will manually review suggested keywords and add those considered to be useful to the final search. The search string currently includes keywords related to: evidence-based practice, evaluations and tests, impacts or effects, ROI or VOI, and conservation and environmental management (S1 and S2 Appendices). We use truncation (‘*’), wildcards (e.g., ‘*’ and ‘?’), and Boolean operators (AND/OR), as appropriate, to form the specific search terms for each bibliographic platform and database’s rules and configurations (S2 Appendix). Librarians at Imperial College London were consulted for the development of the search strings for all databases.

2.3. Abstract screening and stopping criteria

Following the search, we will collate all identified records from the different databases into Mendeley citation manager and remove duplicates using the R package revtools [48]. We will upload the references to ASReview, an open-source tool that uses AI-assisted ‘researcher-in-the-loop’ screening procedures. We intend to use an augmented approach to using ASReview, which will allow us to efficiently speed up the screening process using AI whilst also being as transparent and accurate as possible. This augmented approach will help to identify potentially relevant documents that a human expert alone might exclude erroneously and avoid some of the current limitations of ASReview. This will also encompass the core steps in the SYstematic review Methodology Blending Active Learning and Snowballing (SYMBALS) [49], which we apply to both the abstract screening (active learning to limit the impact of false positives on efficiency) and full-text screening (snowballing to limit the impact of false negatives on comprehensiveness) stages of this review. The combination of applying the SYMBALS methodology to the ASReview tool has also been shown to outperform other methods such as FAST² both in terms of efficiency and recall [49]. We will also download a data file from the ASReview tool that documents all decisions made during the screening process to ensure full transparency and reproducibility [50]. We will use ASReview LAB v2.0a8 [51], or a more recent version if one becomes available – this will be documented in the final review [51].

The basic process of ASReview that we intend to augment is an active learning pipeline with five main elements: 1. Feature Extractor algorithm (embedding model), 2. Classification algorithm, 3. Balancing strategy, 4. Query Strategy, and 5. Stopping Rule. This active learning approach uses continuous ranking of references derived from assessments of abstracts by the author as being potentially relevant or irrelevant, following the eligibility criteria. ASReview is considered accurate and efficient in other reviews that have tested this tool’s performance [50,52–58], however, we intend to augment each of these five elements to improve the rigour and comprehensiveness of our machine learning-assisted review. This is possible thanks to open source nature of ASReview and its built-in functionality to enable custom extensions.

First, instead of the ASReview’s basic TF-IDF feature extractor, we will implement the ModernBERT Embed model, which is an embedding model from the highly effective family of BERT (Bidirectional Encoder Representations from Transformers) models. This model can process sequences up to 8,000 characters, sufficient for abstracts of varying lengths, whilst ASReview’s other feature extractor choice (S-BERT All-mpnet-base-v2 embedding model) can only handle 384 characters and might be insufficient for some longer abstracts. The ModernBERT Embed model, like S-BERT models, is specialised in evaluating sentences, allowing us to map abstracts to 768 different semantic dimensions and use these to find similar papers to the relevant papers we identify.

Second, we will use the ModernBERT Embed model in combination with a logistic regression classifier (instead of the ASReview default of Naïve Bayes), which has been shown to perform best in simulation documents in terms of recall and efficiency with BERT embedding models [55,59] – a logistic regression classifier performs particularly well for test datasets with a low proportion of relevant documents [60].

Third, we will also use dynamic resampling, a data balancing strategy [60], to decrease the imbalance in the data used to train the classifier, as we expect to find a very high ratio of irrelevant documents to relevant ones (typically 97:3 for medical and environmental topics [61,62] – i.e., relatively few positive data points). ASReview requires only at least one relevant and irrelevant article to be added as its ‘prior knowledge’ initially, but starting with an initially small number of ‘seed documents’ could lead to severe bias if such documents are selected in a biased way, even when using data balancing strategies – i.e., a high quality, representative labelled dataset is vital for machine learning tools to perform effectively [50]. Our initial searches using our preliminary search terms found three potential seed documents that were relevant (Table 2). Therefore, to tackle the data imbalance problem further and provide a larger, high-quality labelled dataset, we will also generate an additional 22 synthetic relevant documents (25 relevant seed documents in total) using a state-of-the-art Large Language Model such as Claude Sonnet 4.0 (depending on the best available model at the time of conducting our review, which will be documented in the final review). Similarly, we will randomly select 75 irrelevant seed documents from our initial searches (to achieve a 3:1 ratio for 100 seed documents in total for the training dataset). The first and second reviewers will check the relevant and irrelevant labels for the seed documents as an additional quality filter. This will involve checking the accuracy of the labels and that there is a wide representation of different topics, locations, experimental designs (i.e., good representativeness of examples), including clear positive and negative examples, as well as more marginal examples (i.e., potential ‘edge cases’ [50]). Any disagreements between reviewers in the seed documents selected will be discussed and corrected if necessary – either by regenerating extra synthetic documents or resampling the initial searches.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Preliminary seed or benchmark documents labelled as relevant.

https://doi.org/10.1371/journal.pone.0326521.t002

Fourth, we will use the default query strategy in ASReview, certainty-based sampling whereby the next article presented to be screened is the one with the next highest relevance score (i.e., moving down a ranked list in order of decreasing potential relevance [53]). Certainty-based sampling has been shown to perform well with weighted data balancing strategies [66]. Furthermore, our augmented balancing strategy (through generating synthetic seed documents), and quality filtering of our training set for our classifier discussed previously, will also mitigate potential bias from the selection of initial seed documents that can affect certainty-based sampling.

Fifth, to do any machine learning-assisted review reliably, it is vital to define a stopping rule [67]. However, defining appropriate stopping rules is still a pervasive problem in systematic review methods and for which there is typically a general lack of useful guidance [67]. For example, it is common for arbitrary, inconsistent stopping rules to be used by review papers [67,68], such as once n irrelevant papers screened in a row, or to use methodologies that appear to lack statistical justifications or proper empirical evaluations (e.g., SAFE procedure [69]). Instead, we will use a statistically justified method [70] based on point processes (statistical models to represent the occurrence of random events) whereby rate functions model the occurrence of relevant documents in the ranking and compares four candidates (hyperbolic, power law, exponential, and AP-Prior distribution). The required information is the target recall (ℓ = 95% - as suggested by [71]), confidence levels that the recall has been achieved prior to stopping (p = 95%), the total number of documents in the ranking (n), as well as the initial sample size (α = 5% of n) and the sample increment size (β = 2.5% of n) that control the number of documents examined between each application of the model to check whether to stop. Each time the model runs (if there is sufficient data to fit the rate function), it will output the estimated rank at which that the target recall has been met with the desired level of confidence.

Once the first reviewer has screened enough documents to check the stopping rule, a second and third reviewer will check a random subset of screened documents to calculate inter-rater reliability compared using Cohen’s Kappa [72]. The sample size of this subset will be determined using a power analysis with the R package kappaSize [73] (to test kappa0 = 0.4 vs. kappa1 = 0.6, with alpha = 0.05, prevalence of relevant studies = 0.05 and power = 0.80; approximately 500 documents). If Kappa values of at least 0.6 are found between the three reviewers (generally considered as high agreement), we will proceed to full-text screening and assume that the machine learning classifier was trained reliably by the first reviewer. If Kappa is less than 0.6, we will discuss any disagreements, document them, and correct them if necessary in ASReview. We will then rerun the model to estimate the rank at which we can reliably stop – if necessary, screening will continue until the stopping rule is met again. We will keep a log of excluded documents stating the reason for exclusion and report this using the PRISMA-ScR diagram for all stages of the review using the decision tree presented in Fig 1 and associated eligibility criteria.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Decision tree aid for screening abstracts and full texts – adapted from [74].

https://doi.org/10.1371/journal.pone.0326521.g001

2.4. Full-text screening and snowballing

We will access the full-texts of documents that meet the eligibility criteria from the abstract screening stage and upload them to SysRev – an online tool that allows for screening and data extraction by review teams [75]. Using SysRev also enables the full-text screening process, as for abstract screening in ASReview, to be as transparent as possible because those interested in the review can fully examine decisions about the inclusion/exclusion of documents [75]. The first reviewer will screen the full-text documents against the eligibility criteria and a random subset will also be screened by the second and third reviewers (using kappaSize [73] as per the abstract screening but with the prevalence of relevant articles = 0.5; approximately 90 documents). If Kappa is over 0.6, the relevant full-text documents will be carried forward for data extraction and summarisation in the scoping review. If Kappa is less than 0.6, we will discuss any disagreements, document them, and correct them if necessary, and repeat the process by adding a further 10 documents at a time until Kappa reaches at least 0.6. If we find any documents that should be excluded according to the eligibility criteria, we will remove them and report this in the PRISMA-ScR diagram.

Finally, we will conduct backward snowballing following [49] (considering reference lists of relevant documents), and additional forward snowballing (looking at documents citing relevant documents) for added comprehensiveness. This goes beyond the SYMBALS methodology requirements, which suggests just backward snowballing is typically sufficient [49]. We will use the R package citationchaser [76] to conduct snowballing and repeat the abstract screening stage described previously, using the same point process model stopping rule to decide when to end screening of the snowballed documents. Any relevant documents included from snowballing will be documented accordingly.

2.5. Eligibility criteria

We will include both primary and secondary published and grey literature. Fig 1 details the decision tree we will use to screen the documents.

Our specific inclusion criteria are that the article describes implementing an EBP change (e.g., starting or stopping EBP) in conservation or environmental management and measures the resulting outcomes against a comparator. The outcomes could be human (e.g., measuring decision-makers’ behaviour, beliefs, or attitudes) or environmental (e.g., measuring the impact an EBP change had on the conservation of a species, erosion of a coastline, or on water quality). This includes, for example, the implementation of an explicitly evidence-based or evidence-informed intervention and comparing the outcomes to an appropriate comparator. It also includes providing evidence and measuring how that affects decision-making compared to a comparator without evidence. We will only be able to include documents that explicitly state that evidence has been used (using our definition of evidence in the introduction from [27]) to inform practice or explicitly states that an evidence-based/informed intervention has been evaluated – the type(s) of evidence used will be documented in the data extraction stage too. Reporting of EBP costs, ROI or VOI is not part of the eligibility criteria; these data will be extracted from all eligible documents that report them.

We will include documents written in non-English languages – those in non-English languages will be translated using DeepL, an accurate AI-driven translation tool [77]. However, we will not search in non-English languages (see Section 3 Limitations). Grey literature available in the public domain from websites (in the form of reports etc.) will be considered for the review.

Accordingly, our exclusion criteria are as follows: documents published prior to 1992; book reviews, commentaries, perspectives, or editorials; not on conservation or environmental management; not evaluating process of EBP or an evidence-based intervention(s); no comparator; no human or environmental outcomes reported. At the title/abstract screening stage, we will retain any documents that meet all eligibility criteria or it is unclear whether they do or not for full-text screening (Fig 1).

All eligibility criteria will be discussed between the reviewers prior to and during screening checks and any changes justified and documented in the final review.

2.6. Extracting and summarising data

The first reviewer will perform data extraction in SysRev to extract the following data: outcomes of study (human or environmental, specific metrics used, and summary of findings), country of study, study system or habitat (if appropriate, using IUCN classification), type of publication (e.g., primary or secondary, published or grey literature), year the study was published, EBPs implemented (e.g., literature search, assessment of evidence), evidence types used (e.g., published literature, grey literature, local, practitioner, or Indigenous knowledge), any costs of EBPs numerical or qualitative (e.g., staff time, training costs, if stated), return or value on investment and its method of calculation (ROI or VOI and the type of economic assessment used, if measured) of EBPs, methodological approach (specific experimental or observational study design – e.g., Before-After, Randomised Experiment), type of organisation involved in EBP (e.g., practitioner, policymaker, academic). The categories of qualitative data collected will be refined iteratively depending on the contents of the eligible documents reviewed. We will present all results in table or graphical format and summarise them in text format. Finally, a narrative summary of the review findings will be provided and organized into thematic categories relating to types of EBP changes, their outcomes (human vs environmental), key findings and research gaps, using frequencies and qualitative thematic analysis [78].

2.7. Institutional review board statement

Not Applicable. This study will review already published and publicly available data retrieved from the databases mentioned previously.