PONE-D-22-30207Did the evidence-based intervention (EBI) programme reduce inappropriate procedures, lessen unwarranted variation or lead to spill-over effects in the National Health Service?PLOS ONE

Dear Dr. Glynn,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Apr 28 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Christian Gericke, MD PhD MSc MPH MBA

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following financial disclosure: 

This project is funded by the NIHR Health and Social Care Delivery Research Programme (NIHR130547). TJ’s time is supported by the National Institute for Health and Care Research Applied Research Collaboration West (NIHR ARC West). The views expressed in this article are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

4. We note that Figure 2 in your submission contain map images which may be copyrighted. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For these reasons, we cannot publish previously copyrighted maps or satellite images created using proprietary data, such as Google software (Google Maps, Street View, and Earth). For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure 2 to publish the content specifically under the CC BY 4.0 license.  

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

The following resources for replacing copyrighted map figures may be helpful:

USGS National Map Viewer (public domain): http://viewer.nationalmap.gov/viewer/

The Gateway to Astronaut Photography of Earth (public domain): http://eol.jsc.nasa.gov/sseop/clickmap/

Maps at the CIA (public domain): https://www.cia.gov/library/publications/the-world-factbook/index.html and https://www.cia.gov/library/publications/cia-maps-publications/index.html

NASA Earth Observatory (public domain): http://earthobservatory.nasa.gov/

Landsat: http://landsat.visibleearth.nasa.gov/

USGS EROS (Earth Resources Observatory and Science (EROS) Center) (public domain): http://eros.usgs.gov/#

Natural Earth (public domain): http://www.naturalearthdata.com/

5. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the opportunity to review this piece on the (in)effectiveness of the EBI programme. It is well written and presented and adds to the important literature on -de-implementation.

A previous analysis (by another group) has already assessed the global impacts of the programme, so in a sense this is a ‘subgroup’ analysis looking at individual elements, as well as adding useful analyses including the effects on variation across the country. I think there is clear added value.

Given that the previous analysis has been done by an entirely different group, I think a more explicit comparison (and a clearer sense of the differences in methods between the two papers) would also be helpful for readers. I have not read the other paper but it appears it used a ‘differences in differences’ model as opposed to the ITS design here, and it was not clear to me why the methods varied (and I assume the former design would be stronger, all other things being equal?). That might be explained more clearly in the current paper given the overlap.

I think a short description (maybe in a Box) of the procedures and clinical criteria could be provided – not everyone will have the capacity to seek out additional references and I would prefer that the paper is self-contained.

I would also like a little more on the mechanisms of the EBI – the Discussion suggests it involved ‘publishing nationwide guidance’ but that sounded quite passive? Is it plausible that the methods used would have the sort of impacts that they were looking for, especially in the context of overall reductions over time?

There is a useful new framework for ‘spill over’ (see Francetic et al in Implementation Science Communications - https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-022-00280-8). I am from that group so something of a competing interest, but I wondered if the current paper might benefit linking the work to that framework if it added value.

I am not a statistician so am not able to judge the technical quality of the analyses but they seemed rigorous and I found them clearly described.

The description of the numbers of procedures on page 10 would benefit from percentages and I would link those frequencies to Table 1 so that readers could judge the volume alongside the data on changes.

A minor issue, but I found the term ‘increased trend’ a bit confusing on page 13, and ‘strong evidence’ a bit vague. It would help to tighten the language – I assume ‘strong’ evidence relates to conventional significance? Does ‘increased trend’ imply significance?

Is power an issue for any of these analyses? I assume for high volumes it is not, but what about lower volumes and subgroups by CCG? Are any of the analyses insufficiently precise because of numbers, or are the lower N still enough for confidence to detect policy relevant changes? Is there a sense of what size a policy related change would be to anchor the analyses?

I did struggle with the suggestion in the discussion that the EBI may have had a role in continuing downward trends. How is that plausible given the data? Additionally, is the ITS even capable of assessing such an effect – the section on the ITS on page 17 suggests that this is not the case anyway? I felt that was a ‘reach’ and would prefer to see that removed unless they can justify it more clearly.

In the discussion of the ITS on page 17, I am not sure why the lack of control group is moderated by analyses over 17 procedures. If the design is flawed then multiple uses of the same design is not going to provide any protection (see Marion Campbell’s ‘red sock in a white wash’ analogy for bias). Maybe I am missing the logic but this did not seem plausible to me. I am happy with the ITS though (although interested in why Anderson used ‘differences in differences’).

There is an interesting conclusion that ‘de-adoption programmes are essential’. I understand the sentiment, although one interpretation is that wider, non-programmatic changes are leading to reductions, whereas a specific programme to drive change added nothing. I wonder if the conclusion needed changing to better reflect the actual results that they present?

Reviewer #2: Thank you for this paper which undoubtedly adds to the literature about the challenges of deimplementation. Please find some comments and suggestions below. In general, I think some additional information on the methods and the "so what" of your findings would improve the manuscript further before publication.

Background

Line 85: NHS England and Improvement -> a bit confusing as it stands for readers outside the UK (two separate entities? NHS England and NHS Improvement?)

Line 94: 17 surgical procedures were initially identified -> how? And how were the EBI recommendations reached?

Line 101: Substantial -> would be good to have a clearer understanding of extent (e.g. in %)

Lines 106-107: how is the first task different from what Anderson et al. had already done?

Methods

Line 118: up to 24 procedures per person/case?

Line 119: how you identified eligible episodes of care is not fully clear, consider elaborating with 1-2 lines

Line 120: consider defining episode of care for this article. Is the data linked/linkable across datasets/settings on an individual patient basis? Please describe the data and how you used it for your research questions a bit more.

Line 139 onwards: consider providing an example here as well as you do below for the substitute procedures. For readers not versed in EBI, it is not necessarily intuitive why an EBI procedure code with a non-EBI diagnosis is a related procedure.

Line 145: met with 11 surgeons -> when, selected how, how did you collect information from them etc. (see above about describing data and method. approach a bit more)

Line 149: how did you set the 10% threshold (rationale)?

Line 161: write out ONS, potentially explain its function

Line 169-170: COVID-19 impact on procedures -> would it make sense to discuss a bit more in the discussion if these reductions in case numbers mean reduction of low-value procedures or increased unmet need?

Line 172: analysis instead of analyses

Line 197: Please check if there is a word missing... if "EBI list one" is a thing, perhaps good to explain what.

Results

Line 207: might it make sense to comment on the likely full year value for 2019/2020?

Line 210: good that you provide the range, any chance of getting a sense of distribution too without going to the Annex?

Lines 222 onwards: consider rephrasing this period, it is a bit difficult to read.

Lines 239-240: would be useful to comment on why same direction and not significant are equated here.

Discussion:

Line 292: move the reference bracket to line 291 after "al."

Line 311-316: for readers not familiar with the EBI, it would be good to understand better what if any implementation tools accompanied the EBI and comment on potential alternatives among the ones you describe here. it would be important to reflect on this as the best possible evidence-based recommendation may not affect outcomes if not actively disseminated, and even then other barriers play a role. I think the paper would benefit from a bit more input from the literature on these issues in the discussion.

Lines 328-330: does this mean you actually had a national sample? perhaps good to add on representativity of data in the methods.

Lines 352-354: This needs to be moved to methods and get a bit more detail.

Line 359: analysed instead of identified?

Line 360-362: polish style a bit (now procedures comes up twice in short interval)

Line 366 onwards: should that not be coupled with monitoring efforts to recognise if deimplementation levels are/become satisfactory, though? From what point onwards is a downward trend considered sufficient evidence of a process of deadoption?

Line 372: well-developed de-adoption ->this is a valuation for which you (rightly) do not provide evidence here. Consider rigorously developed, or successful, as alternatives.

Lines 386-389 -> indeed. How, though? It would be good to end with your views on how this can be achieved, e.g. solutions to low-value care

Lines 403-404 -> update as now already beyond that point

Lines 407-410 -> yes! see previous comment onlines 386 onwards. Consider rearranging the discussion a bit to strengthen this narrative and make it more easily visible

Line 418: we need to BETTER understand (I think there is already knowledge on this, but you are right that we need to increase/strengthen it and its application)

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Peter Bower

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Did the evidence-based intervention (EBI) programme reduce inappropriate procedures, lessen unwarranted variation or lead to spill-over effects in the National Health Service?

PONE-D-22-30207R1

Dear Dr. Joel Dominic Glynn

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Dirceu Henrique Paulo Mabunda, M.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #3: Thank you for the opportunity to read this revised manuscript. The manuscript is well written and clear, and the methods used are appropriate. The authors have responded to the comments of the two previous reviewers in a rigorous and pertinent manner.

There is no point of making some additional comments at this stage. I am only making the following (not mandatory) suggestions that the authors might consider if they think they can help further assess the internal validity of the results.

Please find these optional suggestions below.

Trend analysis:

-The authors used an interrupted time series analysis utilising segmented Poisson regression models to compare trends in pre- and post-EBI procedure rates. Such model is based on a restrictive mean-variance equality assumption. How was the choice of using a Poisson model made over other options such as the Negative Binomial regression model? Did the authors performed for instance a Likelihood Ratio test?

- Did the authors examine the possibility of non-linear trends (as the presence of such trends might explain why no significant linear trends were identified)?

- The interpretation of the regression results regarding the ‘related’ procedures is a bit puzzling to me. When discussing their main findings, the authors explain that there is “limited evidence of a spillover effect on related or substitute procedures” (lines 318-322). However, looking at the results, we see that four out of ten related procedures show a significant trend post-intervention. Doesn’t this mean something (although no significant trends post-intervention were found for the EBI procedures)?

- In the study limitation, the authors mention that “there may be some delay between EBI publication and impact due to patients being placed on waiting lists for surgery pre-EBI but not receiving surgery until post-EBI, and for guidance to be adopted into practice”. This hypothesis might be directly tested in the model by changing the point in time considered (i.e., April 2019) to another time point considered plausible by the authors.

Geographic variation analysis:

- Although I understand that using the CSV is relevant since it is not affected either by extreme values or by the random variability within each region, why not also use other standard indicators of geographic disparities such as the extremal quotient (the ratio of the highest regional rate to the lowest), the interquartile ratio (the ratio of the 75th to the 25th percentile) or the coefficient of variation (the standard deviation divided by the mean)? These might help assess the robustness of the results.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Peter Bower

Reviewer #3: No

**********