PONE-D-20-40753

Using pooled analysis of pretested items from the Progress Test to monitor performance of first-year medical students exposed to different curriculum designs: an exploratory study

PLOS ONE

Dear Dr. Hamamoto Filho,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

I highly recommend you address the following.

1. Clarify the use of the assessment as a benchmark assessment rather than a progress test and the overall goal of the study. 2. Clarify more specifically the assessment question objectives and how these relate to the overall heterogeneity. 3. Describe whether any other measures were used to compare the curriculum. As was indicated in the discussion, a single measure can yield low reliability.

Please ensure that your decision is justified on PLOS ONE’s publication criteria and not, for example, on novelty or perceived impact.

For Lab, Study and Registered Report Protocols: These article types are not expected to include results but may include pilot data. 

==============================

Please submit your revised manuscript by Jul 31 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Amy Prunuske

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Thank you for your submissions. The reviewers valued your work and have provided a number of suggestions to improve the manuscript.

I highly recommend you clarify the following.

1. Clarify the use of the assessment as a benchmark assessment rather than a progress test and the overall goal of the study.

2. Please clarify more specifically the assessment question objectives and how these relate to the overall heterogeneity.

3. Describe whether any other measures were used to compare the curriculum. As was indicated in the discussion, a single measure can yield low reliability.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1.Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section. 

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper can make a worthwhile contribution to medical education literature but in it’s current form, it has several flaws:

1. My understanding of progress testing is that it involves repeated testing of the same cohort. This study seems to report on testing of subsequent year 1 cohorts over several years and therefore better fits the definition of benchmarking assessment rather than progress testing per se.

2. The introduction largely focuses on progress testing as an assessment methodology but the discussion and the value of the study seem to be more about the use of pooled analysis for comparison of performance between cohorts and curricula. I wonder if in fact, the focus of this paper needs to shift away from progress testing and towards the statistical methods which can be applied to assess the effectiveness of curricula change.

3. The re-use of questions in subsequent years can be problematic in that students recall and share questions and they become available for subsequent cohorts. Analysis of the performance of repeated items across years should be conducted to identify whether this has occurred in this case. Further, the authors could provide more information about the context of the different medical schools to offer readers a clearer impression of how sharing may or may not occur between them.

4. Data was collected using the interinstitutional progress test from 2005 – why is only data from 2013 used here?

5. Apart from performance in the assessment, did the cohorts differ in other ways? Are demographic data available to describe the cohorts?

Reviewer #2: This manuscript provides a sound example of how pooled analyses of a standardized examination, the Progress Test, can be used to retrospectively evaluate impact of curricular change in medical education. While the findings are limited due to the smaller sample size and high degree of heterogeneity, the authors have done an excellent job of demonstrating how this approach was used, interpreted, and the strengths / limitations of it compared to a single-point comparison analysis. Furthermore, the use of a smaller sample size makes this work relatable to many and provides a template for how others might proceed under similar constraints. Though the work itself isn't ground-breaking, it does provide a very accessible (well written and clearly described) blueprint for how other investigators might use a similar approach to evaluate programmatic change at their own institutions. For this reason, this manuscript offers substantial practical application value to its readers.

Reviewer #3: SUMMARY OF THE RESEARCH

In this manuscript, the authors describe a method of pooled item analysis to evaluate learning in several disciplines between two curricula. Overall the writing was clear, making it easy to follow the results and conclusions of the study. The “old” curriculum consisted of three phases (Basic Science M1-M2, Clinical Science M3-M4, Clerkship M5-M6) where the basic sciences were presented in subject-based courses. The “new” curriculum is divided into two phases (Preclinical M1-M3, Clerkship M4-M6) where the basic sciences are taught within systems-based units. The authors used a subset of 63 questions from the previously published Progress Test. All questions were used on the Progress Test at some point between 2013-2018, and student performance on those questions at those times was used to assess learning in the “old” curriculum. Students in 2019 who had begun the “new” curriculum answered the same 63 questions and their performance was compared to that in the “old” curriculum. The authors compared performance across curricula for each question, but also used a pooled analysis to evaluate differences in students understanding within a content area (ex: basic sciences, public health, surgery). Results showed improved performance on basic sciences and pubic health questions with the new curriculum compared to the old curriculum. I2 analysis showed a high degree of heterogeneity. Study was done well given the data available, but would have liked more description of specific changes to the curriculum that led to improvements, though that was beyond the scope of the study. Overall, it seems that pooled analysis of items proved a useful analytical tool to compare student performance across curricula.

Recommendation: Accept with minor revisions.

DISCUSSION OF SPECIFIC AREAS FOR IMPROVEMENT

Introduction

1. Need reference for content in lines 48-50.

2. Lines 55-57 – please clarify what is meant by “easy to obtain”. It seems the authors mean it is not time efficient to evaluate long-term metrics like clinical performance and assessments of new curricula need to be done in real time to evaluate current learning.

3. Lines 60-65 – please describe the types of questions posed in the Progress Test – are they clinical vignettes? Are they assessing low-level knowledge?

4. Line 61 - what is meant by “repeated measures”? Do students take the test each year of their training)?

Materials & Methods

1. Please describe the typical path of medical training in Brazil. Do all students spend 6 years in medical school? Is the format of your new curriculum the norm there?

2. Study setting and participants

a. Describe any exclusion criteria.

b. Describe changes to the content of the curriculum. The authors mention the social sciences disciplines were taught differently, but does this equate to inclusion of more content or just redistribution of content?

c. Why were first year students used for this study when the first phase lasts at least two years for both curricula?

d. Was there a difference in when certain content was covered in the old and new curriculum relative to when students took this test?

3. Progress Test

a. Line 110 – clarify if this means each student takes the test each year throughout their training

b. The reader needs more context here, please describe types of questions included in this test

4. Line 131 – describe interpretation of I2 analysis

Results

1. Why were there no questions selected from 2015?

2. It seems that including questions from so many different years increases the noise of your sample, why not select questions from a smaller subset of years?

3. Have the authors evaluated student performance on these questions in the old vs. new curriculum when students were M2s?

4. Line 151-153 – this needs clarification. When points appear to the right or left of what? 1?

5. Line 162-164 – this is interesting, what was the topic and what differed in the new curriculum that improved understanding of this topic so dramatically?

6. Line 175 – place the p value inside the parenthesis to match formatting of the other paragraphs

7. Issue with formatting for lines 181-186 (seems related to the editorial software)

Discussion

1. Line 198-200 – is this what would be expected given changes to the curriculum?

2. Line 220-222 – it is concerning that the “control” group consists of multiple cohorts while the “new” group is all within a single cohort. It seems this would introduce a decent amount of noise into the data.

3. It is unclear if the goal of the study is to demonstrate utility of pooled analysis as a method or demonstrate that their new curriculum improved student understanding of basic sciences and public health.

ADDITIONAL POINTS

The comparison of student performance in the old vs. new curriculum is valuable for quality assurance at their institution. It is unclear how novel a finding this is. One would hope that after a major curriculum redesign, improved learning could be shown, but it is unclear where to attribute those improvements. What did they do to improve learning in the basic sciences and public health? Do they have recommendations for other educators? It seems the major conclusion the authors are trying to show is that pooled analysis of items is a useful method. However, the data seems very noisy and difficult to draw conclusions from.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Exploring pooled analysis of pretested items to monitor the performance of medical students exposed to different curriculum designs

PONE-D-20-40753R1

Dear Dr. Hamamoto Filho,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Amy Prunuske

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #4: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #4: The authors present a pooled item analysis to evaluate learning in several disciplines between two curricula. Considering the available data - which are weak from the point of view of the demographic information of the respondents and in terms of sample size - the authors did a good job and the analysis provides useful information.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #4: No