Peer Review History

Original SubmissionOctober 7, 2019
Decision Letter - Oathokwa Nkomazana, Editor

PONE-D-19-27962

Factors Predicting Students’ Performance in the Final Pediatrics OSCE

PLOS ONE

Dear Dr. Al Rushood,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Jul 03 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Oathokwa Nkomazana, MD MSC PhD

Academic Editor

PLOS ONE

Additional Editor Comments:

Thank you for an interesting article.

Please address the comments by the reviewer. In addition please address the following:

1. In the Methods section, it will be helpful to have a section titled: Study setting where you describe the setting of the medical school, where the clinical training is done, focusing on paediatrics; how the training is done; How many paediatric rotations do the students do in the course of training?Are the PBL sessions paper based or are they based on real patients? What is entailed in clinical teaching``/

How are the examination quality assured? You only talk to external examiners in the final OSCE. Is there blue printing of the examination? Is there moderation? Is it internal or are there aspects of external moderation? What does it mean that "Examiners are provided protected time just before exams?" what are the examiners briefed on?

In the data collection, make it clear if you followed the same cohort of students over three years, which is what I think you are reporting, but it's not clear. You say that you enrol 100 students a year but do not explain why you do not have 300 students in your study. Who have you excluded?

In the results section, you say that students who did their rotation last performed better than those who did it second, but not not better that those who did it first. Hw do you explain the difference? Could this be attributed to the differences in the assessments?

In the discussion,, the 7th paragraph that talks to differences in performance based on the timing of the rotation, the inferences are not very clear and it's not clear how this is supported by the reported findings. You conclude the paragraph by applauding the recall of information,; in medical education, however, recall is not considered a very high value but rather information literacy and life ling learning capacity as the half life of medical information is very short.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service.  

Whilst you may use any professional scientific editing service of your choice, PLOS has partnered with both American Journal Experts (AJE) and Editage to provide discounted services to PLOS authors. Both organizations have experience helping authors meet PLOS guidelines and can provide language editing, translation, manuscript formatting, and figure formatting to ensure your manuscript meets our submission guidelines. To take advantage of our partnership with AJE, visit the AJE website (http://learn.aje.com/plos/) for a 15% discount off AJE services. To take advantage of our partnership with Editage, visit the Editage website (www.editage.com) and enter referral code PLOSEDIT for a 15% discount off Editage services.  If the PLOS editorial team finds any language issues in text that either AJE or Editage has edited, the service provider will re-edit the text for free.

Upon resubmission, please provide the following:

  • The name of the colleague or the details of the professional service that edited your manuscript
  • A copy of your manuscript showing your changes by either highlighting them or using track changes (uploaded as a *supporting information* file)
  • A clean copy of the edited manuscript (uploaded as the new *manuscript* file)

3. Thank you for stating in the manuscript Methods section:

'Ethical approval was obtained from The Health Sciences Ethical Committee. The use of students’ grades was approved by the Vice-Dean Academic Office at the Faculty of Medicine. Confidentiality of identity of students included was secured.' 

a. Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.  

b. Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research

4. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

5. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

6. Please include your tables as part of your main manuscript and remove the individual files.

Please note that supplementary tables should be uploaded as separate "supporting information" files.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I read the manuscript with great interest, I suggest the following comments to make it publishable

• There is need to address other factors that may affect the final OSCE like gender, day of the exam (the authors mentioned that they have around 30 students /group and there were 9 stations run in 2 cycles, this mean that the exam either repeated for three days or in three different place), studying these factors will add to the quality of paper.

• Was the in-training OSCE formative without marks, did you provide any feedback for student in the in-training OSCE

• The term error bar is mentioned in the method section but it is not shown in the figure.

• In result section (‘End of block OSCE’ and ‘End of block written’ showed some improvement, especially in the year 2014-15 compared to 2013-14 and 2015-16 (p=0.016 & p=0.031) respectively.) could you clarify this sentence more, what it means and refer to table number. The word improvement here is not perfect with the meaning. (improvement mean that student score improved in this test in comparison to previous test)

• In discussion section, the authors should shed light on the low r value between written and final OSCE, usually the OSCE should be a test of clinical competence and not knowledge try to discuss this point when comparing final OSCE and written test, could be the good correlation between written test and OSCE due to more knowledge concentrated OSCE stations rather than performance OSCE stations.

• To mention reliability of a test there is need for reliability index.

• In conclusion part the first sentence should be removed as it was not detected in the results.

• Try to use some new references that addressed similar problem.

Alkhateeb, N. E., Al-Dabbagh, A., Ibrahim, M., & Al-Tawil, N. G. (2019). Effect of a Formative Objective Structured Clinical Examination on the Clinical Performance of Undergraduate Medical Students in a Summative Examination: A Randomized Controlled Trial. Indian pediatrics, 56(9), 745–748.

English editing is advisable

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Revision 1

1. In the Methods section, it will be helpful to have a section titled: Study setting where you describe the setting of the medical school, where the clinical training is done, focusing on paediatrics; how the training is done; How many paediatric rotations do the students do in the course of training?Are the PBL sessions paper based or are they based on real patients? What is entailed in clinical teaching``/

Our response: A section entitled “Study Setting” was added with the required details (lines 77-87 pages: 4,5).

2. How are the examination quality assured? You only talk to external examiners in the final OSCE. Is there blue printing of the examination? Is there moderation? Is it internal or are there aspects of external moderation? What does it mean that “Examiners are provided protected time just before exams?” what are the examiners briefed on?

Our response:

The pediatric exams are prepared by the staff in the department. Each member contributes in each exam. There is a blue print for each exam that it is followed. A departmental assessment committee, including academic staff in the department, evaluate each exam thoroughly. Then the exam is sent to the Faculty of Medicine exam committee for further evaluation and approval. Exam moderation is done internally. External examiners are involved as examiners in the final OSCE, in addition, they evaluate the written final exam (line 91, page 5).

“Examiners are provided protected time just before exams” Our response: before the start of the final OSCE, Examiners are assigned time to check on patients physical signs in the different exam stations and agree on what students should demonstrate (lines 96-98, page 5).

what are the examiners briefed on? Our response: Few days prior to the final exam, the department of pediatrics, representing by the head of the department and the chair of the assessment committee and other members, hold an orientation session to the external examiners to explain the guidelines and the lay out of the exam, what standards are expected from the students so to define the pass/fail marks. The grading sheets and the global assessment sheets are discussed. (lines 105-108, page 6).

3. In the data collection, make it clear if you followed the same cohort of students over three years, which is what I think you are reporting, but it's not clear. You say that you enroll 100 students a year but do not explain why you do not have 300 students in your study. Who have you excluded?

Our response: We did not follow the same cohort for 3 years. We included the students rotated in pediatrics in the following academic years 2013-14, 2014-15 and 2015-16.

The total number of students involved in the study was 286 students because the faculty admits, on average, 100 students every year, yet 14 students had early drop-out for different reasons (lines 67-69, page 4).

4. In the results section, you say that students who did their rotation last performed better than those who did it second, but not better that those who did it first. How do you explain the difference? Could this be attributed to the differences in the assessments?

Our response: There are no differences in assessments. We expected that students’ performance in the final OSCE is best for those who did their pediatrics rotation towards the end of the year and closer to the final exam. However, this was not consistent, especially when compared with those who did their rotation in the beginning of the academic year. Students, doing their pediatric rotation second (ie in the middle of the academic year), perform less in the final OSCE. An explanation to this would be the fact, the rotation in the middle of the year is interrupted by many holidays (New year, Christmas and our National days). The students might be distracted. Of note, this difference was only statistically significant when we compared the whole study population, it was not demonstrated among students in each year, or when compared last to beginning or middle to beginning of the year. (lines 225-228 p 12)

5. In the discussion, the 7th paragraph that talks to differences in performance based on the timing of the rotation, the inferences are not very clear and it's not clear how this is supported by the reported findings.

Our response: We reported no significant differences in the final OSCE performance between the 3 groups each year. The only significant difference was when we looked at the whole study sample, there was a slight improvement in the final OSCE performance between the students rotating last compared to those rotating in the middle of the year (Means= 79.17 vs 77.19, P=0.028) however, this was not true comparing the third rotation to the first, or second to first. Therefore, there might be no effect of the timing of the rotation on the final OSCE performance. Therefore, dividing students to do different rotations throughout the year, according to the capacity of the training centers, staff availabilities, applied curriculum, should not affect their performance in the final exams (lines 220-228, page12).

6. You conclude the paragraph by applauding the recall of information; in medical education, however, recall is not considered a very high value but rather information literacy and life-long learning capacity as the half-life of medical information is very short.

Our response:

We agree. Recall of information was not meant. It is the life-long learning capacity and the ability to perform the skills learned fluently and efficiently it was clarified in line 232 page12.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

The format was modified accordingly

2. We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service.

The manuscript was edited by AJE

3. Thank you for stating in the manuscript Methods section:

'Ethical approval was obtained from The Health Sciences Ethical Committee. The use of students’ grades was approved by the Vice-Dean Academic Office at the Faculty of Medicine. Confidentiality of identity of students included was secured.'

a. Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

b. Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research

Ethical approval was obtained from the Health Sciences Center Ethical Committee, Faculty of Medicine, Kuwait University. The Board members are the Vice Dean of Research at the Faculty of Medicine as the head of the committee and five other members who are full professors in different specialties.

4. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016.

The ORCID ID of the corresponding author was updated in the Editorial Manager System

ORCID ID: 0000-0001-6148-1707.

5. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or

ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

The data will be available and shared. To be able to share the data, we need to apply for a permission from the administration at the Faculty of Medicine, which is a lengthy procedure and because of the COVID-19 pandemic and the lockdown situation in the country, we will not be able to have the permission in timely manner. We will be able to share the data when the situation is back to normal in the country.

6. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables should be uploaded as separate "supporting information" files.

The tables were included as part of the main manuscript file. The supplementary files were removed.

Response to reviewer #1 comments:

Thank you for your valuable comments. We have responded and made necessary changes a follows:

5. Review Comments to the Author

Reviewer #1: I read the manuscript with great interest, I suggest the following comments to make it publishable.

• There is need to address other factors that may affect the final OSCE like gender, day of the exam (the authors mentioned that they have around 30 students /group and there were 9 stations run in 2 cycles, this mean that the exam either repeated for three days or in three different place), studying these factors will add to the quality of paper.

Our response: We agree. Gender and the day of the examination, if it is conducted in more than one day, are important factors that might affect the exam performance. In our medical school, the end of block OSCE is conducted in one day in 2 hospitals. The final OSCE is run over 2 day in 4 hospitals per day (lines 94-96 in page 5). A study on the effect of students’ demographic information, including gender, age, type of housing, order among siblings and others is being carried out.

• Was the in-training OSCE formative without marks, did you provide any feedback for student in the in-training OSCE?

Our response: The in-training OSCE is graded. Feedback is provided to the students who failed the exam station only. During their 3 months of clerkship, the students perform Mini-Cex, where immediate feedback is given (lines 110-112, page 6)

• The term error bar is mentioned in the method section but it is not shown in the figure.

Our response: This was a mistake. There was an error bar, but we opt to remove it in the final version of the manuscript as it does not add much to data analysis. This sentence should have been removed from the text prior to submission. Thank you for pointing it out.

(Lines 125-126, page 7) was removed.

• In result section (‘End of block OSCE’ and ‘End of block written’ showed some improvement, especially in the year 2014-15 compared to 2013-14 and 2015-16 (p=0.016 & p=0.031) respectively.) could you clarify this sentence more, what it means and refer to table number. The word improvement here is not perfect with the meaning. (improvement mean that student score improved in this test in comparison to previous test)

Our response: The word improvement is misleading, I agree, as it implies comparing the performance of the same students over time. In this paragraph, we refer to table 1, showing the stability of our exam grades over the years, which make the study data comparable and hence, conclusions can be withdrawn. There are no significant differences between the grades of different exam components over the years. However, we think that it is confusing, so it was removed from the text ( lines 144-146, page 8).

• In discussion section, the authors should shed light on the low r value between written and final OSCE, usually the OSCE should be a test of clinical competence and not knowledge try to discuss this point when comparing final OSCE and written test, could be the good correlation between written test and OSCE due to more knowledge concentrated OSCE stations rather than performance OSCE stations.

Our response: We agree, OSCE should measure clinical competence. Many studies reported correlations between written exam performance and OSCE (3,8,18-20 in the references). As we have real patients in the OSCE run in our institution, we think that students should have a good knowledge base in order to perform well. Clinical findings should be sought as the student progresses with the patient encounter. They need to give explanations and answer few questions at the end of the station. The OSCE is conducted to ensure mastering the clinical skills. More clarification to the paragraph, with new references, was added (lines 203-219, pages 11,12).

• To mention reliability of a test there is need for reliability index.

Our response: True. The sentence was removed from the text as it might be confusing (line 205, page 11).

• In conclusion part the first sentence should be removed as it was not detected in the results.

Our response: Agree. The sentence was removed (lines 250,251, Page 13).

• Try to use some new references that addressed similar problem.

Alkhateeb, N. E., Al-Dabbagh, A., Ibrahim, M., & Al-Tawil, N. G. (2019). Effect of a Formative Objective Structured Clinical Examination on the Clinical Performance of Undergraduate Medical Students in a Summative Examination: A Randomized Controlled Trial. Indian pediatrics, 56(9), 745–748.

Our response: Thank you for the reference. This article and other 2 new references were added. References 14, 19 and 20.

English editing is advisable.

Our response: The manuscript was edited by AJE

Attachments
Attachment
Submitted filename: response to Reviewers.docx
Decision Letter - Oathokwa Nkomazana, Editor

Factors Predicting Students’ Performance in the Final Pediatrics OSCE

PONE-D-19-27962R1

Dear Dr.Maysoun Al Rushood ,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Oathokwa Nkomazana, MD MSC PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Formally Accepted
Acceptance Letter - Oathokwa Nkomazana, Editor

PONE-D-19-27962R1

Factors Predicting Students’ Performance in the Final Pediatrics OSCE

Dear Dr. Al Rushood:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Oathokwa Nkomazana

Academic Editor

PLOS ONE

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .