PONE-D-23-21707An International Consensus on Effective, Inclusive, and Career-spanning Short-format Training in the Life Sciences and BeyondPLOS ONE

Dear Dr. Tractenberg,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================Notes from editor: Thank you for your submission to Plos One. The reviewers have suggested some minor changes to the paper before it is ready for publication (outlined below). Please pay particular attention to comments about data availability. Please also amend the abstract to briefly mention the delphi method and the participant demographics.  Best of luck with the resubmission.

For Lab, Study and Registered Report Protocols: These article types are not expected to include results but may include pilot data. 

==============================

Please submit your revised manuscript by Oct 19 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Cathryn Knight

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript: 

   "This material is based upon work supported by the National Science Foundation under DRL/EHR:2027025. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The authors wish to thank Rebecca Leshan and the staff of the CSHL Banbury Center for their guidance and facilitation of our convenings, and other participants in the conference including Charla Lambert."

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. 

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: 

  "This work was supported by the National Science Foundation under DRL/EHR: 2027025 to JWW & RET. Funders played no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript."

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

3. Thank you for stating the following in the Competing Interests section: 

   "B.B.: I have read the journal's policy and  have the following competing interest: I am an employee of DNAnexus.

G.S.M.: I have read the journal's policy and have the following competing interest: I work as an independent consultant who provides short-format training and professional development services. 

K.L.J.: I have read the journal's policy and have the following competing interest: I am an employee of The Carpentries.

E.A.B.: I have read the journal's policy and have the following competing interest: I am an employee of The Carpentries. 

N.C.: I have read the journal's policy and have the following competing interest: I am the owner of CureComms Advisors LLC and was compensated for participating in the meeting and providing strategic content guidance. 

No other authors declare competing interests."   

We note that one or more of the authors are employed by a commercial company: DNAnexus and The Carpentries.

a. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement. 

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement. 

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.  

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

4. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere. [The manuscript was submitted to biorxiv in march 2023. The submitted version is a more concise version (same results), and is formatted for PLOS one. Williams J, Tractenberg RE, Batut B, Becker E, Brown A, Burke M, et al. (2023, 15 March). Optimizing Short-format Training: an International Consensus on Effective, Inclusive, and Career-spanning Professional Development in the Life Sciences and Beyond. bioRxiv 2023.03.10.531570; doi: https://doi.org/10.1101/2023.03.10.531570] Please clarify whether this [conference proceeding or publication] was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

-mentioned above

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: About “statistical analysis”: I selected “I don’t know” because the authors use content analysis, but don’t discuss much about what they did.

About data availability: I selected “No”, though what they have done might be acceptable. The authors don’t provide the (de-identified) documents on which they did content analysis, at least as far as I can discern. It would not be possible to reproduce their work. Perhaps these “raw data” are not required to be shared. Defer to editor.

Other comments:

This is an interesting manuscript from which emerges worthwhile messages for improving learning throughout people’s scientific careers. I’m sure the discussions that went into the development of these principles and recommendations for short-format training were fascinating.

Lines 145-164. This is a lot of text to explain essentially what is in the figure. Comes across as repetitive.

Line 172-3: Subtle difference between the text and figure here. The figure is, I think, more accurate, by stating "formal knowledge about teaching and learning applies". The authors note in more than one place that the instructors of SFT often don't have pedagogical training (i.e. "formal knowledge about teaching and learning", and as an aside, neither do many, many instructors in FHE). So this formal knowledge is not necessarily a shared characteristic in the real world. But it ought to "apply" to both.

Suggestion: perhaps reword slightly to "...the fact that formal knowledge about teaching and learning ought to apply to both forms is perhaps the only shared characteristic"

Line 197-99 I think this sentence might be missing a word. "...adopt them"?

Line 234 (and lots of other places) The 20 items listed in S3 look like a list of questions for discussion – this is not my understanding of the word "vignette". I do recognize that the authors are reporting on meetings they had where they used “vignette”, but it seems like an odd choice.

Line 285-86. The context stated here that most of the participants are doing this work in countries outside where they are based seems important. Are these folks primarily involved in development work in developing countries (e.g. providing SFTs in topics that are not available locally)? If so, are these settings in which the participants work a relevant aspect of the principles generated? Does this bound the applicability of the principles? Are they particularly suited for this context?

Figure 2. The rear wheel is shown as a "cycle", but it is not clearly explained why these four parts connect in a cycle. In fact, the image shows the four items in a different order than the bullet points below the image. I'd recommend the authors figure out how to better connect the parts of the cycle, if it is one. As a reader, I think these four are more like a good list, though a case could be made for a cycle (e.g. what links "Inclusive" back to "Best Evidence"? Possibly collecting evidence about the inclusivity of a learning experience will contribute to "best evidence"...). The justification for a "cycle" is not communicated yet, in my opinion.

For the front wheel, I think the authors are implying an amplifying feedback loop, where if you reach more learners, you need to figure out how to offer more courses or make the courses bigger, or involve more instructors, and, if you want to continue to grow (reach more learners), you need to ensure you have systems in place to support that growth. Reaching more people with effective and inclusive SFTs, and making this possible by sharing good materials and coordinating systems are excellent goals. This wheel is easier to see as a cycle, but what's the end point of this amplifying feedback loop? Saturation of the learner pool?

In general, although I love bicycles, this image seems a bit forced.

Line 335: I believe this should reference "S5" instead of "S4.

S6 and Tables 2-5: I recommend that in S6 and in the tables that the authors number the 14 items, rather than use letters. The letters are confusing. E.g. in Table 2, the four recommendations spell FILM. This is a totally unnecessary distraction. Just number these 1-4, then start with 5 in the next grouping. I'm guessing there was much discussion about not wanting to number things which might imply a priority, but lettering simply muddles your list for a reader.

If you don't want to lead with what is currently Recommendation F, lead with L – L (catalysis), to me, is at the heart of this entire effort.

Table 2: FAIR is used here as an acronym, but these principles are not referenced until line 532.

Lines 352-353. When you say "individuals", do you mean individuals working essentially alone? Or do you mean individual instructors who are part of a larger group of SFT providers who collect evidence from their courses? This sentence implies that funders are most likely to listen to individuals working alone, which might not be what you mean. For example, you note below that it may be difficult for individuals to convince their colleagues to share goals and standards. Funders may be a step harder for individuals.

Reviewer #2: The manuscript by Williams et al. provides a report on the outputs of a project which aimed to improve the delivery and impact of Short Format Training (SFT) workshops. The manuscript describes the process that the project went through to develop the final recommendations that were produced, and then explores the potential implementation of these recommendations across different "user groups" (i.e., individual trainers, communities, institutions).

The manuscript is well-written, and does a good job of describing the current state of SFT-based training initiatives (including coverage of current challenges), before moving on to explain the processes that the authors went through to generate the proposed stet of recommendations.

Of note, the authors' highlighting of the relatively low-impact of current SFT (i.e., in terms of how much long-term utility attendees actually gain from this type of training) is extremely valuable (although also rather sobering), particularly in the light of how much has been invested in developing and delivering this style of training in recent years. Rather than dismissing this approach to content delivery, the authors are proposing some useful and readily-implementable recommendations to help improve SFT impact.

Given the popularity and ubiquity of Short Format Training across a wide range of disciplines, this thoughtful and well-structured manuscript is a timely and highly relevant submission.

Specific comments:

1. Line 235 - given the breadth of the potential target audience (i.e., almost anyone involved in developing, delivering and/or coordinating SFT) it might be worth briefly defining "phenomenography", which will not necessarily be something that non-qualitative researchers will be familiar with.

2. Lines 289-290: it would be useful to see the numbers involved from each country (e.g., "Canada (x), France (y)" etc).

3. Lines 296-298: Presumably the "average" used here was the mean? (this should be specified). Given the highly skewed nature of these figures (e.g., 0-7000 for "new instructors trained"), the median would be a more appropriate measure of the "average". Also, based on the journal's requirements for data availability, the raw data behind these averages needs to be made available (this could easily be done as a small spreadsheet in the supplementary information).

4. Line 302: "Catalytic learning" should be defined somewhere in the document. I realise that it is loosely defined in Figure 2, but I think a more foraml defintion in the text (or within a specific box - see next comment), along with a relevant citation, would be helpful.

5. Line 320: as per my comment above about "phenomenography", I think it would be worth briefly defining the Delphi technique here. I wonder if having a "Definitions Box" might be useful, just to help readers who are not coming from an educational research background?

6. Line 342: "i.e.," should be written as "that is" if it appears in the main text (rather than within parentheses).

7. Line 363: it isn't obvious what is meant by "professionalizing" in this context - this should be made clear. It could mean "utilsing a formal paid training system" (which I'm assuming is the intent), but it could also be interpreted as "acting in a more professional manner".

8. Line 367: the term "formal organizations" is used here without definition. The examples associated with "organizations and institutions" in the next paragraph almost look like a definition for "formal organizations" - could this definition be moved up so that it is clear what "formal organizations" is referring to?

9. Lines 431-432: reference 50 is talking specifically about ableism, however the authors' mention of "creation of inclusive environments" would generally be taken to refer to more than persons with disabilities (e.g., minorities, neurodiverse individuals, indigenous communities etc). I think the wording in this part of the paragraph needs to be modified to make sure that "inclusive environments" is an broad as possible, and then rephrase the ableist-specific material as one example of how to co-design for inclusivity.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

An International Consensus on Effective, Inclusive, and Career-spanning Short-format Training in the Life Sciences and Beyond

PONE-D-23-21707R1

Dear Dr. Tractenberg,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Cathryn Knight

Academic Editor

PLOS ONE