PONE-D-25-13149Widespread misidentification of SEM instruments in the peer-reviewed materials science and engineering literaturePLOS ONE

Dear Dr. Amaral,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by May 30 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Zeheng Wang

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

We thank Dr. Justin Notestein for his helpful feedback, Dr. Thomas Stoeger for assisting us with accessing OpenAlex data and Helio Tejedor Navarro for technical support. R.A.K.R. was supported in part by the National Institutes of Health Training Grant [T32GM008449] through Northwestern University’s Biotechnology Training Program. R.A.K.R. gratefully acknowledges funding from the Dr. John N. Nicholson fellowship from Northwestern University; Moderna Inc., Identifying bias and improving reproducibility in RNA-seq computational pipelines. J.M. gratefully acknowledges funding from the Weinberg College Baker Program in Undergraduate Research. S.S.H. gratefully acknowledges support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University. L.A.N.A. gratefully acknowledge funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies [1956338].

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

R.A.K.R. was supported in part by the National Institutes of Health Training Grant [T32GM008449] through Northwestern University's Biotechnology Training Program. R.A.K.R. gratefully acknowledges funding from the Dr. John N. Nicholson fellowship from Northwestern University; Moderna Inc., Identifying bias and improving reproducibility in RNA-seq computational pipelines. J.M. gratefully acknowledges funding from the  Weinberg College Baker Program in Undergraduate Research. S.S.H. gratefully acknowledges support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University. L.A.N.A. gratefully acknowledge funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies [1956338].

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

3. We note that you have indicated that there are restrictions to data sharing for this study. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

Before we proceed with your manuscript, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., a Research Ethics Committee or Institutional Review Board, etc.). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see

https://journals.plos.org/plosone/s/recommended-repositories. You also have the option of uploading the data as Supporting Information files, but we would recommend depositing data directly to a data repository if possible.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. Please provide a complete Data Availability Statement in the submission form, ensuring you include all necessary access information or a reason for why you are unable to make your data freely accessible. If your research concerns only data provided within your submission, please write "All data are in the manuscript and/or supporting information files" as your Data Availability Statement.

5. Please amend your list of authors on the manuscript to ensure that each author is linked to an affiliation. Authors’ affiliations should reflect the institution where the work was done (if authors moved subsequently, you can also list the new affiliation stating “current affiliation:….” as necessary).

6. Please amend either the abstract on the online submission form (via Edit Submission) or the abstract in the manuscript so that they are identical.

7. Please remove your figures from within your manuscript file, leaving only the individual TIFF/EPS image files, uploaded separately. These will be automatically included in the reviewers’ PDF.

8. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

9. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments :

This manuscript is exceptionally well-written, highly original, and thought-provoking. It attracted an unexpectedly large number of reviewers, all of whom provided very positive evaluations. We have therefore decided to accept the manuscript, pending minor revisions based on the reviewers' suggestions.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

Reviewer #5: Yes

Reviewer #6: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The paper explores the reliability of materials science and engineering (MSE) research by analyzing the misidentification of scanning electron microscope (SEM) instruments in published articles. A semi-automated approach was developed to scan figures for SEM metadata and compare it with the instrument identified in the text. The analysis, which covered over a million articles published since 2010, revealed that 21.2% contained mismatched SEM metadata, while 24.7% did not report some of the instruments used. I appreciate the great efforts made by the authors.

Through their analysis, the authors find the potential risks posed to MSE literature due to SEM misidentification, suggesting that these discrepancies may point to broader concerns such as systematic fraud and the presence of paper mills. Furthermore, articles with SEM misidentifications were more frequently associated with prior observations of improprieties recorded on post-publication peer review platforms. The authros also examined trends in authorship and institutional affiliations, noting that certain regions and institutions had a disproportionately high occurrence of SEM misidentification.

Overall, the merits of the paper is sound and the paper is worth to be published. I believe this paper is crucial in raising awareness about the issue of SEM misidentification and its broader implications for research integrity. By bringing this problem to light, I hope to encourage researchers, reviewers, and editors to place greater emphasis on accurately reporting SEM instruments. Beyond just addressing SEM misidentification, I see this issue as a gateway to uncovering larger systemic problems in scientific publishing, such as research misconduct and the operation of paper mills. This work could inspire further investigations into the reliability of published studies and lead to the development of better tools for identifying problematic research. The semi-automated detection method proposed here has the potential to be implemented by journals and publishers, strengthening peer review processes and improving overall research quality.

Reviewer #2: This manuscript presents a systematic investigation into the misidentification of scanning electron microscope (SEM) instruments in the materials science and engineering (MSE) literature. The authors employ a semi-automated pipeline to analyze over a million articles, identifying significant discrepancies between reported SEM metadata in text and images. The study highlights critical issues in scientific reporting and suggests broader implications for research integrity. The work is well-structured, methodologically rigorous, and addresses an important gap in the literature. However, some areas require clarification and further discussion.

Major comments

The high false negative rate (71.1% of PubPeer-identified cases were missed) suggests limitations in the OCR-based approach. The authors should discuss potential improvements, such as alternative OCR tools or manual supplementation, to increase sensitivity.

The reliance on high-resolution images for metadata extraction may bias the results toward recent publications or specific publishers. The authors should further discuss how this bias might affect the generalizability of their findings.

The exclusion of supplementary materials from the analysis is a notable limitation. The authors should justify this decision or address how supplementary data might alter their conclusions.

While the study correlates SEM misidentification with other errors (e.g., Tauc plot mistakes), it does not establish causality. The authors should clarify whether these errors are symptomatic of broader issues (e.g., paper mills) or isolated oversights.

Minor Comments

Some figures (e.g., Figure 2) are dense and could be simplified for better readability. Consider breaking complex figures into subpanels or providing clearer labels.

The term "problematic articles" is used frequently but could be more precisely defined (e.g., "articles with SEM misidentification").

Ensure all references are up-to-date, particularly for tools like Tesseract and OpenAlex, which may have newer versions or related studies.

Overall Recommendation

This manuscript makes a significant contribution to the literature on research integrity in MSE. The methodological approach is robust, and the findings are compelling. However, addressing the limitations and incorporating the suggested improvements would strengthen the manuscript further.

Reviewer #3: The paper by Reese A.K. Anderson, Jeonghyun Moon, Spencer S. Hong, and Luis A. Nunes Amaral, proposed for publication in PLOS ONE, addresses a critical issue in quantitative science: the reproducibility of results. This problem is well-documented in biomedicine and social sciences, where large-scale replication studies have frequently failed to confirm reported findings. However, it has not been extensively studied in the physical-chemical sciences and engineering, despite high-profile retractions in areas such as room-temperature superconductivity, organic electronic devices, and Majorana fermions. With the exponential increase in scientific publications, there is growing interest in developing automated or semi-automated methods to identify erroneous or counterfactual claims in research articles. Publishers are beginning to integrate these methods into their protocols to detect unreliable content in unpublished manuscripts and combat automated mass production of scientific papers.

To address this issue, the authors focus on the use of scanning electron microscopy (SEM) in materials science and engineering (MSE), a widely adopted technique for characterizing materials and their properties. SEM images typically include metadata banners providing details such as the manufacturer, instrument model, and imaging parameters. The study was inspired by reports on the post-publication peer review platform PubPeer, where users observed that metadata from SEM images in many articles did not correspond to the information provided in the text. These discrepancies were often accompanied by other inconsistencies. To investigate this issue, the authors developed a semi-automated method to detect SEM misidentification and applied it to over 1,000,000 articles published in 50 journals from four different publishers, with impact factors ranging from 2.1 to 24.2.

Due to publisher restrictions on large-scale downloads, the study primarily analyzes articles from Elsevier, which has a more flexible data-mining policy. The developed method proved robust against false positives through manual verification, allowing the authors to identify patterns of misattributed SEM brandmarks and models. One example includes systematic errors in the calculation of the optical band gap using the Tauc plot method from electrochemical measurements. The study found that articles including both SEM images and Tauc plot were more likely to miscalculate the band gap if the SEM metadata was misreported. Additionally, recurring trends were identified based on the countries of origin of articles with incorrect SEM identification.

The paper is well-written, with a clear discussion of the methodology and its limitations, and addresses a topic of broad interest to the scientific community. Thus, I consider it suitable for publication in PLOS ONE. However, I would appreciate further clarification on the following points:

1. Selection of initial dataset: The authors state that they begin with an initial dataset of 1,067,108 articles. However, as indicated in Figure 3, only 16% of these contain SEM-related terms in figure captions. This suggests that the majority do not use SEM as a characterization method or include SEM images in the main text or supplementary materials. Merely mentioning "scanning microscopy" does not necessarily imply the presence of SEM images. If the initial dataset is meant to highlight the technique's relevance, this should be made clearer. Otherwise, considering the full dataset skews the perceived percentage of misreporting.

2. Definition of "resolution": The term "resolution" in the article is somewhat ambiguous, particularly in relation to the presence or absence of metadata banners in SEM images. Typically, resolution refers to the number of pixels in height and width. However, the authors focus solely on image height, noting that when it is too low, experimental metadata cannot be recovered. Often, when height is reduced, the banner is intentionally cropped—likely because SEM software does not allow users to adjust font size, making the metadata unreadable in publication formats. If this is the case, further clarification is needed on how cropped images are categorized.

3. Citation trends of misidentified SEM articles: According to the "Systemic Characteristics" section, articles with misidentified SEM metadata receive more citations on average than those with correct information. This is an intriguing observation. Could this be due to the prevalence of open-access journals, where articles with poor experimental details might be more frequently cited? Alternatively, could this indicate another form of misconduct related to citation practices? I would be interested in the authors' perspective on this correlation.

4. Impact factor and SEM reporting: The study finds that lower-impact-factor journals are more likely to contain incorrect or incomplete information, which aligns with expectations. However, I was surprised to see in Table S1 that for Materials Today—the highest-impact journal analyzed—only 2 out of 474 articles containing SEM images included brandmarks, and only one of these was correctly reported. My assumption (and hope) is that the remaining 472 articles included correct SEM information elsewhere, even if it was not retrievable from metadata. The authors discuss the method’s limitations, but in their conclusions, they suggest that these limitations inherently lead to an increased occurrence of misreported SEM data. This statement should be made with more caution.

Overall, the study presents a valuable contribution to the discussion of research integrity and reproducibility in materials science. Addressing the points raised above would further strengthen the clarity and impact of the findings. I also suggest to review the following:

Title. The use of acronyms in the titles should be avoided even if

Figure 1. There is a typo in the text (extant).

Figure 6. In contrast to previous caption, here after any entry a, b, a minuscule letter is used. Revise the uniformity of notation all over the manuscript.

Figure S4. The first phrase of the caption is not clear (no verb).

Reference. The reference style is not uniform across all the entries, and for many of the items the information is not complete (as it happens in mass-produced papers…). A full stop should be included at the end of each item entry.

Reviewer #4: The manuscript presents the differences between the make and model of the SEM mentioned in the text and the data obtained from the image. The study made an investigation of a large number of papers on this subject by a semi-automated approach. Although the results of the study are valuable, I have concerns that some cases require evidence beyond allegations and that these cases may have legal consequences. Therefore, I would like to inform you that the allegations in Figure S5 have the potential to lead to legal proceedings.

-In Figure 4b, the samples labeled as ‘Mfr’, it is not stated but it is understood that the abbreviation of the manufacturer. Also, in the whole text it was mentioned as both ‘make’ or ‘manufacturer’ and the abbreviation is not used. It would be better to use one of them in order to provide the uniformity throughout the text.

-In heading of fig 5. b) ‘For 30 fo the 43’ please correct it.

-In line 240 ‘The five most common authors of articles with misidentified SEMs authored between 13 and 42 problematic articles, respectively.’ What does the respectively correspond?

-In line 289 ‘The presence of repeated authorship (Fig. 7d)’ There is no figure 7d.

- The resolution of the graphs is so low.

Reviewer #5: This is a solid and detailed work about a semi-automatic pipeline to identify SEM instrument information issues in published research papers. The authors focused on the materials science field and examined more than 1 million published works from 50 journals with a diversified set of impact factors. The authors made several significant claims supported by clearly identified numbers. I especially appreciate the: (1) well-prepared figures, (2) well-organized sections, and (3) comprehensive supporting figures and tables.

I support the publication of this work, after some minor revisions are made, including:

1. In Figure 2, image height is used as the indicator of the image resolution. I don’t think this is strictly correct, since usually the image resolution is represented by both height and width pixel counts. I think it would be better if the y axis can be updated to a unit like pixels per inch to indicate the image resolution.

2. Line 162: the claim about the underestimation of global misidentification rate is not necessarily true. The authors should (1) provide an estimation of the misidentification rate of the OCR-unreadable articles and (2) compare it to the OCR-readable misidentification rate. Only then can the authors make an estimated claim of the misidentification rate shift.

3. Paragraph starting from line 193: the authors found the proposed pipeline has a high false negative rate (showed in Figure 5c). How does this observation affect the number of papers with incorrectly identified SEM instruments in Figure 4a? Since this false negative rate is high (71.1%), the authors should consider duplicating some of the discussions from line 278 in this paragraph for easier-accessible information for the reader when reading this part.

4. Other minor issues:

4.1 Figure 2b, include the full form of “OCR” in the legend or in the figure caption.

4.2 In the supplementary spreadsheet, fix the caption for Table S4 (referred as Table S3) and Table S5 (referred as Table 5). The font size can also be consolidated across the sheets.

4.3 Figure 5b caption: “For 30 fo the 43” should be corrected.

Reviewer #6: Very solid piece of work! It raises very important issues about reproducibility and validity of reported results and suggests options to analyse and detect most common faults that prevent FAIR principles.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: Yes: Nazife Nur Yazgan

Reviewer #5: No

Reviewer #6: Yes: Aleksandr Mironov

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: PLOSONE_review.docx

Widespread misidentification of scanning electron microscope instruments in the peer-reviewed materials science and engineering literature

PONE-D-25-13149R1

Dear Dr. Amaral,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Zeheng Wang

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: