PONE-D-20-04537

Community partners’ responses to items assessing stakeholder engagement:  Cognitive response testing in measure development

PLOS ONE

Dear Dr. Thompson,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please note that reviewer number 2 and 3 are the same person. The request was sent twice, and he or she answered likewise. I have kept the two recommendations because there are few different comments at the end of both reviews that might be useful for you.

I recommend to take a careful consideration of the thoughtful revision made by both reviewers, specially of reviewer number 1. They are asking for minor changes, but they might clarify the text and make it more accessible for the readership.

Please submit your revised manuscript by Oct 17 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Roxanna Morote Rios, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified how verbal consent was documented and witnessed.

3. Please refer to any sample size calculations performed prior to participant recruitment. If these were not performed please justify the reasons or cite similar literature. Please refer to our statistical reporting guidelines for assistance (https://journals.plos.org/plosone/s/submission-guidelines.#loc-statistical-reporting).

4.We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Overall the paper makes a meaningful contribution, especially considering the need for qualitative evaluations of quantitative measures. The experience of the survey-taker, participant, or stakeholder (whatever the nomenclature) should be an important part of the review process. However, it is not a completed product. There are some issues in the paper that should be resolved before moving forward with the manuscript. For example, there are several relevant areas lacking clarity, especially in the consistent use of terminology, and this lack of clarity undercuts the goals of the paper.

The clarity issue is exemplified by the discussion under the Item Selection section of the paper (ll. 124-147). An original survey is discussed that has 60 items, then in the next paragraph the authors test a 96-item version. Are these two versions of the same survey? If not, please state in a straightforward fashion that they are two different surveys. If so, how and why did the survey grow to 96 items? The last paragraph of that section states that the number of survey items was narrowed from 48 to 32. Which 48? Is this the 48 quantitative or the 48 qualitative items? Or is it another subset of the 96 items? Or are these 48 items unrelated to the 96 items mentioned earlier? Also in this discussion, a Delphi process is introduced. The process either should be defined briefly right after first mention or covered in an appendix or footnote. Given the issues with the number of items and how items were chosen, the discussion of "16 items from each scale total" on line 180 is likely to cause further confusion.

As noted, the paper has potential to make a contribution to the literature on measurement, cognitive interviewing, and community engagement. The paper is at its strongest when the authors describe that contribution in the second paragraph of the paper (ll. 65-75). This description is clear and to the point; it should serve as a model for how to convey your other important points in the paper. The other areas that require revision for clarity, typos, or other reasons are listed below by page and line number.

-Page 3, line 57: Citations demonstrating the increased interest in community-engaged research would strengthen your opening argument.

-I recommend moving the first sentence on line 87 to the previous paragraph before "One approach to identifying..." Then the following paragraph would begin with "Cognitive interview methods..." The point of the "Researchers recognize..." sentence sets up the last sentence of the preceding paragraph better than it does the following paragraph.

-Page 5, line 106: missing quotation marks at the end of the sentence.

-Page 6: item number issues already noted above

-Page 8, line 184: This would be a good place to insert the actual probes, which are discussed in the preceding and following paragraphs. Even if only a subset are listed or described, this would help the reader have a better understanding of the nature of the interviews.

-Page 9, line 190-191: The sentence starting with "Behavioral coding..." is confusing. Are the following items (ll. 192-201) the probes (see point above)? Coding is the researchers' process, not an interaction with a participant. But this sentence implies that behavioral coding involved the participant.

-Page 10, line 211: The first sentence discusses relevant codes. What is the threshold for relevance? How did you arrive at that? Did you follow best practices, prior research, etc.? It is important to be clear what overall guiding principle was employed because the codes and themes covered here are the keystone of the whole paper.

-Page 10, lines 212-222: Very good section. This description of the process was clear and helpful.

-Page 10, line 225: Extra space not needed before Results section.

-Table 2, page 12: Helpful, informative table.

-Page 14, line 274: Can get rid of this quote. It's already covered in the preceding sentences.

-Page 17, line 332: The authors say that the items were difficult to comprehend. I would be careful here. You didn't demonstrate that (or maybe you did but it isn't demonstrated in the paper); what you showed was that the wording was a hurdle that affected how community members responded to the item. Those are two different things.

Reviewer #2: The primary objective of this study is clearly defined and highly relevant to studies requiring stakeholder engagement. The authors identify that measures of stakeholder engagement are not very strong methodologically. They add to the literature by addressing measurement of stakeholder engagement in terms of literacy concerns, attitudes about information needed to judge engagement, and response preferences for items used in public health community-engaged research. The methods clearly describe their approach which draws on 16 individuals for one on one cognitive response interviews. They clearly describe methods for refinement in Items from an initial survey containing 60 items to 48 and then to 32, using a modified Delphi process. Could a figure be added to the manuscript to show the research and decision path from the 60 items down to the 32 items.

The cognitive interview participants were diverse in age, education, and predominantly African American women. The results described the item response and the steps taken to remove items and to clarify questions that had raised issues for participants. Importantly, the results support the recommendation that academic partners and researchers should guard against the assumption of common understanding is participants found some items vague and needing more context. Comprehension issues were of greater concern in measure development than response options.

Accordingly, the details described in this paper demonstrate the rigor and refinement the authors bring to the issue of measuring community engagement. Their conclusions are well justified and should help advance the field.

The tables add to the manuscript.

The title for Table 2 might be expanded to give more context.

Reviewer #3: The primary objective of this study is clearly defined and highly relevant to studies requiring stakeholder engagement. The authors identify that measures of stakeholder engagement are not very strong methodologically. They add to the literature by addressing measurement of stakeholder engagement in terms of literacy concerns, attitudes about information needed to judge engagement, and response preferences for items used in public health community-engaged research. The methods clearly describe their approach which draws on 16 individuals for one on one cognitive response interviews. They clearly describe methods for refinement in Items from an initial survey containing 60 items to 48 and then to 32, using a modified Delphi process. Could a figure be added to the manuscript to show the research and decision path from the 60 items down to the 32 items.

The cognitive interview participants were diverse in age, education, and predominantly African American women. The results described the item response and the steps taken to remove items and to clarify questions that had raised issues for participants. Importantly, the results support the recommendation that academic partners and researchers should guard against the assumption of common understanding is participants found some items vague and needing more context. Comprehension issues were of greater concern in measure development than response options.

Accordingly, the details described in this paper demonstrate the rigor and refinement the authors bring to the issue of measuring community engagement. Their conclusions are well justified and should help advance the field.

The tables add to the manuscript.

The title for Table 2 might be expanded to give more context.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Community partners’ responses to items assessing stakeholder engagement:  Cognitive response testing in measure development

PONE-D-20-04537R1

Dear Dr. Sanders Thompson

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Roxanna Morote Rios, Ph.D

Academic Editor

PLOS ONE