Dear Dr. Roudi,

Thank you very much for submitting your manuscript "The quality and complexity of pairwise maximum entropy models for large cortical populations" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Jonathan David Touboul

Academic Editor

PLOS Computational Biology

Daniele Marinazzo

Section Editor

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Olsen et al fit a large number of Ising models to Neuropixel recordings from multiple cortical areas of feely moving rats from a public dataset. They analyze how the quality of the model degrades with different features of the data such as population size, time binning or firing rates. The paper seems technically sound, and rigorous in varying different aspects of the analysis pipeline, which empirically expand theoretical results from Ref.10. It also introduces a couple of interesting statistical elements, in particular a new Ising partition function estimator and some lessons on the complex relationship between different empirical metrics of model quality used in practice. Experimentally we are not learning anything fundamentally new: it is a much more precise characterization of the long known observation that a simple Ising model only accounts for data well in small populations. Still, the manuscript does the grunt work of checking how this conclusion could be affected by various data and analysis considerations.

Detailed notes:

- I find dubious the scientific relevance of fitting a single Ising model to data that combines several computationally distinct sensory/behavioral conditions from multiple brain areas, but that concern is somewhat outside the scope of this paper. Still, I wonder if lumping together multiple conditions in this way also affects analysis in ways that DO matter for the goal at hand: naively one would think that combining subsets of data with different statistics (due to different computational regimes) results in mixture distributions with more complex associated pairwise correlation structure. Please discuss.

- The explanation of the core metrics needs substantial improvement. In particular I get the moment matching argument for the entropy difference definition of the KL, but it is sloppily done; similarly, the form of the factorized model is not terribly intuitive or properly explained. I imagine that a reader not already quite familiar with MaxEnt models would have a hard time following that. Either provide references together with each equation to make clear where it’s coming from, or (ideally and) explain it in an accessible way in the results.

-Similarly, while I appreciate the symmetry of having all quality measures ranging between 0 and 1 with zero ‘as good as factorized model’ and one ‘as good as actual data’, this involves additional nonstandard steps for some of the metrics and those need a little justification: why sqrt(l2 distance ratio) specifically? Please comment.

-What does “larger variability in firing rates” means precisely when discussing increases in G for the N<20 analysis? 1) the means of different neurons being more different, I.e. population heterogeneity, or 2) more variance in individual neural responses (which scales with the mean for both the factorized Bernoulli model or a more realistic Poisson firing )?

-Does the structure of the larger neural populations sub-factorize by area? If looking at the estimated Js in a mixes population including subsets of neurons from different brain regions then is that structure largely block diagonal, with weak across area interactions and stronger within area couplings?

-since the partition function estimator is critical for a good part of the results, its construction needs to be explained more clearly. I also don’t understand the logic of not describing its properties in the paper itself in a self-contained way.

- Critically, the results text is missing a clear explanation of how the entropy of the real data is being estimated for N>20; also how does one practically go from an estimate of Z to that for S_pair?

- Overall, across all analyses the critical variable affecting model quality is the product N\\bar\\nu\\delta t. The fundamental message of all analyses is to argue for a decrease in model performance as a function of this quantity. Given this agenda, one would need a little more initial motivation to justify why is this the critical quantity in the original theory and why should one expect it to be meaningful outside the perturbation regime of the original theory.

-I personally don’t find the SK subsection of the Results useful or experimentally relevant, despite the precedent of similar analyses in the retina. I also fundamentally question the interpretation of such metrics given that the data distribution was obtained by marginalizing across experimental conditions.

Minor:

- Section … is mentioned in the 2nd paragraph go Methods/Dataset , probably a legacy of a previous numbered section version of the text

-When introducing the constraints on the first and second moments (eq 1a,b) they are referred to as mean and correlations which is strictly speaking a little misleading.

-Title in Fig 2C is wrong, should be Different delta t

-Caption fig 3: nu bar typo

-Also not sure what the color is supposed to mean in 2G, and both panels in figure 3. All need meaningful color legends

DATA and CODE sharing: the data is public but the code for the results it is not, at least as far as i can see.

Reviewer #2: The manuscript describes quantification of maximum entropy models to fit statistics of recorded networks in the brain of a foraging rat. The recorded cells were distributed in several brain regions including visual, auditory, motor and somatosensory cortices. The authors perform analysis of the success of the PME models to predict several other statistics of networks’ dynamics. Overall, the analysis looks good and well performed. The authors also analyze predictions to larger populations and claim that the PME failed to predict the activity structure of these networks, a well know fact.

Reviewer #3: The review is uploaded as an attachment

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No: Data ok, missing github of code or similar.

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Michele Castellana

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Attachments

Attachment

Submitted filename: report.pdf

Dear Dr. Roudi,

We are pleased to inform you that your manuscript 'The quality and complexity of pairwise maximum entropy models for large cortical populations' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Jonathan David Touboul

Academic Editor

PLOS Computational Biology

Daniele Marinazzo

Section Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Thank you for addressing my concerns.

About point 14: the emphasis of that comment was on the fact that the second moment is related to 'correlations' but not mathematically the same thing.

take a random variable x,y and E denoting expectations under p(x), then

mean = E[x]

second moment = E[x^2]

variance = E[x^2] - E[x]^2

covariance = E[xy]-E[x]E[y]

correlation = (E[xy]-E[x]E[y]) / sqrt(Var(x)Var(y))

the constraint parametrized by J concerns the second moment, not correlations as such (although they are related and the literature is often sloppy and confounding the two)

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No