How should scientific review and critique support policy?
A. Treves
L. M. Elbroch
F. Koontz
C. M. Papouchis

Abstract
Decisions that hope to advance broad public interests require a reliable foundation in facts. We provide a post-publication review of Laundré & Papouchis (2020) and compare it with a review included in the report, “Human-Cougar Interactions: A Literature Review Related to Common Management Questions,” undertaken by the Washington State Department of Fish and Wildlife to inform decisions and regulations made by that State’s Fish and Wildlife Commission. We focus on two aspects of objectivity in that report: (1) the constitution of the membership of the persons that performed the review; and (2) how they conducted their review. Our conclusions more broadly reveal how scientific review and critique should and should not be done to support policy decisions, regulations, and legislation, and how policy-makers can be enlightened or misled by scientific review. We conclude with recommendations for what the public and policy-makers should look for in a sound review of scientific evidence.

Introduction

Decisions that hope to advance broad public interests require a reliable foundation in facts. For policies that concern the biophysical environment, science is considered invaluable. Indeed, US environmental agencies have a legal duty to avoid arbitrary or capricious decisions [1]. Some agencies’ duties demand consideration of the best available science. Yet, identifying the best science, and considering it impartially for the sake of the greatest good, is not straightforward even if decision-makers can insulate themselves from political pressures or self-interest. Many environmental agencies rely on teams of qualified scientists to judge what evidence is available and which is best. This is especially important for government commissions that regulate uses of wild species and their environment or direct agency policy because commissioners typically are political appointees with little science training or experience.

Independent review of research findings is an ideal of good science. Moreover, the make-up of review teams and their working methods should fulfill at least two essential criteria as recommended by the US federal government and National Academies of Science for all pre- and post-publication reviews: (1) that reviewers are independent of competing interests (i.e., members can come to their own conclusions and are free to express these in open debate) and independent of each other (i.e., they represent diverse worldviews to prevent domination by one view and its presuppositions about evidence); and (2) that reviewers use accepted standards of evidence and judge data based on scientific criteria rather than anecdote, assumption, personal beliefs or organizational values [2, 3].

Post-publication review teams may be necessary for commissions and government agencies because peer review processes of scientific journals cannot guarantee that any given, single study is reliable or replicable (i.e., the finding can be reproduced by independent investigators). Likewise, a single post-publication review cannot settle an issue forever, as scientific progress in methods and new studies can invalidate older findings. In the face of such paucity of scientific replication and certainty, policy-makers must place trust in their review teams.

Objectivity, defined as “to observe, measure, or infer with impartiality; detachment” (accessed 28 April 2022, https://www.oed.com/), is considered essential to science. Nevertheless, two common pitfalls occur in its practice: 1) people view scientists as either completely objective or completely unobjective, and 2) scientific judgments are sometimes based not on the methods used in observation, measurement, or inference, but on other unscientific criteria.
One extreme form of the binary view of objectivity is to claim no one can be objective. This claim commits the error of attributing impartiality and detachment to individuals rather than to observation, measurement, and inference (scientific actions). Instead, Naomi Oreskes [4] encourages us to think of objectivity as a continuum, and one that improves with practice. Improvements can occur personally and in a community of scientists. For example, relying on a diverse community of independent scientists to resolve uncertainties and disagreements without regard for any one scientist’s reputation or authority [4].
A clear example of the second pitfall comes from U.S. jurisprudence on agency science. In the U.S., courts routinely defer to the scientific claims of government agencies in their areas of expertise (e.g., the Chevron standard). Therefore, courts often award legal victories to the government establishment without examining the scientific methods they used. This is especially troublesome in the instances that agencies produce reports that summarize their reviews of scientific evidence, that are not externally reviewed. We must evaluate the scientific actions of observation, measurement, inference, not the actors or the communication styles.

A Case Study in Objectivity for Post-publication Review

Here, we conduct a post-publication review of Laundré & Papouchis [5] hereafter L&P and contrast our review with the post-publication review of this same paper conducted by the Washington Department of Fish and Wildlife (WDFW) and published in an agency report shared with the Commission and the public [6]. Our conclusions more broadly reveal how scientific review and critique should and should not be done to strengthen policy, how policy-makers can be enlightened or misled by scientific review, and what the public should look for when a scientific review is conducted.

Description of our team.
Following journal policy, we disclose all potentially competing interests in the disclosures portion of this comment below but here we summarize our direct connections to the principal actors whom we address below.

AT co-authored a newspaper opinion with the late J. Laundré. His work is cited by L&P and by the WDFW Team (see disclosures of which articles below).
FK, has collaborated with WDFW on several endangered species recovery programs, volunteered on three WDFW advisory committees, and served on the Washington Fish & Wildlife Commission in 2021.
LME: Elbroch is a puma biologist and works for the wild cat science and conservation organization, Panthera. LME worked as a volunteer for J. Laundré studying pumas in 2000. My work is cited by L&P and by the WDFW Team (see disclosures of which articles below).
CMP was co-author with the late J. Laundré on the study in question (L&P 2020), contributing to its conceptualization, methodology, validation, and review & editing of the original draft. He did not contribute to the review of L&P 2020 that follows.
Our review
Overall, L&P exhibited several hallmarks of good science: hypothesis testing, citation to evidence, and open data, given that the data were all pulled from public databases. The authors tested four primary predictions, and the fourth prediction included two related predictions, for a total of six:
"If sport hunting is achieving the management goals sought by state wildlife managers, then California, in the absence of a sport hunt of pumas, should have: 1) higher puma population densities; 2) higher per-capita number of problematic puma-human encounters; 3) higher rate of puma predation on domestic sheep and cattle; and 4) higher levels of puma predation on ungulate populations, resulting in lower hunting opportunities for sport hunting of ungulates… If these predictions are supported by the 40+ year data base available, then it would lend support to the hypothesis that sport hunting of pumas is a reasonable management strategy to obtain the desired results as stated above. If these predictions are not supported, then it would be reasonable to reject this hypothesis.” p.4 [5]

L&P argue that the data they employed to test their various predictions are the same data used to justify puma sport hunting, and therefore the data were the best data with which to proceed. In general, however, we believe that the authors would have provided greater insights into the influence of sport hunting on the response variables of interest discussed herein had they utilized a stronger analytical framework and a more diverse set of covariates. The authors’ decision to stick with only the data found in public databases, however, was a judgement difficult to assess with science alone. If indeed these are the data driving agency decision-making about sport hunting, then it appears a defensible approach to stimulating discussion and reflection about current assumptions.

Our second general criticism was with the scale of analysis. The authors argue that state agencies make decisions at the scale of states, but we are unclear whether agencies make these assumptions at the geographic scale of entire states, or for smaller game management units (GMUs). We know of examples of both (GMU-level decision-making (e.g., targeted puma hunting in Oregon in response to puma-livestock conflict, public safety issues or failure to meet deer or elk management objectives [7]) and state-wide decision-making, e.g., “PLAN GOAL: Maintain a healthy cougar population within their current distribution while considering human safety, economic concerns, other wildlife species, and maintaining hunting traditions through 2025…Objective 1 Maintain cougar populations within their current statewide distribution” p.3, 21, [8]. In sum, both L&P and the state agencies should conduct appropriate analyses at the appropriate scales.

Below we evaluate each of their predictions in turn:
Prediction 1. California will have higher puma population densities
The authors standardized abundance data across states to pumas/100 km2, and then plotted these data in a figure; there were no confidence levels, so we could not evaluate variation or its significance quantitatively. There were no statistics employed to test this hypothesis, and instead the authors directly compared each state’s calculated pumas/100 km2 visually and in narrative. Because 4 states with puma hunting had higher reported density than California, they rejected the hypothesis that sport hunting reduces puma density and called the results “equivocal at best” p.8, L&P.

We found this weak evidence in support for or against the prediction. Puma hunting results in additive mortality [9] but the impact of hunting on puma abundance varies with many factors not considered by L&P [10], including the intensity (i.e. proportion of the population killed, duration of heavy killing by humans) and the population dynamics of pumas in adjacent habitats unaffected by intentional human-caused mortality (e.g., immigration [11]. Certainly, hunting can reduce puma numbers in local areas for short periods, but over longer time frames, this is less certain (e.g., [12]. For L&P, state agencies, and future analysts, we recommend that the requisite data on correlates of puma abundance and puma mortality from all causes are collected at similar, smaller scales, even smaller than GMUs in some cases, if anyone seeks to understand the long-term effect of intentional killing of pumas.

It is also well established in the field of ecology that animal abundance is driven largely by resource availability—water, forage (prey for pumas), and refugia where animals are safe. The authors did not include any of these variables in their test of puma abundance across the West, which varies from temperate rainforests to dry deserts. The authors also did not appear to include other forms of human-caused mortality, such as road strikes and depredation removals, which in the case of California may be comparatively higher than other states, e.g., [10]. At the scale of state-wide puma densities, we would expect these ecological variables to play important roles in explaining puma abundance.

Prediction 2: California will have a higher number of per capita puma-human incidents
We believe the prediction is fair, and the author’s inferences are defensible, even though we would again criticize the scale of analysis. Rather than a statistical test, the authors used a similar narrative approach as for Prediction 1 in assessing support for this hypothesis. Puma-human encounters are rare, complex, and beyond our scope [13]. Given that California is among the states with the lowest frequency of terms of human-puma incidents, we found the authors’ interpretation of the summary data compelling, but perhaps simplistic.

Prediction 3: California will have a higher percentage of puma predation on domestic livestock
We believe the authors methods and inferences were sound. Including data from multiple years, the authors generated variation around summary statistics on puma-livestock incidents, and therefore were able to use significance testing to evaluate this prediction. Their within-state before-and-after comparison was a stronger approach than above. They analyzed if the number of pumas killed by human hunters in any given year had any correlation with incidents of predation on livestock the following year, across all 10 western states. This supports recent findings in peer-reviewed literature linking puma harvest or removal rates to puma-livestock incidents [14-16]) and in several other carnivore species also (for empirical results [17, 18]; for reviews of evidence see[19-21]. At the massive scale of this analysis, this is compelling evidence for the potential relationship between killing pumas and increasing conflict with people and livestock, even in the absence of other covariates tested.

Prediction 4: California will have higher puma predation on ungulate populations, specifically deer
We believe this initial prediction is based on a misunderstanding of puma ecology, but the misunderstanding may be that of state agencies and not the authors. Puma kill rates are remarkably consistent across ecosystems [22] and therefore the only way to increase puma predation on deer would be to increase puma abundance, which is the same prediction as #1 above. The authors went on to test two additional predictions linked to this one, which are more biologically relevant:

First, “After 10 years of intensive puma control, states with sport hunting of pumas should experience higher deer densities and densities of deer taken by hunters [a standardized metric reflecting deer killed by permitted hunters) than California,” and second, “There should be a positive correlation between deer hunter success and the sport killing of pumas the previous year.” We believe the methods and inferences of the authors were sound for these tests. For these predictions the authors included significance tests to evaluate if there is a correlation between puma hunting and deer abundance or hunter take of deer. Two recent meta-analyses support the findings of L&P and show the limited benefits to ungulate population dynamics gained from killing carnivores [23, 24], and more specifically, the unlikely influence of carnivore mortality on overall ungulate abundance.

Finally, we note that L&P has similar shortcomings as do many studies that attempt before-and-after comparison of the effects of predator control, reviewed in [19-21, 25, 26]. All such studies and subsequent ones suffer from the similar weakness of selection bias and treatment bias. Namely, the subjects that underwent treatment and control (if any) were not randomly selected or treatment was applied in a biased fashion to subjects selected by researchers or managers. These are the most prevalent forms of bias in many scientific fields and especially in predator management and ecology because of the difficulty (but not impossibility) of randomized assignment. L&P took the precaution of within-subject analysis which provides superior inference to correlations that do not control for subjects and their experiences over time. We call on scientists to aim higher and respect the studies that reduce confounding variables more highly than those that do not.
State agency post-publication review of L&P
In 2022, the Washington Department of Fish and Wildlife (WDFW) published the report, “Human-Cougar Interactions: A Literature Review Related to Common Management Questions,” (hereafter WDFW Team). The WDFW chose the Team explicitly for expertise and objectivity: “The Human-Cougar Interaction Science Review team was created…[for] review of the science on the topic of human-cougar conflict objectively.” p. 2-3, [6] [emphasis added]. First, we assessed the diversity in experience and worldview among WDFW team members. From the report:
“The [Team] was created under the guidance of [WDFW’s] Chief Wildlife Scientist, Game Division Manager, and Wildlife Program Director. WDFW members were selected whose roles included cougar research and management or who brought unique skills related to the assessment of scientific methods and literature. WDFW sought external (non-WDFW) team members to bring additional perspectives to the work of the team. The primary criterion for external team members was that they had experience with cougar research and management or had strong research design skills or had exceptional quantitative skills.” p. 2-3, [6].
The WDFW convened 11 authors, of which six were employees of WDFW: two authors held the office of the above-referenced Chief Wildlife Scientist and Game Division Manager, one author was the Deputy Director of the Wildlife Program, and 3 others were subordinate employees of these WDFW managers (pp. 71-73, [6]. Therefore, a majority of the 11-member expert team and authors were employees of the state agency summarizing policy-relevant guidance to the agency and overseeing Commission. To imagine why this might pose a problem for impartiality, consider the difficulty of being impartial when your boss is observing you.

“Additionally, WDFW staff interviewed external members to see if they could approach the task of doing a critical review of the science on the topic of human-cougar conflict objectively (i.e., did not already have a fixed view of the literature on the topic).” p. 2-3 [6]. Interviews are fertile ground for confirmation bias in which one gets the answer one prefers. The costs to impartiality are particularly likely when leaders of an agency that distributes funds are conducting the interviews. Even for those who might never earn a dollar from engaging with the WDFW, there are non-financial competing interests associated with government.

Concerns about government capture by special interests have a long history in environmental regulation [4, 27, 28]; carnivore conservation and management is no exception [29-32], including for cougars [33, 34].

Finally, we don’t know what is meant by a “fixed view of the literature” in the quotation above. If the interviewers knew the literature well enough to evaluate if an interviewee had a fixed view, it suggests some presuppositions about the literature may have already existed.

The diversity of the WDFW Team was low by other criteria: 100% current or former employees of government agencies, among other homogeneous traits (pp. 71-73, [6]. Prior research on carnivore management shows agency scientists’ worldviews differ significantly from their constituents [35, 36], or differ from majorities of scientists [37]. The latter landmark study surveyed 593 grizzly bear scientists to ask about protections for bears. “Overall, a majority (74%) of scientists recommended continued Endangered Species Act protections for Greater Yellowstone Ecosystem grizzly bears.” However, “agency experts were 7.3 times more likely to recommend delisting grizzlies” over maintaining protections [37].

The WDFW Team evaluated L&P in a paragraph, quoted in full below:
“Laundré and Papouchis (2020) made questionable assumptions [1] in their design, and their methodology had substantive shortcomings in data assimilation and analysis [2]. The authors misapplied a treatment-control-design (i.e., claimed a treatment-control design was used when it was not) [3], used poorly justified and questionable variables for deer and livestock abundance [4], modeled phenomena at a statewide scale that no published work has ever suggested operated at that scale [5] (e.g., predation effects on deer), and dismissed general outcomes (e.g., that higher levels of human harvest can reduce cougar numbers) that have been well documented to have occurred in numerous field studies across several states [6]. It appears likely that data dredging was a prominent feature in the analyses [7] of Laundré and Papouchis (2020). An a priori advocacy goal dominates the authors’ Introduction and Conclusion sections [8].“ p.18-19, [6], emphasis and bracketed numbers added; we confirm the above paragraph is the only written critique of [5].

Initially, we note that 6 of the eight bracketed statements above are unsubstantiated assertions of fact because there is not a single citation included to support them. We did, however, find some supportive definitions and literature for [3] and [7] in the methods section of [6],. Below we address each callout in turn.
[1] “questionable assumptions”:
We did not find support for this criticism because the WDFW Team did not define or identify the assumptions that were questionable. Nor did they present evidence to support this accusation. For example, in our own review we specifically questioned the assumptions of scale and cited evidence from puma kill rates and from state agency management plans to do so (above). Therefore, [1] has no validity in its current form as a sweeping dismissal.
[2] “substantive shortcomings in data assimilation and analysis":
We found some support for this statement. We are unclear as what was meant by “data assimilation,” but we believe L&P used the available public data in innovative methods. The WDFW did not explicitly explain what shortcomings they identified in the analyses, but as you will see above, we too criticized some methods, and argued for changes in scale or for broadening the list of potential covariates that could have been explored in evaluating predictions. We also felt that the authors made defensible assumptions, analyses and inferences in other instances. Therefore, we find criticism [2] too sweeping and non-specific.
[3] “claimed a treatment-control design was used when it was not”:
We concluded that this criticism [3] was misleading and erroneous as a point of scientific design. The WDFW Team defined “treatment-control” as “…exploring dynamics of cougars or their prey typically necessitates a control/treatment design using population manipulation achieved through intentional removals or increases or decreases in cougar or ungulate harvest (e.g., Logan and Runge 2021).” p. 6, [6]. Note that the example they cite does not cite L&P. Therefore, we examined whetherL&P’s methods conformed to the definition in the quote above to evaluate if their criticism in [6] is accurate.
In L&P, the first paragraph of methods stated, “State and Federal data sets used in our analysis include: 1) estimates of puma abundance, 2) numbers of pumas killed yearly by sport hunters and other causes, 3) estimates of deer populations, 4) estimates of the number of deer killed yearly by hunters, 5) estimates of the inventory of livestock, cattle and sheep, and 6) estimates of the number of livestock, cattle and sheep killed by pumas.” p.4 [5]. Therefore, it appears L&P studied “Intentional removals or increases or decreases in cougar or ungulate harvest” p. 6 [6] just as required by the WDFW Team’s definition. While the intent of the cougar removals or decreases or increases of cougar and deer populations was not to provide L&P with data, such natural experiments are often exploited in science to draw inference about large, wild-populations [23, 38-41].
If the WDFW Team’s criticism [3] of L&P was instead that the data were not intentionally divided into treatment and control but rather haphazardly divided by events on the ground, then the Team should have considered the possibility of selection bias (e.g., [19, 20]. However, that seems unlikely because the WDFW Team cited [42] for their definition of treatment/control, and those authors wrote about their own work, “Our research was an un-replicated case study on 1 geographic area having a before and after treatment effect design without a separate control area where pumas were not hunted.” p.5, [42]. That design is very similar to the design in L&P, and furthermore, neither the replicated treatment/controls nor the harvest of pumas in [42] was designed by the researchers. Probably, L&P produced stronger inference because they compared several different sites at the same time over years. Therefore, we find criticism [3] misleading and invalid.
[4] “poorly justified and questionable variables for deer and livestock abundance”:
We conclude that this criticism is unwarranted. The data analyzed by Laundré and Papouchis (2020) were pulled from public databases maintained by state and federal agencies, and in our opinion, very relevant to the predictions being tested. We were not alone, as integral to the peer-review process of a journal such as PLoS One is the assumption that independent peer reviewers evaluated the suitability of the variables and took into account their shortcomings, before they supported the publication of the paper. We would argue that additional covariates and data would have been useful in testing the predictions more thoroughly and building a more defensible scientific method, but the variables included seemed pertinent to the questions being asked. The WDFW Team did not substantiate its criticism and made it in a cursory fashion in its report, therefore we are inclined to trust the PLoS One reviewers on this issue.
[5] “modeled phenomena at a statewide scale that no published work has ever suggested operated at that scale”:
We conclude that this criticism is both inaccurate and accurate, depending upon the prediction in question. First, such statement necessitates a systematic review of the literature ([43], which the WDFW Team did not undertake, so they seem to be making this accusation based only on the literature they are familiar with. Peebles et al. (2013), for example, analyzed human-puma conflict data at the GMU scale but made inferences at the state-level. Mattson et al. (2011) analyzed puma attacks on people and made inferences about scales larger than states. Further, even if this accusation were accurate, it would not invalidate the approach. Novel approaches are essential to scientific progress. We share the WDFW Teams apparent concerns about analyzing data at the statewide scale, but we specified how the state may sometimes be appropriate above. Therefore, we find criticism [5] inaccurate, sweeping, and inflammatory.
[6] “dismissed general outcomes (e.g., that higher levels of human harvest can reduce cougar numbers) that have been well documented to have occurred in numerous field studies across several states“.
We agree with the WDFW Team about their specific example, even though the WDFW team provided no evidence to support this general criticism. We discussed the importance of time and geographic scale when assessing the impact of hunting on puma abundance in our own review. We cannot assess criticism [5] further because we would be forced to interpret their vague claim that L&P dismissed general outcomes. If the WDFW Team meant to conclude that human-caused mortality always results in a long-erm decline in cougar abundance, we suggest instead that intensity and diversity of human-caused mortality need to be considered. Therefore, we fund criticism [6] cursory and vague.
[7] "data dredging was a prominent feature in the analyses”:
We concluded that this accusation was unjustified. The WDFW defined data dredging as “…(analysis unguided by specific questions in the hopes of finding something of interest; Nuzzo 2014, Parker et al. 2016).” p.14 [6]. L&P however, made a concise statement of one hypothesis with 6 predictions, which we outlined above in our own review. The Results were relevant to the six predictions also. The data used by L&P were justified and collected from public sources. We find no validity to criticism [7]. Indeed, the word ‘dredged’ has an unscientific tone to it.
[8] An a priori advocacy goal dominates the authors’ Introduction and Conclusion sections.”
We concluded this criticism [8] is a vague ad hominem attack. It contains a common erroneous assumption that scientists should not advocate for anything [44]. In the current context, this criticism [8] seems ironic because the WDFW Team is itself drawing inference with the goal to advocate. Furthermore, the journal PLoS One that published L&P instructs referees to judge submissions on their scientific merit, not the importance, urgency, or other unscientific criteria. Therefore, the WDFW Team replaced its own partial judgment with that of the ostensibly impartial, anonymous, independent peer reviewers recruited by a third-party (the editors of PLoS One).
Recommendations for policy-makers using Scientific reviews
We disagreed with much of the review of L&P conducted by the WDFW Team. We are concerned that the WDFW team eliminated the possibility that this paper be included in the education of policy-makers on human-carnivore interactions. Every study has strengths and weaknesses that should be identified following accepted scientific standards that are openly cited and considered thoughtfully by any post-publication review teams. Peer-reviewed science, such as that included in PLoS One have been vetted by reviewers selected after declaration of potentially competing interests. Post-publication reviews conducted by homogeneous groups lacking an external review process may not hold the same standards of objectivity. We disclosed all of our potentially competing interests in this Comment and subjected or writing to editorial review by PLoS One as a way to avoid this potential pitfall. This case study illustrates the importance of policy-makers understanding the standards for objective, independent reviews of scientific evidence as an aid in their decisions.

There are remedies for failures of impartiality and a loss of objectivity. First and foremost, is diverse representation and inclusion. “The greater the diversity and openness of a community and the stronger its protocols for supporting free and open debate, the greater the degree of objectivity it may be able to achieve as individual biases and background assumptions are ‘outed’…” p.63 [4]. The recent movement to Open Science with its incentives for transparency, self-correction, and replication of scientific findings is also an excellent start for scientific actions as we have defined them here. Among the many tools for improving our own objectivity as scientists, Open Science provides a coordinated toolkit. Its encouragement for transparency in all actions all involved actors, its requirements for data sharing, its provision of registered reports, and its focus on the reproducibility of results offer hope for a brighter future with more objective science.

For agencies and organizations seeking an authentic independent review of science before decisions are made, we summarize our general recommendations as follows:
A. Relinquish ideas and actions to control the constitution or methods of the review team because undue political influence readily interferes with the scientific process of open debate and criticism.
B. Diversify the pool of candidates for review teams and enlarge that pool to avoid ‘echo chambers’ that promote or even reify one worldview;
C. Articulate the goals of the review in impartial scientific language as used in systematic reviews (e.g., https://www.conservatione...);
D. Never gag the review team members or edit the language or modes of communication by the review team members lest you lose the trust of scientists and the public at large;
E. Allow minority reports and comments, as consensus tends to make dissenting voices inaudible; and
F. Apply consistent scientific standards to avoid the appearance or reality of post hoc or hypocritical review. Stipulate the standards of scientific evidence beforehand using third-party, impartial sources.

References

1. Defenders of wildlife et al. V s. Jewell et al.: U.S. District Court for the District of Montana Missoula Division; 2016.
2. NAS National Academies of Sciences EM. Fostering integrity in research. Washington, DC: The National Academies Press; 2017.
3. Nelson A, Lubchenco J. Strengthening scientific integrity. Science. 2022:10.1126/science.abo0036.
4. Oreskes N. Why trust science? Princeton, NJ: Princeton University Press; 2019.
5. Laundré JW, Papouchis C. The elephant in the room: What can we learn from california regarding the use of sport hunting of pumas (puma concolor) as a management tool? PLoS ONE 2020;15(2):e0224638.
6. Kertson B, McCorquodale S, Martorello DA, Anderson J, C., Aoude A, Beausoleil R, et al. Human-cougar interactions: A literature review related to common management questions. In: Wildlife WDoFa, editor. Olympia, Washington, USA2022.
7. ODFW. Hunting cougar in oregon. In: Wildlife ODoFa, editor. 2019.
8. UT DWR. Utah cougar management plan v.3 2015-2025 In: Group UDoWRCA, editor. 2015.
9. Wolfe ML, Koons DN, Stoner DC, Terletzky P, Gese EM, Choate D, et al. Is anthropogenic cougar mortality compensated by changes in natural mortality in utah? Insight from long-term studies. Biol Conserv. 2015;182:187–96.
10. Vickers T, Sanchez J, Johnson C, Morrison S, Botta R, Smith T, et al. Survival and mortality of pumas (puma concolor) in a fragmented, urbanizing landscape. PLoS one. 2015;10:1-18.
11. Robinson HS, Wielgus RB, Cooley HS, Cooley SW. Sink populations in carnivore management: Cougar demography and immigration in a hunted population. Ecol Appl. 2008;18(4):1028-37.
12. Proffit KM, Garrott R, Gude JA, Hebblewhite M, Jimenez B, Paterson J T, et al. Integrated carnivore-ungulate management: A case study in west-central montana. Wildlife Monographs. 2020;206:1–28.
13. Mattson D, Logan K, Sweanor L. Factors governing risk of cougar attacks on humans. Human–Wildlife Interactions. 2011;5:135–58.
14. Peebles K, Wielgus RB, Maletzke BT, Swanson ME. Effects of remedial sport hunting on cougar complaints and livestock depredations. PLoS ONE. 2013;8(11):e79713.
15. Teichman KJ, Cristescu B, Nielsen SE. Does sex matter? Temporal and spatial patterns of cougar-human conflict in british columbia. PLOS ONE. 2013;8(9):e74663.
16. Dellinger JA, Macon DK, Rudd JL, Clifford DL, Torres SG. Temporal trend and drivers of mountain lion depredation in california. Human -Wildlife Interactions. 2021;15:162-77.
17. Santiago-Avila FJ, Cornman AM, Treves A. Killing wolves to prevent predation on livestock may protect one farm but harm neighbors. PLoS ONE. 2018;13(1):e0189729
18. Grente O. Présentation des objectifs et de la méthodologie de la thèse sur l'efficacité des tirs de loup et la gestion adaptative du loup, menée conjointement par l'oncfs et le cefe. Gières, France2021.
19. Treves A, Krofel M, McManus J. Predator control should not be a shot in the dark. Front Ecol Environ. 2016;14:380-8.
20. Treves A, Krofel M, Ohrens O, Van Eeden LM. Predator control needs a standard of unbiased randomized experiments with cross-over design. Frontiers in Ecology and Evolution. 2019; 7 402-13.
21. Khorozyan I. Defining practical and robust study designs for interventions targeted at terrestrial mammalian predators. Conserv Biol. 2021;in press:1–11.
22. Cristescu B, L. M. Elbroch, J. A. Dellinger, W. Binder, C. C. Wilmers, Wittmer HU. Kill rates and associated ecological factors of an apex predator. Mammalian Biology. 2022;102:291–305.
23. Clark TJ, Hebblewhite M. Predator control may not increase ungulate populations in the future: A formal meta-analysis. J Appl Ecol. 2021;58(4):812-24.
24. Miller SD, Person DK, Bowyer RT. Efficacy of killing large carnivores to enhance moose harvests: New insights from a long-term view. Diversity. 2022;14(11):939-54.
25. Khorozyan I, Waltert M. How long do anti-predator interventions remain effective? Patterns, thresholds and uncertainty. Royal Society Open Science. 2019;6(9).
26. Khorozyan I, Waltert M. Variation and conservation implications of the effectiveness of anti-bear interventions. Scientific Reports. 2020; 10,:15341.
27. Wood MC. Advancing the sovereign trust of government to safeguard the environment for present and future generations (part i): Ecological realism and the need for a paradigm shift. Environmental Law. 2009;43:44-88.
28. Schindler DW. The impact statement boondoggle. Science. 1976;192(4239):509.
29. Clark SG, Cherney DN, Clark D. Large carnivore conservation: A perspective on constitutive decision making and options. In: Clark SG, Rutherford MB, editors. Large carnivore conservation: Integrating science and policy in the north american west. Chicago: University of Chicago Press; 2014. p. 251-88.
30. Treves A, Chapron G, López-Bao JV, Shoemaker C, Goeckner A, Bruskotter JT. Predators and the public trust. Biological Reviews. 2017; 92:248-70.
31. Treves A, Batavia C. Improved disclosures of non-financial competing interests would promote independent review. Academia Letters. 2021;Article 514:1-9.
32. Treves A. Best available science" and the reproducibility crisis. Front Ecol Environ. 2022;20(9):495.
33. Mattson DJ, Clark SG. People, politics, and cougar management. In: Hornocker MG, Negri S, editors. Vcougar: Ecology and conservation. Chicago: Cougar: Ecology and Conservation (pp. 206-220). University of Chicago Press. ; 2010. p. 206-20.
34. Mattson DJ. State-level management of a common charismatic predator: Mountain lions in the west. In: Clark SG, Rutherford MB, editors. Large carnivore conservation: Integrating science and policy in the north american west. Chicago: University of Chicago Press; 2014. p. 29-64.
35. Koval MH, Mertig AG. Attitudes of the michigan public and wildlife agency personnel toward lethal wildlife management. Wildl Soc Bull. 2004;32(1):232–43.
36. Gill RB. The wildlife professional subculture: The case of the crazy aunt. Human Dimensions of Wildlife. 1996;1(1):60-9.
37. Karns GR, Heeren A, Toman EL, Wilson RS, Szarek HK, J.T. B. Should grizzly bears be hunted or protected? Social and organizational affiliations influence scientific judgments. Canadian Wildlife Biology & Management. 2018;7(1):18-30.
38. Oro D. Grand challenges in population dynamics. Frontiers in Ecology and Evolution. 2013;1 doi: 10.3389/fevo.2013.00002:2.
39. Estes JA, Terborgh J, Brashares JS, Power ME, Berger J, Bond WJ, et al. Trophic downgrading of planet earth. Science. 2011;333(6040):301-6.
40. Hebblewhite M, White CA, Nietvelt CG, McKenzie JA, Hurd TE, Fryxell JM, et al. Human activity mediates a trophic cascade caused by wolves. Ecology. 2005;86(8):2135-44.
41. Ripple WJ, Estes JA, Beschta RL, Wilmers CC, Ritchie EG, Hebblewhite M, et al. Status and ecological effects of the world’s largest carnivores. Science. 2014;343(6167):1241484.
42. Logan KA, Runge JP. Effects of hunting on a puma population in colorado. Wildlife Monographs. 2021;209:1-35.
43. Haddaway NR, Bethel A, Dicks LVeaEpwlrahtftNEE, 1582–1589 (2020).
44. Treves A, Paquet PC, Artelle KA, Cornman AM, Krofel M, Darimont CT. Transparency about values and assertions of fact in natural resource management. Frontiers in Conservation Science: Human-Wildlife Dynamics. 2021;2:e631998.