The authors have declared that no competing interests exist.
Analyzed the data: JJ ZS. Wrote the paper: JJ ZS BV DES. Reviewed the literature: BV DES ZS.
We analyzed the impact of a requirement introduced in December 2010 that all applicants to the Canadian Institutes of Health Research indicate whether their research designs accounted for sex or gender. We aimed to inform research policy by understanding the extent to which applicants across health research disciplines accounted for sex and gender. We conducted a descriptive statistical analysis to identify trends in application data from three research funding competitions (December 2010, June 2011, and December 2011) (N = 1459). We also conducted a qualitative thematic analysis of applicants' responses. Here we show that the proportion of applicants responding affirmatively to the questions on sex and gender increased over time (48% in December 2011, compared to 26% in December 2010). Biomedical researchers were least likely to report accounting for sex and gender. Analysis by discipline-specific peer review panel showed variation in the likelihood that a given panel will fund grants with a stated focus on sex or gender. These findings suggest that mandatory questions are one way of encouraging the uptake of sex and gender in health research, yet there remain persistent disparities across disciplines. These disparities represent opportunities for policy intervention by health research funders.
It is increasingly recognised that scientific evidence often fails to account for sex and gender; consequently it is not always clear whether results can be equally applied to men and women. To address this gap funding agencies and journals are beginning to develop policies and approaches to enhance the uptake of sex and gender considerations by health researchers. The European Commission recently funded GENDER-NET, an ERA-NET project that involves nine national funding agency partners from across Europe who are considering ways to enhance the uptake of sex and gender considerations in research. Similarly, in 2012 the European Association of Science Editors struck a gender policy committee with a mandate to advance gender- and sex-sensitive reporting and communication in science, including the development of a common standard for journals
In North America, the National Institutes of Health (NIH) has long had a policy on the inclusion of women and minority groups in clinical research, and in October 2014 will require applicants to “report their plans for the balance of male and female cells and animals in preclinical studies…with parallel changes in review activities and requirements”
In this paper, we analyze the impact of introducing a requirement that all applicants to CIHR indicate whether and how they are taking sex and gender into account in their research. Our intent is to highlight the extent to which applicants are considering sex and gender, identify areas of health research where sex and gender are well and poorly integrated, and reflect on opportunities to inform policy and practice aimed at fostering the inclusion of sex and gender in health research.
CIHR is a signatory on the Government of Canada's Health Portfolio – Sex and Gender-Based Analysis Policy. This policy underscores the importance of integrating gender and sex into health research when appropriate, for there is significant evidence to “demonstrate that biological, economic and social differences between women and men contribute to differences in health risks, health services use, health system interaction and health outcomes”
Sex and gender influence the health of men, women, boys, girls, and gender diverse people, as evidenced by a large and growing body of literature establishing how the biological and the social intersect at every level, from the cellular to the societal
In recognition of the important influences of sex and gender on health, in December 2010 CIHR made a change to its grant application forms, requiring that all applicants respond to two questions: Are sex (biological) considerations taken into account in this study? Are gender (socio-cultural) considerations taken into account in this study? Initially only those responding in the affirmative to either question were asked to describe how sex and gender considerations would be taken into account in the proposed research design. This was changed after one funding cycle so that negative responders also had to provide an explanation. Descriptions are limited to 2,000 characters. (For the exact wording of the questions, see
1 | Are sex (biological) considerations taken into account in this study? Yes/No |
2 | Are gender (socio-cultural) considerations taken into account in this study? Yes/No |
3 | If YES please describe how sex and/or gender considerations will be considered in your research design. (2000 character limit) |
4 | If NO please explain why sex and/or gender considerations are not applicable in your research design. (2000 character limit) |
The literature on the integration of gender and sex in health research spans research funding and policy
In 1999, ZonMw implemented a policy stipulating that in order to receive financial support studies must address diversity factors, such as sex, age, and ethnicity in health research. Funding applicants and reviewers are provided with guidelines that contain general information about ZonMw's commitment to address diversity factors; in addition, instructions for grant applicants and reviewers highlight the inclusion of specific questions on diversity issues (including sex). A 2007 study showed that applicants who received program specific instructions on diversity were more likely to account for sex and other aspects of diversity (e.g., age, ethnicity) than those who received general guidelines
In the United States, the NIH's policy is supported by legislation. Since 1993, the NIH Revitalization Act has required inclusion of women and minorities in clinical research as a condition of funding. The NIH
CIHR's approach is policy-driven, in that the mandatory questions for applicants were implemented in response to a wider federal government policy on sex- and gender-based analysis. Similar to the approach taken by ZonMw, CIHR requires its applicants to answer mandatory questions on sex and gender; unlike the NIH, it does not mandate the inclusion of specific populations in research designs, nor does it have corresponding peer review criteria on sex and gender. In light of this hybrid approach, we set out to determine how researchers respond to a policy change that requires them to consider the relevance of sex and gender by responding to two questions when applying for research funding. In particular, we address the following questions:
1) In an open funding competition, how many researchers indicate they are considering sex and gender?
2) What areas of health science show evidence that sex and gender are well integrated or poorly integrated?
3) Are there particular themes in researchers' responses that could inform future policy and practice?
The Open Operating Grant Program (OOGP) is the largest of CIHR's open calls for proposals, and provides operating funds to support all areas of health research. CIHR categorizes health research in four broad themes: (1) Biomedical, (2) Clinical, (3) Health Systems and Services, and (4) Social, Cultural and Environmental Factors that Affect the Health of Populations (truncated here as Population Health Research). We conducted a descriptive analysis of the OOGP applicants' responses to mandatory questions on the integration of sex and gender.
Proposals to the OOGP must align with CIHR's mission “to create new scientific knowledge and to enable its translation into improved health, more effective health services and products, and a strengthened Canadian health care system”
We were interested in trends over time and conducted a qualitative analysis of a subset of responses. Included in the analysis were data gathered from funded applications from three competition cycles (December 2010, June 2011, and December 2011). We looked at trends across CIHR's four research themes as well as across CIHR's 50+ peer review panels organized by disciplines/topics. Panel members – who are researchers – review and numerically score OOGP applications.
We analysed data from three OOGP funding competitions: December 2010, June 2011, and December 2011. The total number of applications submitted for each competition was fairly consistent (N = 2,298, 2,294, and 2,285, respectively). In keeping with CIHR's privacy policies we analysed data only from applications that were successfully funded.
There was an overall increase in the percentage of researchers responding affirmatively to the sex and gender questions over the course of the three competitions, which funded a total of 1459 projects (
The proportion of respondents indicating that they did not consider sex or gender declined over time, while the proportion of respondents indicating that they considered sex and/or gender showed a corresponding increase. The addition of a requirement that respondents answering “no” provide a rationale for doing so appeared to correlate with an increase in the number of respondents answering in the affirmative.
The highest proportion of researchers indicating that they took sex into account was in the clinical research field, and the highest proportion indicating they were taking gender into account was in the population health field. Those in the biomedical field were more likely than others to indicate they took neither sex nor gender into account in their research with over 80% of respondents indicating this in December 2010 and June 2011 and over 60% in December 2011 (
We observed variation across disciplines, with applicants in the biomedical sciences being most likely to indicate that they were not integrating sex or gender in their research designs. Clinical researchers were most likely to account for sex, while population health researchers were most likely to account for gender.
Overall, women PIs were more likely than men to indicate that their projects involved either sex (39% of women, 26% of men) or gender (25% of women, 12% of men) (
Research Area | P.I. Sex | Y-Y | Y-N | N-Y | N-N | TOTAL | |||||
38 | 5% | 95 | 13% | 2 | 0% | 602 | 82% | ||||
6 | 3% | 43 | 20% | 1 | 0% | 167 | 77% | ||||
33 | 22% | 42 | 28% | 5 | 3% | 69 | 46% | ||||
25 | 25% | 27 | 26% | 7 | 7% | 43 | 42% | ||||
8 | 20% | 5 | 13% | 3 | 8% | 24 | 60% | ||||
12 | 19% | 7 | 11% | 12 | 19% | 31 | 50% | ||||
20 | 30% | 16 | 24% | 15 | 22% | 16 | 24% | ||||
45 | 55% | 14 | 17% | 7 | 9% | 16 | 20% | ||||
*3 records did not have the sex of primary investigator indicated.
Y-Y = “yes” to sex and “yes” to gender; Y-N = “yes” to sex “no” to gender; N-N = “no” to sex, “no” to gender.
We conducted a two-tailed binomial significance test to determine which set of applications to the 51 panels (
Panel | Pertinent to Sex | Pertinent to Gender |
Aboriginal Peoples' Health (ABH) | 10 of 17 | 9 of 17 |
Biochemistry & Molecular Biology - B (BMB) | 33 of 45 | 15 of 45 |
Social & Developmental Aspects of Children's & Youth's Health (CHI) | 24 of 41 | 20 of 41 |
Gender, Sex & Health (GSH) | 15 of 19 | 14 of 19 |
Psychosocial, Sociocultural & Behavioural Determinants of Health (PB1) | 15 of 30 | 13 of 30 |
Psychosocial, Sociocultural & Behavioural Determinants of Health (PB2) | 16 of 27 | 21 of 27 |
Public, Community & Population Health – (PH1) | 18 of 31 | 13 of 31 |
Clinical Investigation - A (CIA) (Reproduction, Maternal, Children and Youth Health) | 20 of 33 | |
Nutrition, Food & Health (NUT) | 20 of 37 | |
Public, Community & Population Health – (PH2) | 12 of 21 | |
Humanities, Social Sciences, Law & Ethics in Health (HLE) | 11 of 20 | |
Health Policy & Systems Management Research (HPM) | 13 of 28 | |
Health Services Evaluation & Interventions Research (HS1) | 12 of 28 | |
Health Services Evaluation & Interventions Research (HS2) | 13 of 30 | |
Social Dimensions in Aging (SDA) | 5 of 13 |
(Italicized items have a P-Value above 0.05/no significant tendency).
Panel | Pertinent to Sex | Pertinent to Gender |
Biochemistry & Molecular Biology - A (BMA) | 0 of 26 | 0 of 26 |
Cell Biology & Mechanisms of Disease (CBM) | 3 of 34 | 0 of 34 |
Cell Physiology (CP) | 0 of 27 | 0 of 27 |
Cancer Progression & Therapeutics (CPT a.k.a. CT2) | 6 of 56 | 3 of 56 |
Developmental Biology (DEV) | 1 of 22 | 0 of 22 |
Immunology & Transplantation (IT) | 4 of 37 | 0 of 37 |
Systems & Clinical Neurosciences (NSA) | 5 of 39 | 1 of 39 |
Molecular & Cellular Neurosciences (NSB) | 3 of 31 | 1 of 31 |
Behavioural Sciences - B (BSB) (Clinical Behavioural Sciences) | 1 of 18 | |
Cancer Biology & Therapeutics (CBT) | 2 of 30 | |
Molecular & Cellular Biology of Cancer (MCC) | 3 of 29 | |
Microbiology & Infectious Diseases (MID) | 3 of 30 | |
Behavioural Sciences - A (BSA) (Behavioural Studies in Animal Models) | 0 of 22 | |
Clinical Investigation - B (CIB) (Arthritis, Bone, Skin and Cartilage) | 2 of 49 | |
Cardiovascular System - A (CSA) (Cells & Tissues) | 0 of 21 | |
Cardiovascular System - B (CSB) (Basic translational research in Cardiovascular Sciences) | 0 of 27 | |
Pharmaceutical Sciences (PS) | 0 of 38 |
(Italicized items have a P-Value above 0.05/no significant tendency).
Panel | Pertinent to Sex | Pertinent to Gender |
Biological & Clinical Aspects of Aging (BCA) | ||
Biomedical Engineering (BME) | ||
Behavioural Sciences - C (BSC) (Behavioural Studies and Neural Imaging) | ||
Clinical Investigation – C (CIC) | ||
Clinical Investigation - D (CID) (Translational Research in Cardiovascular Sciences) | ||
Diabetes, Obesity, Lipid & Lipoprotein Disorders (DOL) | ||
Endocrinology (E) | ||
Genetics (G) | ||
Genomics (GMX) | ||
Haematology, Digestive Disease & Kidney (HDK) | ||
Knowledge Translation Research (KTR) | ||
Movement & Exercise (MOV) | ||
Medical Physics & Imaging (MPI) | ||
Palliative & End of Life Care (PLC) | ||
Pharmacology & Toxicology (PT) | ||
Randomized Controlled Trials (RC1) | ||
Respiratory System (RS) | ||
Virology & Viral Pathogenesis (VVP) |
(All tendencies have a P-Value above 0.05/no significant tendency).
Eight panels were significantly less likely to award grants focusing on either sex and gender. These were: Biochemistry & Molecular Biology – A; Cell Biology and Mechanisms of Disease; Cell Physiology; Cancer Progression and Therapeutics; Developmental Biology; Immunology and Transplantation; Systems and Clinical Neurosciences; and Molecular and Cellular Neurosciences. Four additional panels were determined to be significantly less likely to award grants focusing on sex, and five additional panels were determined to be significantly less likely to award grants focusing on gender. The 19 remaining panels did not significantly diverge from the normal distribution in either of the values measured. These results suggest that the integration of sex and gender is divided upon disciplinary lines, with the behavioural and public health communities having adopted the integration sex/gender and those panels based on cellular processes having apparently resisted voluntary incorporation of these considerations.
We conducted a content analysis of the responses to the questions and the accompanying abstract for 200 randomly selected funding records from the 1459 successful grants, and categorized them by pattern of response to the sex and gender questions (e.g., yy, yn, ny, nn). Of this sample, twenty-six applicants indicated that their design integrated both sex and gender, 34 indicated that their design integrated sex, seven indicated that their design integrated gender, and 133 indicated that their design did not integrate sex or gender. We were most interested in determining patterns in responses, whether responses were appropriate and justified, and if any areas of science seemed particularly recalcitrant to notions of sex and gender.
In most categories, there was evidence of applicants using the terms sex and gender interchangeably. For example, applicants used the term “gender” in reference to animal model-based studies focused on sex and in relation to investigations of biological differences: “An equal number of male and female animals will be studied to avoid any [genetic] bias resulting from gender differences.” Interestingly, we were more likely to see gender used as a synonym for sex, however, studies primarily focused on gender did not tend to conflate it with sex.
Among the proposals that integrated sex and/or gender, we observed respondents equating a sampling parameter with a sex- and/or gender-based analysis. We saw this pattern in studies with samples composed only of males or females, or in studies with sampling parameters that included both males and females (be they cell lines, animals, or humans). For example, a proposed study with a sample composed only of women or an equal number of men and women was given as an explanation of how sex/gender was integrated (e.g., “We will recruit an equal number of women and men…in this study”). However, in these rationales, the descriptions of the methods did not specify a plan for analyzing these data by sex and/or gender.
A number of abstracts for studies that indicated there was an integration of sex referred to sex as a covariate, stratifying by sex or controlling for sex, or to strategies for avoiding differential item functioning on gender variables. For example, “At a population level, girls and women are consistently less physically active than boys and men. Accordingly, our individual level analyses will control for sex.” Several of the abstracts describing designs that incorporated the use of secondary data focused on sex, based on the conviction that these data are better able to capture sex than gender, while others were able to make a compelling case for using the categories of male and female as proxies for both sex and gender.
Among the studies that did not integrate sex or gender, many respondents used the justification that sex or gender were not relevant to basic science or to non-human research focused on animal or cell models. As one respondent succinctly put it, “This is a basic science research project.” As another explained, “No human subjects used in this study.” Some respondents indicated that they were not considering sex or gender because the issue at hand was “equally important” or affected both men and women. Others indicated that sex and gender were not relevant because they had a single-sex sample (only males/only females), thereby suggesting that we can only study sex or gender by comparing females and males. For example, “Sex and/or gender are not applicable in this research proposal, the particular [condition] is only associated with men;” “This study is being performed entirely in women.” One respondent deemed sex and gender not relevant because their research focused on system-level factors. We also observed a number of respondents justifying the omission of sex and gender on the basis of a lack of evidence – for example, no prior evidence of sex or gender-based differences.
Our analysis showed an overall increase in the proportion of CIHR-funded researchers incorporating sex and gender in their research designs. This trend varied by discipline, with biomedical researchers being least likely to account for sex and gender, clinical researchers being most likely to account for sex, and population health researchers being most likely to account for gender. Women PIs were more likely to respond in the affirmative than their male counterparts, adding a potentially intriguing dimension to ongoing efforts to foster the participation of women in science and how these efforts might intersect with efforts to foster the integration of sex and gender in health research
Our study had several limitations. Our analysis was confined only to successful applicants because privacy requirements prevented us from analyzing data for unsuccessful applicants. Given that CIHR's funding success rate currently ranges between 15-19%, this excludes a considerable proportion of the applicant pool. Moreover, our analysis was limited to data from the three funding competitions after implementation of the mandatory questions on sex and gender (December 2010, June 2011, and December 2011). As such, the findings reported here are from a relatively short period. At a broader level, we realize the need for caution in making causal claims concerning the impact of changing guidelines. Increasing attention to gender and sex in health research is surely the result of a constellation of factors, including broader societal trends and shifting understandings of what constitutes scientific excellence.
Indeed, the trends we observed in our data reflect broader trends in health research and in science. Reviews of the literature continue to show failure to account for sex and gender in research in fields such as neuroscience and biomedicine
To address the lack of analysis and reporting of gender- and sex-specific results, there has been discussion of the role of journals in emphasizing the importance of accounting for sex and gender in health research, and a call for international journals to require authors to report results by sex and gender
As we have shown in this paper, funding agencies have a key role to play in enabling this shift
As we develop and launch training materials and other initiatives we will continue to grapple with the challenge of how to enable meaningful and appropriate integration of sex and gender throughout the health research process. Our approach to date has included the implementation of the mandatory questions, researcher and peer reviewer education, and outreach to journals, among other activities and initiatives. It should be noted that the mandatory questions are not associated with mandatory peer review criteria – that is, peer reviewers are not required to score applicants on their answers to the sex and gender questions. We recognize that the implementation of mandatory questions without corresponding peer review criteria has its limitations, in that it may inspire researchers to reflect on how sex and gender figure into their research designs without necessarily resulting in tangible changes in research practice. It is for this reason that we are engaging in parallel efforts to train researchers and peer reviewers on the integration of sex and gender, as well as collaborating nationally and internationally on the development of sex and gender reporting requirements for journals. All of our forthcoming initiatives will build on what we have learned from our findings, in recognition of the importance of tailoring our tools and offerings to the appropriate scientific communities and their research practices.
We acknowledge the research assistance of Viola Machel, Jodi Proctor, Joël Pomerleau, and Ekaterina Riazantseva.