https://orcid.org/0000-0001-5429-935X
Figures
Abstract
The importance of participation in academic conferences is well known for members of the scientific community. It is not only for the feedback and the improvement of the work, it is also about career development, building networks and increasing visibility. Nevertheless, women continue to be under-represented in these academic events and even more so in the most visible positions such as speaking roles. This paper presents the development of a tool based on performance indicators, which will allow monitoring and evaluating gender roles and inequalities in academic conferences in order to tackle the underrepresentation of women. The study identifies relevant perspectives (participation, organizational structure and attitudes) and designs specific lists of performance indicators for each of them. The tool is based on a combination of two multicriteria techniques, Analytic Hierarchy Process and Analytic Hierarchy Process Sort, and a qualitative analysis based on in-depth interviews and information gathered from a focus group. The use of the AHP multi-criteria decision technique has allowed us to weight the indicators according to the opinion of several experts, and with them to be able to generate from these weightings composite indicators for each of the three dimensions. The most relevant indicators were for the participation dimension. Additionally, the tool developed has been applied to an academic conference which has been monitored in real time. The results are shown as a traffic light visualization approach, where red means bad performance, yellow average performance and green good performance, helping us to present the results for each indicator. Finally, proposals for improvement actions addressed to the red indicators are explained. The work carried out highlights the need to broaden the study of gender equality in academic conferences, not only regarding the participation but also the performance of different roles and functions.
Citation: Corona-Sobrino C, García-Melón M, Poveda-Bautista R, González-Urango H (2020) Closing the gender gap at academic conferences: A tool for monitoring and assessing academic events. PLoS ONE 15(12): e0243549. https://doi.org/10.1371/journal.pone.0243549
Editor: Dragan Pamucar, University of Defence, SERBIA
Received: October 3, 2020; Accepted: November 24, 2020; Published: December 7, 2020
Copyright: © 2020 Corona-Sobrino et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available in the following public repository: Dataset - A tool for monitoring and assessing gender gap in conferences (Version 1) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.4277246.
Funding: Grant Number OR2019-60221 Funder: Open Society Foundations Programme: Open Society Initiative for Europe Award: Expanding the Female Talent Pipeline in Europe https://www.opensocietyfoundations.org/ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction: The gender gap in science
A lack of diversity limits the progression of science. Gender diversity is crucial, however, numbers reveal that gender inequalities persist [1, 2].
The study of the gender gap in science has gradually broadened to include different perspectives. Some of them are well known, such as the different proportion of top-level researcher women working in R&D in Europe [2]. This is related to the so-called “leaky pipeline” effect, which refers to “where even subjects with gender parity (or even a majority of women) at undergraduate and postgraduate levels see declining proportions of women with increasing seniority, and very few in senior positions” [3].
In line with this perspective, a large number of existing studies have examined the persistence of gender imbalance in the scientific structure [1, 4–6]. It has been suggested that women have a lower wage [7, 8], are promoted more slowly [9], receive less research funding [10], and their proportion in editorial and faculties boards is lower [2]
A different line of studies of gender imbalance has focused on knowledge production. There is a wide choice of literature discussing the persistence of gender differences in the production and publication of scientific knowledge, such as bias in the journal reviewing process [11], differences in the number of citations of their work in comparison with their male colleagues [11], a gender asymmetry in collaborations [12] or even, a disparity in commenting on published academic research [13].
Although there are many studies considering gender inequalities in academia, research in the participation in academic conferences remains limited [3, 14–16]. Efforts have been made to highlight the critical under-representation of women [17, 18]. However, these previous studies can only be considered the first step towards a deeper understanding of the dimension of the gender gap in academic conferences.
Different research studies turn a spotlight on diverse aspects of women’s participation in academic events but these studies have almost exclusively focused on one specific factor to analyse the gender differences in conferences, such as the proportion of female speakers [16, 19–21], the gender disparities in different leadership positions [22, 23] or the influence of female organizing committee members in the selection of the speakers [19, 24]. Also, previous studies concentrate on a single specific knowledge area, for instance, the proportion of invited women in the evolutionary biology discipline [14, 15, 20] or in the academic tourism discipline [23, 25], among others. One exception is Nittrouer’s work which examines gender differences in colloquium speakers in six disciplines (biology, bioengineering, political science, history, psychology, sociology) [16]. To the best of the authors’ knowledge, no previous research has either considered different dimensions that can occur in the analysis of female participation in conferences or can be applied to different knowledge areas. Therefore, our new approach is needed in order to study gender disparity in different areas of knowledge and to develop an in-depth study of different roles and types of participation.
This paper addresses the need for a combinative study of different dimensions of the gender gap in conferences, so far lacking in the scientific literature. We propose the development of a tool based on performance indicators, which will allow monitoring and evaluating gender roles and inequalities in academic events in order to tackle the underrepresentation of women. Since our main priority is to focus on mechanisms to enhance female representation in academic conferences, we propose indicators in a novel way using mixed methods with the support of the literature review and expert knowledge as a source of information and decision-making techniques.
Our goal is to identify all the relevant perspectives or dimensions related to the gender gap and to design specific lists of performance indicators for each of them. These measures will allow the organizers of academic conferences to monitor their performance according to each specific dimension. Performance indicators are supposed to shape behaviour and practices in some desirable direction, in our case into an academic events practice ‘with no gender gap’.
For all the above reasons, gender gap should be treated as a multicriteria problem and therefore MCDA techniques are suitable for carrying out its monitorization. Thus, the complexity of this tool based on performance indicators will be tackled with the combination of two multicriteria techniques: Analytic Hierarchy Process (AHP) and Analytic Hierarchy Process Sort (AHPSort). AHP allows us to assign weights to the indicators in order to develop composite indicators. Additionally, AHPSort classifies the conferences into the different levels of the criteria for their evaluation.
Social roles theory
The absence of women in specific positions and developing specific roles might negatively influence the ambitions of other women [15]. It can generate a barrier in the sense of belonging to specific spheres, which sometimes are considered as “male fields”.
Social roles theory provides a framework to understand the importance of being part of academic conferences. According to this theory, people try to figure out which characteristics are required to be successful in a given role by appraising the characteristics of the people who mainly develop that role [26]. The problem is that social roles underline gender stereotypes. These stereotypes “reflect perceivers’ observations of what people do. If perceivers often observe a particular group of people engaging in a particular activity, they are likely to believe that the abilities and personality attributes required to carry out that activity are typical of that group of people” [27].
According to this theory, women (and maybe other under-represented minorities) may feel less capable, less prepared, or have less self-confidence to carry out certain roles [1]. Also, they may feel that they do not belong in that environment. Even more difficult could be the attribution of significant roles regarding such environments, for instance, participating as a keynote speaker in an academic conference. Consequently, if the gender balance in organising a conference is not taken into account, gender stereotyping in science will continue to be the norm [28].
This theory helps us with the idea of generating spaces which encourage and boost the visibility of women and it also helps future generations who can take them as a reference. In this line, Martin’s [6] study underlines that “if we are going to encourage women into careers in science we need also to provide role models for them to aspire to. We need to show that being a woman and being a successful scientist are not mutually exclusive. One way of doing that is to give women scientists a platform to present their research”.
For these reasons, the understanding of a broad representation of the different roles and behaviour for women in a conference needs to be addressed. The idea is to offer a tool that analyses what happens at conferences as regards to gender issues. Thus, such an understanding should contribute to the future of science where men and women can succeed regardless of gender [28].
The importance of attending academic conferences
The importance of participation in academic conferences is well known among members of the scientific community. It is not only for the feedback and the improvement of the work but it is also about career development, building networks, and increasing visibility [3, 29]. Following Hinsley’s work [3], conferences enhance the opportunities to increase a scientific reputation (perceive ability and scientific contribution) and also a social reputation (behaviour and professionals’ relationships) in the scientific community.
Attending conferences is required for the successful development of an academic career. In line with the academic career model [30–32], an academic career is composed of three different interacting careers: cognitive, community, and organizational. The cognitive career consists of the production of scientific knowledge. It is composed of the publications and the process to develop them (topics, publications practices, or projects, among others). The second one, the community career, is related to a series of stages and reputational status (apprentice, master, elite…) which are characterised by the increase of scientific autonomy, leadership, and responsibility. Finally, the organizational career consists of the sequence of positions which implies different salaries, access to infrastructures and resources, performance expectations and roles. These different careers develop simultaneously.
The academic career model [32] can be linked with the importance of attending and participating in academic conferences. Involvement in these events improves the quality of the work for the cognitive career, as well, enhancing self-esteem and individual visibility, and it implies an enrichment of the networks for the community career. Also, it is a prerequisite in some evaluation systems to promote to a different position in the organizational career. Thus, attending conferences is a crucial part of the development of a scientific career. Nevertheless, women continue to be under-represented in academic events [33, 34].
Gender balance in academic conferences should be a fundamental requirement. However, numbers reveal a difference in the proportion of female speakers [16, 19, 35], a disparity in presentation times by gender [15, 36, 37] or an imbalance in the number of female organizing committee members [19]. For instance, Schroeder [20] considers that the under-representation of women is partly attributable to a larger proportion of women, than men, declining invitations.
The explanation about the imbalance suggests that bias against women plays an important role in generating these differences [35, 38, 39]. There are conscious or unconscious biases, subtle or blatant, which have a great impact on the absence of female scientific models [1, 40]. The under-representation of women in science is a multi-dimensional problem. Strategies to face the gender gap should consider all these factors.
The use of indicators to measure the gender gap
Some previous efforts have been carried out within the EU in order to define indicators on gender equality in the scientific realm, such as the monitoring Responsible Research an Innovation (RRI) policies [41, 42]. These approaches suggest that the focus should be on processes of institutional change to see whether general ambitions are translated into concrete forms of action. Furthermore, the European Commission publishes the She figures report [2], which is particularly addressed to policymakers, researchers and their employers, by monitoring human resources statistics and indicators in the research and technological development sector and gender equality in science. The GIM tool [43] is also based on indicators which seek to help partners assess complex gender dynamics, in the context of the implementation of the Agenda 2030.
While the indicator panels mentioned above provide a good overview of the participation of women and men in different sectors and at different levels, they do not seem to provide insight into the cultural issues associated with gender inequality in conferences. Similarly, they do not offer much insight into institutional arrangements and mechanisms for fostering gender balance in this specific goal.
In light of the diverse notions involved in the measurement of gender issues, women’s participation becomes decisive. The difficulties related to gender policies include insufficient resources, resistance, lack of clarity [44] on the integration of different actors, and dependence on gender experts’ knowledge in the development of a gender perspective in policy. Participatory dynamics have been suggested to overcome these risks and achieve the implementation of successful policies [45]. Therefore, the methodology chosen to define the performance indicators will have a participatory nature.
The final purpose of our new panel of performance indicators will be to support and provide evidence and arguments to inform political debates and policymaking in the field of gender equality in academic events.
The rest of the paper is organized as follows: In section 2 we present the research approach followed in the paper, in sections 3–7 the different steps of the proposed methodology are developed and explained, and finally in section 8 a reflection on the results and the conclusions of the research is carried out.
Research approach
Research ethics
The entire research process was approved by an official ethics committee. The data in this study were analysed anonymously. For the interviews and the focus group, we followed a two-stage process: anonymisation of whole transcriptions, and then anonymising individual data extracts. Participants signed an informed consent about study purpose, its aims and how the study data could be used.
For the AHP survey, all participants were informed about the purpose of the questionnaire and the confidentiality of their data. Participants were aware of their right to refuse to participate.
Methodological approach
The proposed methodological approach of this research is presented in the following figure (Fig 1). It is developed through two main stages: the design of the general methodology and the application to a specific conference. The goal of the first one is to identify all the relevant perspectives or dimensions related to the gender gap and to design a specific list of performance indicators for each of them. The aim of the second one is to monitor the performance of a selected conference. All the indicators will be measured, and a traffic light visualization result will be obtained for each of them.
Finally, some recommendations and guidelines will be addressed to the organizers of academic conferences in order to improve their performance according to each specific dimension and to shape practices in some desirable direction into an academics events practice ‘with no gender gap’.
In the following sections a detailed description of the procedure carried out is presented.
Proposal of the first list of indicators based on a literature review
Different research studies (Table 1) turn a spotlight on diverse aspects of women’s participation in academic events, such as the differences in exposure time between women and men, the importance of women’s participation as a member of the organizing committee to provide greater opportunities for women to speak, or the idea of “token-women” to fill the female quota at events, among others. Our new approach is needed to study gender disparity in different areas of knowledge and to develop an in-depth study of different roles and types of participation.
The first list of indicators (Table 1) emerged after a comprehensive literature review on gender gap at academic conferences and events in order to identify which dimensions (or criteria) should be considered when monitoring the gender gap. We identified 4 different dimensions and 18 indicators, both qualitative and quantitative.
Refinement of the list of indicators
The objective of this phase was to identify the more relevant and suitable indicators from the list obtained in the literature survey in order to produce a tailored reduced set of indicators. This phase was addressed in two steps with two different experts’ groups: 1) In-depth interviews and 2) A participatory session.
In-depth interviews with experts
We conducted seven in-depth interviews, online and real time, with gender experts and relevant academics to discuss the first list of indicators in order to identify the more relevant and feasible ones. Interviews attempted to cover a wide range of academic categories, different knowledge areas (social and technical visions) and institutional affiliations (Table 2). The main selection criterion was academic relevance. In this sense, we understand academic relevance as a combination between the quality of research publications and projects and the attendance to academic conferences. In addition, interviewees were selected on the basis of opinions and recommendations from members of the corresponding scientific community regarding the most linkage with gender topics in their academic discipline.
Interviews were conducted between 2019 and 2020 and lasted around 40 and 60 minutes each, fully recorded and transcribed. They were analysed using qualitative content analysis [30] and then coded and managed using N-VIVO software for qualitative data analysis.
The semi-structured interviews focused mainly in the discussion of the dimensions and indicators. Additionally, interviewees offered very interesting information about each individual’s trajectory and experience within the academic world, and on collective and individual strategies and ideas to tackle the gender gap in conferences. Table 3 shows a brief of the arguments presented by the interviewees according to the topic.
An illustration (Fig 2) is provided to exemplify the main ideas to reject and refine the list of indicators which were offered by our interviewees. Also, a new dimension was provided by them: “Gender attitudes perception (gender behaviour, social dynamics, staging)”.
Participatory session and second list of indicators
A focus group discussion was organized in order to develop a deliberative process to integrate stakeholder values with technical judgments [54]. The discussion was conducted by two moderators with expertise in participatory decision-making methodologies and qualitative analysis.
For this focus group, a new set of experts has been selected, since in-person participation was required. Eleven participants took part in this focus group meeting (Table 4). They are all at different levels of their career path and have diverse disciplinary backgrounds, as well as different gender approaches (social and high-tech visions). Experts were selected on the basis of academic relevance (quality of their research publications and projects and the attendance to academic conferences). The main strength of the group was the merger in the different expertise, knowledge and visions with a solid methodological experience, which have provided an international and integrative vision.
The first list of indicators (Table 1) and the main ideas offered by our interviewees to refine the list of indicators (Fig 2) were presented to the participants. An in-depth discussion was oriented in order to analyse and reduce the list as well as the complexity of the tool. During this discussion, one new indicator appeared: “% of women who attend the event”. Dimensions and indicators were re-organized. Finally, the initial list was reduced to 11 indicators grouped in 3 dimensions. In Table 5. the resulting list is presented with a detailed description.
Prioritization of indicators based on experts´ judgements. The use of AHP
The objective of this phase is to identify the more relevant indicators from the final list obtained in order to produce a tailored reduced set of indicators, built upon the hypothesis that there are gender indicators which are more important to be considered in certain academic conferences than in others.
Prioritization of gender indicators according to their level of importance has to be considered from an expert point of view [42]. Gathering and considering the different opinions and judgments of different gender experts can be a difficult task. While the literature deals extensively with the issues of gender indicators, it lacks a prescription of an easy-to-use, yet rigorous, methodology for ranking these indicators. Moreover, when the information available is biased and uncertain, as is the case of gender equality issues in academic events, it is necessary to make estimates. In such cases, experience and knowledge of the problem are more important than the prioritization model itself. Therefore, it is preferable to focus efforts on finding a renowned group of experts and get them involved in the process. For this purpose, we counted on the eleven participants of the focus group described in the previous step, who have diverse discipline and knowledge backgrounds and different gender approaches.
The results of the prioritization process will provide information about the importance of each indicator in order to reduce overall gender gaps in academic conferences. This will allow them to promote actions to improve their performance as regards these indicators.
The Analytic Hierarchy Process (AHP), a method proposed by Saaty [55], is based on theories of relative measurement of intangible criteria. AHP is a well-known technique that deconstructs a decision-making problem into several levels in such a way that they form a hierarchy with unidirectional relationships between levels. The top level of the hierarchy is the overall goal of the decision making problem. The following lower levels are the tangible and/or intangible criteria and sub-criteria that contribute to the goal. This technique allows us to deal with qualitative information about the criteria and its contribution to the objective of the decision making problem. The bottom level consists of the alternatives to be evaluated according to the criteria. AHP uses pairwise comparisons to allocate weights to the elements of each level, measuring their relative importance by using Saaty’s 1-to-9 scale. The method also calculates a consistency ratio (CR) to verify the coherence of the judgements elicited, which should not be greater than 0.10 in order to be accepted. Mathematical foundations and steps of AHP can be found in Saaty [55, 56].
AHP has been chosen because it is one of the best known multicriteria analysis techniques and easy to understand by the experts who participate. There is little doubt that AHP has been frequently adopted. Along with its traditional applications, it has also been used lately in conjunction with others methods: data envelopment analysis (DEA) [57], fuzzy sets [58, 59], neural networks [60], SWOT-analysis [61] among others.
AHP has been used for the prioritisation of the final list of indicators shown in Table 5. The proposed hierarchy is very simple. At the top is the goal of the problem. In this case, our goal is "Prioritization of the gender criteria in order to tackle the under-representation of women in academic conferences." On the second and third levels are the three dimensions and the ten indicators for prioritizing, respectively. According to the AHP method, questionnaires were carried out to allocate weights by means of © 2007–2020 Expert Choice Software v.6.0.013.38933.
The AHP method was explained to the experts during the participatory session. Subsequently, an online questionnaire was sent to them. The answers to the questionnaire were collected by the ©Expert Choice Software. The results obtained were analysed with the help of this online software, which is widely used to support the resolution of AHP problems. Prioritization results for each individual expert were obtained according to their judgment. In order to obtain a group judgement, aggregation of individual judgments (AIJ) were performed using the geometric mean for all the experts [62, 63].
Weights of the criteria show the priority obtained for each indicator, a non-dimensional value that can be considered the relative importance of each one. Care was taken to ensure that all pairwise comparison matrices had a consistency ratio (CR) of less than 10%. It assesses the degree of inconsistency an expert has when eliciting their judgments. Whenever judgments were inconsistent, it was suggested that experts reconsider them so that they would fall within the acceptable limit.
Weights of the dimensions. At the first level of the hierarchy
Table 6 shows the results obtained for each individual expert and the group results in this first level of the prioritization model. The group results have been obtained using the geometric mean of all the individual judgments in order to obtain the aggregation judgement, as it is recommended by Saaty [64]
The weights of the dimensions provide an important insight of each participant overall and an underlying conception of what gender equality means in academic conferences. Having the indicators classified in three dimensions with their weights makes it easier to adapt the evaluation tool to all possible conference contexts (face-to-face, on-line, hybrid, which is very interesting in this new pandemic era, when a significant increase in online conferences is expected.
The experts`priorities indicate that Female participation is more important than Organizational structure and that this last one is more important than Gender attitudes perception.
To analyse each particular conference the group weight will be used.
Weights of the indicators. At the second level of the hierarchy
Weights of the sub-criteria at the second level of the hierarchy show the priority obtained for each indicator, a non-dimensional value that can be considered as the relative importance of each one. The global priority results obtained for all the group of experts were used in order to build composite indicators for the event (Table 7). With this global information of which indicators should be reinforced to improve the gender behaviour of the conference, guidelines can be provided.
The local priority results were used to explain how the tool works inside each dimension (Table 7).
The local priority results show the weights distribution of each indicator (at the second level of the hierarchy) with respect to the dimensions (at the first level of the hierarchy). With these results, we can analyse the predominant indicator within its dimension. As we can observe, the most important indicator for D1 Female participation in Scientific Conference is C1.2% of women who participate according to the type of active participation (34,95%) and specifically the case of keynotes speakers. Regarding the results for D2 Organizational structure oriented to reduce Gender Gap in Scientific Events indicators, C2.3 Event facilities regarding gender policies (child-care policy, etc.) is the predominant indicator (33,79%).
The global priority results show the weights distribution of each indicator with respect to the goal, i.e. with respect to 100% percentage. With these results, we can analyse the predominant indicator in general. Again, we can observe that C1.2 and C2.3 are the most important, both above 10%.
Definition of measurement scales and thresholds for each indicator according to what each indicator assesses
After the relative importance of all the indicators has been obtained in the previous step of the methodology, we need to consider all aspects related to the feasibility of the actual application of these indicators. A detailed analysis of the capacity to collect data from the indicators will be conducted. Thus, an in-depth study to identify how best to collect the data required, and their corresponding measurement scales, is needed.
Measurement scales should allow us on the one hand to identify the measured value with a determined category or class (green, amber, red) and on the other hand to construct composite indicators for each dimension based on all the individual indicators. This in turn would allow the definition of labels for the three different outcomes of the composite indicators which will indicate the overall performance in gender gap of the conference.
To identify a value within a predefined category is a sorting problem. Values measured for the indicators should be sorted into the three different categories (good, average, bad) or according to our traffic light visualization method: green, amber and red.
Measurement of indicators
To sort the values obtained in the monitoring process of the conference we propose to use AHPSort [65]. It is an extension of AHP [55] for sorting problems. It was proposed in 2012 by Ishizaka et al. [65] and has since been applied to several case studies [66–69]. AHPSort has the advantage of requiring far less pair-wise comparisons than the original AHP.
AHPSort is used for the sorting of alternatives into predefined ordered categories. The aim of this method is to classify the alternatives into the different categories of the criteria for their evaluation. The problem is structured using alternatives, criteria and categories for each criterion. The main steps of AHPSort can be summarized as follows:
- Structuring the problem: to start with, the problem needs to be defined and structured as regards the classic AHP method: goal of the problem, criteria cj, j = 1,…, m, and alternatives ak k = 1,…, l.
- Establishing the classes: since it is a sorting problem, we need also to define the categories for each criterion that we called classes Ci,i = 1,…,n, where n is the number of classes. The classes are ordered and have a label (good, average, bad). In order to properly define the classes, local limiting profiles lpij have to be fixed, indicating the minimum performance needed for each criterion j to belong to a class Ci.
- Developing AHP pairwise comparisons: to continue, pairwise comparisons for each single alternative ak with all the limiting profiles lpij for each criterion j are to be fulfilled. From these comparison matrices, the local priority pkj for the alternative ak and the local priorities pij of the limiting profiles lpij are calculated with the eigenvalue method [55].
- The assignment to a class Ci is done by comparing pk with lpi. The alternative ak is assigned to class Ci which has the lpi just below the global priority pk [65].
- If pk ≥ lp1 → ak ∈ C1
- If lp2 ≤ pk < lp1 → ak ∈ C2
- If pk < lpn-1 → ak ∈ Cn
For more details on this sorting method please see [65].
In our case the alternatives would be the conferences monitored, the criteria the list of indicators obtained and the categories of the three predefined classes for each indicator (green, amber and red).
For each of the classes a local limiting profile has to be fixed, indicating the minimum performance needed on each value measured for each indicator to belong to a class. Reference limiting profiles for the three categories are fixed according to the gender related literature and in equality studies published by the different European governments [2, 42, 70, 71]. They indicate the minimum level for a value measured to achieve the amber and the green classes that we called equilibrium threshold (amber) in the first place and the parity threshold (green) in the second place.
By means of AHP pairwise comparisons, each alternative obtains a local priority. To find out in which category the alternative falls, it will only be necessary to see what local priority it has obtained and to see where it is placed in relation to the local priorities of the limiting profiles: green, amber or red.
Local limiting profiles
The local limiting profiles or thresholds are obtained after a literature review and consultation of gender policy documents [2, 41, 42, 72–75].
Two thresholds have been differentiated. The minimum performance to achieve the amber class is the equilibrium threshold (40%-60%) that is, there are at least 40 percent women. Meanwhile, the minimum to achieve the green class is the parity threshold (50%-50%), that is, there are at least 50 percent women. These values have to be considered always in relation to the total number of women who belong to a certain knowledge area. The classification in knowledge areas and the percentage of women in each of them have been obtained according to a political report [75].
Conference monitored and results
The conference we have monitored is of the Innovation discipline and took place in Norway in January 2020. It was the pre-COVID19 period and therefore fully face-to-face. A total amount of 385 persons attended the conference: 147 women and 238 men.
The thresholds for each indicator were calculated according to two main sources:
- A database on information about parity figures in the different scientific areas. This conference is classified within the knowledge area of Social Sciences, whose parity threshold is 41,9% [75].
- The number of participants attending the conference and the type of participation.
Results measured
In the following table (Table 8) we present the limiting profiles and measurements obtained for the different indicators.
Results classified and their traffic light visualization
Each indicator has been assessed by applying the classification technique AHPSort (Fig 3). This technique allows us to classify the results obtained with the measurement of each indicator into one of the classes (red, amber or green). Following the AHP procedure, pairwise comparisons were fulfilled and the local priorities for each single one measured for the conference and for all the limiting profiles for each indicator were calculated. Comparisons were developed by organizers and participants of the conference. They are well aware of the evolution of the conference and the situation of its area of knowledge in terms of gender parity.
In relation to this conference and its results, we can conclude that the only dimension that gets the green light is D3 (Fig 3), which indicates that the attitudes observed in the "unofficial" dynamics of the conference were very positive and did not induce gender bias.
The D2 dimension, referring to the organizational structure, is the one that has the worst behaviour and gets a red light (Fig 3). This indicates that both the organizing committee and the scientific committee had a strong gender bias. All of their indicators except one have also obtained a red light, which means that individually they also have bad behaviour. In the next section, we will reflect on how to improve behaviour in this dimension.
Finally, D1 has obtained an amber light result (Fig 3). When looking at the individual results of its indicators we see that there are very different kinds of behaviour, which will have to be analysed individually in the next section for future recommendations. An amber result creates the expectation that with a little improvement of some of its weak points it could get a green result with some ease.
Recommendations and future guidelines for the conference organizers according to the results obtained
We focus on giving some advice related to the indicators that have obtained the worst results. These recommendations have been gathered from the interviews, the literature search, the focus group, and the experience of the authors.
C1.2% of women who participate according to the type of active participation (as chair, keynote, oral speaker, poster speaker).
To improve the number of moderators, the experts recommended that organisers of academic conferences should extend their contact networks beyond the usual circles. The need to achieve a balanced number (at least 40%) following the maxim "Gender equity is everybody's concern" [23] should be stressed.
Another important point to encourage a diverse participation is to rethink the registration format in the conferences [51]. According to the experts consulted, one of the main problems in attending and participating in events, apart from limited funds, is the extent of these. The interviewees raised the possibility of making daily registrations that would solve the problem of family conciliation and the scarcity of funds to attend conferences.
The improvement in the number of speakers was considered by the experts to be more complex according to certain areas of knowledge. The first important point they highlight is the importance of organizing diverse panels: "avoid all male panels" [76]. To avoid this problem, the following guidelines are given:
- invite women in the first place, because they refuse more invitations to conferences [5, 20] So that if there is a cancellation, there is room for manoeuvre to continue inviting another woman,
- send the invitations sufficiently in advance to favour the conciliation of family and work,
- consult the databases of women experts and scientific communities,
- expand networks and avoid eminently male networks, and
- be aware that there are biases, both conscious and unconscious [4, 14, 77], in the selection of experts, it is important to identify them in order to avoid them.
C2.1% of women who are part of the organizing committee and C2.2% of women who are part of the scientific committee.
For the people interviewed these indicators are important in order to achieve more diverse events. Experts propose that the substantial changes in favour of diversity in the organization of events, come from the presence of women in decision-making and participation bodies, such as these committees. As demonstrated by Casadevall et al. [34] there is a correlation between the presence of women in the organizing committee and their participation as speakers.
Similarly, to correct these indicators, experts recommend the need to expand professional networks to reach a greater number of female academics. In this case, they point out the need to include academics in the early stages of their academic career to avoid the effect known as "leaky pipeline" [17].
C2.3 event facilities regarding gender policies.
This indicator seeks to facilitate issues of family conciliation but also considers those actions carried out in favour of gender equality. In this regard, the most common recommendation is to offer spaces for family conciliation (e.g. breastfeeding or childcare rooms).
The experts considered it necessary to have "anti-harassment" policies that are made explicit by the organizing committee, either online, in the program or in the development of the event itself, such as "violet points", so that all attendees are aware of their existence and, if necessary, can access them.
C2.4% of track sessions and works which involve gender issues.
For the experts consulted, one of the ways to improve this indicator is to call for participation in the conference (Call for Abstract/Papers), in an inclusive manner, which encourages academics to participate and not to be excluded. Likewise, it is necessary to influence the participation of female academics in the early stages of their careers.
On the other hand, depending on the area of knowledge, it may be appropriate to include a specific "track" that addresses gender issues in the area of interest of the conference. It is also necessary to encourage the inclusion of a gender perspective in the whole research process [78].
Conclusions
General conclusions of the paper
This paper proposes a methodology for assessing the gender gap in academic conferences. The methodology consists of two parts, a general one, which fixes the indicators and their weights and can be used in any other conference, and a particular one in which these indicators have been measured for a specific conference in the field of social sciences and innovation.
With regard to the general methodology, we have been able to draw some conclusions. The use of multidisciplinary working groups has allowed us to have a more complete vision of the different approaches to the gender gap and thus to obtain a holistic and robust list of indicators.
We want to emphasize as being very relevant that these indicators have been classified in three dimensions: participation, organizational structure and attitudes, which allows their analysis both individually and combined between them. For example, in the case of online conferences, the third dimension could not be evaluated since it requires the presence of the evaluators at the event. Also, in the case of conferences where we are only allowed to attend but do not have access to the organizing team, only the first dimension could be evaluated. Therefore, having the indicators classified in three dimensions makes it easier to adapt the evaluation tool to all possible conference contexts, which is very interesting in the post-COVID19 era, when a significant increase in online conferences is expected.
In addition, the use of the AHP multi-criteria decision technique has allowed us to weight the indicators according to the opinion of several experts, again multidisciplinary, and with them to be able to generate from these weightings, composite indicators for each of the three dimensions. The most relevant indicators were for the participation dimension: the number of female keynote speakers (27.2%), for the organizational structure dimension: event facilities regarding gender policies (33.4%), and finally for the third dimension: gender attitudes perception, which would take all the weight since it is a unique qualitative indicator.
In this way, not only are results obtained with traffic light colours for the individual indicators, but also for each of the dimensions. Having each indicator and each dimension classified with a colour makes it much easier to see which indicators are performing well and above all which are not and need to be improved.
Regarding the particular application of the proposed methodology to a specific conference we can offer the following conclusions:
The assessment of each of the indicators has been carried out by applying the classification technique AHPSort. The use of this technique allows the classification into categories (traffic light colours) of the results obtained with the measurement of each indicator. Its application requires the expert knowledge of the people who set the limiting profiles that will serve as thresholds for each category. Therefore, this last stage of the methodology is not above criticism when it comes to predicting its replicability. In the case of the conference worked on in this paper, the authors were able to obtain information from people in the organization and above all from people who were very familiar with the conference and had attended it in all its previous editions. This allowed them to make the pairwise comparisons required by the AHPSort technique without great difficulty. Therefore, in these conclusions we want to reflect this fact, which we think will not always be easy to achieve in future conferences that wish to be monitored.
In general, we also want to emphasize that the creation of a tool to monitor academic events requires being able to use it independently of the discipline. Therefore, the applicability of our tool favours its use in any discipline. Likewise, the tool also allows us to compare results between different conferences.
The way to achieve a balance in the conferences is given by intentional changes. This tool favours being able to concentrate on the weakest points of the conference and to carry out, intentionally, the required changes.
The work carried out highlights the need to deepen the study of gender equality in academic conferences, not only regarding the participation but also the performance of different roles and functions. Therefore, the development of a tool that takes into account the different dimensions, both of participation and representation, is fundamental.
Specific conclusions of the case study
In general, we have drawn a series of conclusions from the results of this study:
- It is essential to make intentional changes in order to organize diverse academic events and conferences for the benefit of science and the academic community.
- It is necessary to generate spaces which encourage and boost the visibility of women and helps future generations to take them as a reference according to the social role’s theory.
- It is necessary to encourage the participation of women at different levels, not only in attendance but also in participation, leadership and the development of different roles.
- It is important to make changes from within the organizational structures of the events, including more women in Organizing Committees and Scientific Committees. It would be convenient to carry out brief and specific training sessions or "guidelines" for the sessions' chairs in order to maintain a balance in the questions session, the control of times or other questions that can affect a balanced development of the event.
- It is essential to elaborate and make explicit the equality policies followed by the organization of the event, as well as "anti-harassment" policies and to make their action protocols visible.
Future recommendations
- Avoid panels composed only of men or only of women. Work to achieve diversity understood in a broad way (gender, ethnicity, work categories, age…).
- Select and invite the female keynote speakers first, in order to avoid cancellations resulting in men-only panels.
- Take into account possible biases when making certain organizational decisions (distribution of spaces, order of intervention…).
- Be aware of the small details for the development of the event (the different heights of the platform, the differences in the tone of voice between women and men…).
- Rethink the format of the conferences so that they favour family reconciliation. Offer one-day registration possibilities [79].
Limitations of the work
In the field work, a binary gender system (male-female) has been assumed in order to delimit and speed up research. However, the authors are aware of the different forms of expression and identification that can occur in this type of event. Likewise, other forms of ethnic diversity or social minorities have not been taken into account, due to the need to limit the research.
Owing to the need for physical presence in the sessions to record participation times, it has only been possible to measure this indicator in the sessions called keynotes.
The most complex part is the work with experts when considering all the pairwise comparisons required by the AHPSort. It will not always be easy to count on the collaboration of people linked to the organization of the event and with a historical vision of the evolution of the conference. That is why it is important to study the "terrain" beforehand and to have as many allies as possible within the conference organizing committee.
References
- 1. García-González J, Forcén P, Jimenez-Sanchez M. Men and women differ in their perception of gender bias in research institutions. PLoS One. 2019;14: 1–21. pmid:31805114
- 2.
Commission E. She figures 2018. Luxembourg; 2019.
- 3. Hinsley A, Sutherland WJ, Johnston A. Men ask more questions than women at a scientific conference. PLoS One. 2017;12: 1–14. pmid:29036191
- 4. Holman L, Stuart-Fox D, Hauser CE. The gender gap in science: How long until women are equally represented? PLoS Biol. 2018;16: 1–20. pmid:29672508
- 5. Sleeman KE, Koffman J, Higginson IJ. Leaky pipeline, gender bias, self-selection or all three? A quantitative analysis of gender balance at an international palliative care research conference. BMJ Support Palliat Care. 2019;9: 146–148. pmid:28270395
- 6.
Martin G. An annotated bibliography of work related to gender in science. 2014.
- 7. Jabbaz M, Samper-Gras T, Díaz C. The Gender Pay Gap in Scientific Institutions. Case Study. Converg Rev Ciencias Soc. 2019; 1.
- 8. Charlesworth TES, Banaji MR. Gender in Science, Technology, Engineering, and Mathematics: Issues, Causes, Solutions. J Neurosci. 2019;39: 7228–7243. pmid:31371423
- 9. van den Brink M, Benschop Y. Gender practices in the construction of academic excellence: Sheep with five legs. Organization. 2012;19: 507–524.
- 10. Witteman HO, Hendricks M, Straus S, Tannenbaum C. Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency. Lancet. 2019;393: 531–540. pmid:30739688
- 11. Lariviere V, Ni C, Gingras Y, Cronin B, Sugimoto CR. Global gender disparities in science. Nature. 2013;504: 211–213. pmid:24350369
- 12. Araújo EB, Araújo NAM, Moreira AA, Herrmann HJ, Andrade JS. Gender differences in scientific collaborations: Women are more egalitarian than men. PLoS One. 2017;12: 1–10. pmid:28489872
- 13. Wu C, Fuller S, Shi Z, Wilkes R. The gender gap in commenting: Women are less likely than men to comment on (men’s) published research. PLoS One. 2020;15: 1–15. pmid:32236109
- 14. Débarre F, Rode NO, Ugelvig L V. Gender equity at scientific events. Evol Lett. 2018;2: 148–158. pmid:30283672
- 15. Jones TM, Fanson K V., Lanfear R, Symonds MRE, Higgie M. Gender differences in conference presentations: A consequence of self-selection? PeerJ. 2014;2014: 1–15. pmid:25346879
- 16. Nittrouer CL, Hebl MR, Ashburn-Nardo L, Trump-Steele RCE, Lane DM. Gender disparities in colloquium speakers at top universities. PNAS. 2017. pmid:29255050
- 17. Bos AL, Sweet-Cushman J, Schneider MC. Family-friendly academic conferences: a missing link to fix the “leaky pipeline”? Polit Groups, Identities. 2017;0: 1–11.
- 18. Morehouse Christal, Alla Garcia SF. An end to Manels: Closing the Gender Gap At Europes Top Policy Events II. Open Soc Found. 2018.
- 19. Mehta S, Rose L, Cook D, Herridge M, Owais S, Metaxa V. The speaker gender gap at critical care conferences. Crit Care Med. 2018;46: 991–996. pmid:29596072
- 20. Schroeder J, Dugdale HL, Radersma R, Hinsch M, Buehler DM, Saul J, et al. Fewer invited talks by women in evolutionary biology symposia. J Evol Biol. 2013;26: 2063–2069. pmid:23786459
- 21. Barinsky GL, Daoud D, Tan D, Cerasiello SY, Silva NA, Grube JG, et al. Gender Representation at Conferences, Executive Boards, and Program Committees in Otolaryngology. Laryngoscope. 2020. pmid:32673426
- 22. Silva N, Cerasiello S, Semonche A, Sotayo A, Luis J, Shao B, et al. Gender Representation at Neurological Surgery Conferences. World Neurosurg. 2019;129: 453–459. pmid:31212029
- 23. Munar AM, Biran A, Budeanu A, Caton K, Chambers D. The Gender Gap in the Tourism Academy: Statistics and Indicators of Gender Equality. While Waiting for the Dawn. Copenhagen; 2015.
- 24. Sardelis S, Drew JA. Not “Pulling up the Ladder”: Women Who Organize Conference Symposia Provide Greater Opportunities for Women to Speak at Conservation Conferences. PLoS One. 2016; 1–20. pmid:27467580
- 25. Walters T. Gender equality in academic tourism, hospitality, leisure and events conferences events conferences. J Policy Tour Leis Events. 2018;7963: 17–32.
- 26. Carter AJ, Croft A, Lukas D, Sandstrom GM. Women’s visibility in academic seminars: Women ask fewer questions than men. PLoS One. 2018;13: 1–22. pmid:30260980
- 27. Eagly AH, Steffen VJ. Gender stereotypes stem from the distribution of women and men into social roles. J Pers Soc Psychol. 1984;46: 735–754.
- 28. Martin JL. Ten Simple Rules to Achieve Conference Speaker Gender Balance. PLoS Comput Biol. 2014;10: 10–12. pmid:25411977
- 29. Fathima FN, Awor P, Yen YC, Gnanaselvam NA, Zakham F. Challenges and coping strategies faced by female scientists—A multicentric cross sectional study. PLoS One. 2020;15: 1–14. pmid:32956356
- 30. Gläser J, Laudel G. Life With and Without Coding: Two Methods for Early - Stage Data Analysis in Qualitative Research Aiming at Causal Explanations. Forum Qual Soc Res Sozialforsch. 2013;14: 1–25.
- 31. Gläser J, Laudel G. The Three Careers of an Academic. Discuss Pap Berlin TU Berlin, Cent Technol Soc. 2015.
- 32. Gläser J, Laudel G. A Bibliometric Reconstruction of Research Trails for Qualitative Investigations of Scientific Innovations. Hist Soc Res. 2015;40: 299–330.
- 33. Klein RS, Lane TE, Bethea JR. Speaking out about gender imbalance in invited speakers improves diversity. Nat Immunol. 2018;18: 475–478.
- 34. Casadevall A, Handelsman J. The presence of female conveners correlates with a higher proportion of female speakers at scientific symposia. MBio. 2014;5: 1–4. pmid:24399856
- 35. Isbell LA, Young TP, Harcourt AH. Stag Parties Linger: Continued Gender Bias in a Female-Rich Scientific Discipline. PLoS One. 2012;7: 2–5. pmid:23185407
- 36. Carley S, Carden R, Riley R, May N, Hruska K, Beardsell I, et al. Are there too few women presenting at emergency medicine conferences? Emerg Med J. 2016;33: 681–683. pmid:27534977
- 37. Flaten HK, Goodman L, Wong E, Hammes A, Brown MR. Analysis of Speaking Opportunities by Gender at National Dermatologic Surgery Conferences. Dermatol Surg. 2020;46: 1195–1201. pmid:31834070
- 38. Cislak A, Formanowicz M, Saguy T. Bias against research on gender bias. Scientometrics. 2018;115: 189–200. pmid:29527071
- 39. Lee JC, Sugimoto RC, Zhang G, Cronin B. Bias in Peer Review. J Am Soc Inf Sci. 2013;1: 2–17.
- 40. Cislak A, Formanowicz M, Saguy T. Bias against research on gender bias. Scientometrics. 2018;115: 189–200. pmid:29527071
- 41.
Strand R, Spaapen J, Bauer MW, Hogan E, Revuelta G, Stagl S, et al. Indicators for promoting and monitoring Responsible Research and Innovation—Report from the Expert Group on Policy Indicators for Responsible Research and Innovation. European Commission—Directorate-General for Research and Innovation. 2015.
- 42. Otero-Hermida P, García-Melón M. Gender equality indicators for research and innovation from a responsible perspective: The case of Spain. Sustainability. 2018;10.
- 43.
WECF International. The Gender impact assessment and monitoring tool. 2018.
- 44.
Woodward A. Building Velvet Triangles: Gender and Informal Governance. Edward Elg. In: Piattoni S, Christianse T, editors. Informal Governance and the European Union. Edward Elg. London; 2004. pp. 76–93.
- 45. Woodward A. European gender mainstreaming: Promises and pitfalls of transformative policy. Rev Policy Res. 2003;20: 65–88.
- 46. Larson AR, Sharkey KM, Poorman JA, Kan CK, Moeschler SM, Chandrabose R, et al. Representation of Women Among Invited Speakers at Medical Specialty Conferences. J Women’s Heal. 2019;00: 1–11. pmid:31687866
- 47. Shishkova E, Kwiecien NW, Hebert AS, Westphall MS, Prenni JE, Coon JJ. Gender Diversity in a STEM Subfield–Analysis of a Large Scientific Society and Its Annual Conferences. J Am Soc Mass Spectrom. 2017;28: 2523–2531. pmid:28952050
- 48. Vallence A-M, Hinder MR, Fujiyama H. Data-driven selection of conference speakers based on scientific impact to achieve gender parity. PLoS One. 2019; 1–10. pmid:31365586
- 49.
Democracy WF for. Final Report. World Forum for Democracy. Gender equality: Whose battle? Strasbourg; 2018. Available: https://rm.coe.int/world-forum-for-democracy-2018-final-report/16809231b5
- 50. Silva N, Cerasiello S, Semonche A, Sotayo A, Luis J, Shao B, et al. Gender Representation at Neurological Surgery Conferences. World Neurosurg. 2019;129: 453–459. pmid:31212029
- 51. Tulloch AIT. Improving sex and gender identity equity and inclusion at conservation and ecology conferences. Nat Ecol Evol. 2020;4: 1311–1320. pmid:32747775
- 52. Commission E. El género en la investigación. 2011.
- 53. West JD, Jacquet J, King MM, Correll SJ, Bergstrom CT. The Role of Gender in Scholarly Authorship. PLoS One. 2013;8. pmid:23894278
- 54. Belton V, Stewart T. Multiple Criteria Decision Analysis: An Integrated Approach. 2002.
- 55. Saaty TL. The analytic hierarchy process: planning, priority setting, resources allocation. Mc. Graw Hill; 1980. pmid:21132218
- 56.
Saaty TL. Fundamentals of decision making and priority theory with the Analytic Hierarchy Process. First Edit. Pittsburgh: RWS Publications; 1994.
- 57. Han Y, Zhou R, Geng Z, Bai J, Ma B, Fan J. A novel data envelopment analysis cross-model integrating interpretative structural model and analytic hierarchy process for energy efficiency evaluation and optimization modeling: Application to ethylene industries. J Clean Prod. 2020;246: 118965.
- 58. Bobar Z, Božanić D, Djurić K, Pamučar D. Ranking and assessment of the efficiency of social media using the fuzzy AHP-Z number model-fuzzy MABAC. Acta Polytech Hungarica. 2020;17: 43–70.
- 59. Kutlu Gündoğdu F, Kahraman C. A novel spherical fuzzy analytic hierarchy process and its renewable energy application. Soft Comput. 2020;24: 4607–4621.
- 60. Jena R, Pradhan B, Beydoun G, Nizamuddin , Ardiansyah , Sofyan H, et al. Integrated model for earthquake risk assessment using neural network and analytic hierarchy process: Aceh province, Indonesia. Geosci Front. 2020;11: 613–634.
- 61. Ebrahimifar S, Naji-Azimi Z, Rahim Nia F. Combination of SWOT Analysis, Analytic Hierarchy Process, and Monte Carlo Simulation to Identify the Strategic Positioning of Crisis Management at the Ferdowsi University of Mashhad, Iran. Sci J Rescue Reli. Quarterly Scientific Journal of Rescue and Relief; 2020 Oct.
- 62.
Saaty TL. The Analytic Network Process: Decision Making with Dependence and Feedback. RWS Publications; 2001.
- 63.
Saaty TL, Peniwati K. Group decision making: drawing out and reconciling differences. RWS Publications. Pittsburgh: RWS Publications; 2008.
- 64.
Saaty TL, Peniwati K. Group Decision Making: Drawing out and Reconciling Differences. RWS Publications. Pittsburgh: RWS Publications; 2008.
- 65. Ishizaka A, Pearman C, Nemery P. AHPSort: An AHP-based method for sorting problems. Int J Prod Res. 2012;50: 4767–4784.
- 66. Krejćí J, Ishizaka A. FAHPSort: A Fuzzy Extension of the AHPSort Method. Int J Inf Technol Decis Mak. 2018;17: 1119–1145.
- 67. Lolli F, Ishizaka A, Gamberini R, Rimini B, Ferrari AM, Marinelli S, et al. Waste treatment: an environmental, economic and social analysis with a new group fuzzy PROMETHEE approach. Clean Technol Environ Policy. 2016;18: 1317–1332.
- 68. Ishizaka A, López C. Cost-benefit AHPSort for performance analysis of offshore providers. Int J Prod Res. 2019;57: 4261–4277.
- 69. Miccoli F, Ishizaka A. Sorting municipalities in Umbria according to the risk of wolf attacks with AHPSort II. Ecol Indic. 2017;73: 741–755.
- 70.
GVA G de la GV. Conselleria de Educación, Investigación, Cultura y Deporte. 8355 Spain: Diari Oficial de la Generalitat Valenciana; 2018 p. 39. Available: http://www.ceice.gva.es/web/ciencia/convocatorias-actuales.
- 71.
Sanchez de-Madariaga I, De la Rica-Goiricelaya S, Dolado-Lobregad JJ. Libro Blanco. Situación de las Mujeres en la Ciencia Española. Madrid; 2011. Available: http://www.ciencia.gob.es/stfls/MICINN/Ministerio/FICHEROS/UMYC/LibroBlanco-Interactivo.pdf
- 72. Elsevier. The Researcher Journey Through a Gender Lens. 2020; 1–180.
- 73. Lombardo E. EU Gender Policy. Eur J Women’s Stud. 2003;10: 159–180.
- 74.
European AW-I governance in the, 2004 undefined. Building velvet triangles: gender and informal governance. Edward Elgar Cheltenham.
- 75.
GVA G de la GV. Conselleria de Educación, Investigación, Cultura y Deporte. 8355 Spain: Diari Oficial de la Generalitat Valenciana; 2018 p. 39.
- 76. Corpas M, Gehlenborg N, Janga SC, Bourne PE. Ten simple rules for organizing a scientific meeting. PLoS Comput Biol. 2008;4: 4–6. pmid:18584020
- 77. Ceci SJ, Williams WM. Understanding current causes of women’s underrepresentation in science. Proc Natl Acad Sci U S A. 2011;108: 3157–3162. pmid:21300892
- 78. Commission E. El género en la investigación. 2011.
- 79. Budd A, Dinkel H, Corpas M, Fuller JC, Rubinat L, Devos DP, et al. Ten Simple Rules for Organizing an Unconference. PLoS Comput Biol. 2015;11: 6–13. pmid:25633715