Introduction

One of the most well-documented disparities in STEM is gender disparity [1,2]. This issue is especially notable in the cancer clinical trial domain, with underrepresentation of women in the leadership of pivotal trials documented as recently as within the last decade [3]. Furthermore, gender disparities in healthcare accessibility have also been documented, and gender disparity also affects public health, policy-making, and diversity metrics [4]. The study of gender disparity in scientific authorship and other contexts often requires the determination of gender from very limited data, e.g., author forenames. Software [5–10] that infers gender from forenames could potentially enable researchers to automate gender prediction in large datasets. Commercial gender prediction services [11,12] such as Genderize and Gender API programmatically predict gender from first names. The gender R package [13] is an open-source alternative to these proprietary gender prediction tools.

Gender prediction software has demonstrated high accuracy when evaluating non-Asian first names, but often falters when evaluating names from Asian cultures [14]. Further, the presence of diacritical marks and hyphens reportedly affects the accuracy of gender prediction in some tools [15]. Few studies [16] to date have evaluated differences in accuracy in gender predicting software between non-Asian and Asian names. To our knowledge, no studies have evaluated how different ways of delimiting two-part first names e.g. Jean-Pierre vs Jean Pierre vs Jeanpierre, affect gender prediction accuracy.

We compared the gender prediction accuracy of Genderize, Gender API, and the gender R package using a large manually curated registry of cancer clinical trialists with labeled genders and diverse nationalities. In addition, we quantified the accuracy of these tools by author nationality and compared different strategies for delimiting two-part forenames, which are common in the English language spelling of Korean, Chinese, Singaporean, and Taiwanese names.

Materials and methods

Three gender prediction tools: 1) Genderize; 2) Gender API; and 3) the gender R package, were tested on a gold-standard registry of cancer clinical trialists with manually determined binary gender. Trialists’ names and affiliations were sourced from the HemOnc knowledge base, a continually growing resource created to capture the standard-of-care treatments in the fields of hematology and oncology. The methodology for building the HemOnc knowledgebase has been previously described [17]. Likewise, the edibility criteria for including trialists in the HemOnc knowledgebase is defined on HemOnc.org [18]. The binary gender classifications used in our study refer to socially constructed gender categories, not biological sex [19,20]. Names in HemOnc are primarily sourced from the MEDLINE records of published clinical trials and undergo extensive normalization to account for the presence of diacritics, middle initials, misspellings, multipart last names represented as middle names, and other variations. When first names are not available through MEDLINE, the original manuscripts are examined for this information. Binary gender is determined by a combination of automated mappings of typically masculine or feminine forenames (e.g., John; Rebecca), web searches of publicly available information such as biographies on academic web pages, and consensus determinations including consultation with native speakers. If gender cannot be determined after these efforts, the author is labeled as “unknown gender.” A subset of journals does not provide forenames; in these cases, the gender is labeled as “could not be determined.” Country affiliations sourced from MEDLINE also undergo extensive normalization.

Results

Out of 40,273 unique clinical trialists present in the HemOnc KB as of 2023-11-21, 37,420 (92.9%) had a resolvable name and were thus eligible for gender determination (S1 Fig). This group was sourced from 7,473 clinical trial manuscripts published between 1947–2023. After excluding trialists with gender not yet determined (n = 4,360, 11.7%), those with a determined unknown gender (n = 78, 0.2%), and those with a determined gender but initial-only first names (n = 14, <0.1%), the final analysis set included 32,968 trialists with predetermined binary gender. Of the 32,968 trialists, 11,398 (34.6%) were designated as women. There were 7849 unique names after normalizing first initial/middle name combinations to only include a middle name. The remainder of names were shared by more than one individual. Michael was the most common name, with 473 (1.4%) occurrences. Only 1,899 (24.2%) of names occurred more than twice.

Of 25,240 trialists with a known site affiliation, 24,930 (98.8%) were affiliated with sites in a single country and were assigned to the country of their affiliated institution when querying Genderize and Gender API with nationalities. When excluding clinical trialists without a recorded country of origin, the number of trialists and unique names was 24,930 and 6,756, respectively. The final analysis set included trialists from 87 countries, the most abundant being the US with 9,485 (38%) affiliated trialists. There were 7,569 first name-country combinations that occurred only once. The most common first-name-country combination was David-US with 201 (0.8%) instances. Only 1,760 (7.1%) of first name-country combinations appeared more than twice.

All names that were misgendered more than once are catalogued in S1 through S6 Tables. The name Andrea was most frequently misgendered when calling the Genderize and Gender API services without countries in the API call. Jan was the most misgendered name for the gender R package SSA method and Genderize when countries were included. The most frequently misgendered names for Gender API when countries are included and the gender R package IPUMS method were Laurence and Nicole respectively. The 100 most common trialist name-country combinations are presented in S7 Table.

Gender prediction accuracy when country of origin was not supplied (baseline case)

The overall accuracy of Genderize when predicting gender for the full dataset without supplying country was 96.6% with 2.3% incorrect gender predictions and 1.1% of names yielding no prediction (Table 1). Similarly, the overall accuracy of Gender API was 96.1% with 2.7% incorrect gender predictions and 1.1% of names resulting in no prediction. The accuracy of the gender R package’s predictions was lower, with 79.8% and 85.7% accuracy with the IPUMS and SSA methods, respectively. Names of men were misgendered as women less than 3% of the time for all gender prediction tools (Table 1). Names of women were misgendered over 3% of the time for all services except the gender R package when using SSA data as a reference. The difference in the percent of correct gender predictions between Genderize and Gender API was significant in favor of Genderize (p<0.001). Likewise, the accuracy difference between the gender R package methods were also significant (p<0.001), in favor of the SSA method. The accuracy differences between the gender R package methods and both Genderize and Gender API were significant, in favor of Genderize and Gender API in all cases (p<0.001). Gender API demonstrated slightly higher gender prediction accuracy when two-part names were delimited with a space: the percent of correctly inferred genders rose from 96.1% to 96.3%. Precision, Recall, and F1 scores for all gender inference tools are presented in S8 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Accuracy of gender predictions on 32,968 included trialists.

https://doi.org/10.1371/journal.pdig.0000456.t001

After restricting Genderize’s predictions to trialists affiliated with a single country, the percentage of correct, incorrect, and missing predictions were 96.2%, 2.6%, and 1.2% respectively (Fig 1A). Genderize achieved the highest accuracy when evaluating first names from German authors, and the lowest accuracy when evaluating names from South Korean, Chinese, Singaporean, and Taiwanese authors. When evaluating the same 24,929 clinical trialists with Gender API, the percentage of correct, incorrect, and missing predictions were 95.8%, 3%, and 1.3% respectively. Gender API also had high accuracy when predicting the gender of German authors, and the lowest accuracy when evaluating names from South Korean, Chinese, Singaporean, and Taiwanese authors. The difference in accuracy between Genderize and Gender API is significant (p<0.001), in favor of Genderize.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Accuracy of Genderize and Gender API.

Panel A shows the gender prediction accuracies when countries are not included in the API call. Panel B shows the results when countries are included in the API call. The top 3 countries with the most trialists and the top 4 East-Asian/Southeast-Asian countries are plotted. Method a is genderize and method b is Gender API. Each bar is labeled with the fraction and percentage of correct gender predictions. Two-part first names were appended together without a delimiting character.

https://doi.org/10.1371/journal.pdig.0000456.g001

Gender prediction accuracy when using different characters to delimit two-part forenames

24,930 of the 32,968 (75.6%) trialists in our dataset were affiliated with sites in a single country and were assigned to the country of their affiliation when querying Genderize and Gender API with nationalities. Gender prediction accuracy when evaluating two-part names was higher when countries were not included in the API call in all contexts except when calling Genderize with the first half of a two-part name, e.g., Jean-Pierre as jean. Genderize was most accurate (76.4%) when no character was used to delimit two-part names, e.g., Jean-Pierre represented as jeanpierre (Fig 2). Genderize provided zero predictions for two-part first names delimited with a space. In contrast, Gender API achieved the highest gender prediction accuracy when delimiting two-part names with a space (83.5%). Gender prediction accuracy for two-part names was worse than for one-part names when countries were not included in the API call and two-part names were separated without a delimiter: OR 0.07 (95% CI 0.06–0.08) for Genderize and OR 0.08 (95% CI 0.07–0.09) for Gender API, respectively (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Accuracy of the gender R package.

Columns a and b show the accuracy of the gender R package’s SSA and IPUMS methods respectively. The top 3 countries with the most trialists and the top 4 East-Asian/Southeast-Asian countries are plotted. Only trialists affiliated with sites in a single country are plotted.

https://doi.org/10.1371/journal.pdig.0000456.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Accuracy of gender predictions based on delimiter between two-part names.

Panel A is Genderize and Panel B is Gender API. Plot facets correspond to the type of delimiter separating two-part names. Stacked bars correspond to correct, incorrect, and no predictions respectively. Bars are labeled with the count and percent of correct gender predictions.

https://doi.org/10.1371/journal.pdig.0000456.g003

The accuracy of Genderize and Gender API were evaluated for statistical significance by comparing the percent of correct gender predictions between delimiter categories. The difference in Gender prediction accuracy between Genderize and Gender API when evaluating two-part names without a delimiting character and including countries in the API call was not significant. All other comparisons between Genderize and Gender API were significant in favor of Gender API (p<0.001).

Gender prediction accuracy by API-reported confidence thresholds

There was high agreement overall between gender prediction services and the gold standard labeled dataset (Fig 4). Genderize reported over 50% confidence in gender predictions for 32,573 (98.8%) trialists. Similarly, Gender API reported over 50% confidence for 32,587 (98.8%) trialists. Gender API demonstrated a correlation of 0.91 between its reported confidence and actual accuracy, compared to Genderize’s correlation of 0.82. The Brier scores for Gender API and Genderize were 0.0077 and 0.0048 respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Experimental name prediction accuracy at different API probability cutoffs.

Names with gender predictions were aggregated into the following API reported probability bins: 50%-55%, 55%-60%, 60%-65%, 65%-70%, 70%-75%, 75%-80%, 80%-85%, 85%-90%, 90%-95%, 95%-100%. The API reported probabilities within each bin were averaged and plotted on the x-axis. The experimentally determined gender prediction accuracies for the names in each bin are visualized on the y-axis.

https://doi.org/10.1371/journal.pdig.0000456.g004

Discussion

Tools [23,24] that infer gender from first names are commonly used by meta researchers to evaluate gender ratios in academia. A study [32] by Holman et al in 2018 reported persistent gender disparities in traditionally male-dominated fields. Gender inference technology like Genderize and Gender API facilitate programmatic gender mapping for methodologies that evaluate gender ratios based on first names.

Genderize and Gender API both demonstrated over 95% overall accuracy on our gold-standard dataset of cancer clinical trialists. Genderize was slightly more accurate than Gender API when countries were not included in the API call. Conversely, Gender API performed slightly better than Genderize when countries were included. For both services, including countries reduced the number of incorrect gender assignments at the cost of increasing the number of names with no predicted gender (Fig 1, S9 and S10 Tables). The gender R package performed worse than Genderize or Gender API (Table 1). The high accuracy of Genderize and Gender API, coupled with relatively modest usage fees, suggest that these proprietary tools may be superior to the gender R package for many use cases.

Genderize and Gender API differed in how their accuracy was affected by the delimiter separating two-part first names. Genderize was most accurate when two-part first names were appended together without a delimiter (Fig 2). In fact, Genderize appeared to be incompatible with two-part first names that were delimited by a space as the service yielded zero correct predictions when evaluating such names. Conversely, Gender API performed best when two-part first names were delimited with a space. The slightly higher overall gender prediction accuracy attained by Genderize compared to Gender API is partially an artifact of our decision to append two-part names without a delimiter in the baseline comparison, since Gender API performed best when two-part names were delimited with a space.

The gender R package was less accurate than both Genderize and Gender API on all fronts (Table 1, Figs 1 and 2). Between the two gender R package methods, the SSA method proved more accurate. Interestingly, the SSA method misgendered men as women more frequently than the reverse. In contrast, all other tools evaluated in this study misgendered women as men more frequently.

The present study provides an updated accuracy benchmark for tools that infer gender from first names. Furthermore, this study adds to a growing body of literature on the accuracy of gender inference tools. Santamaria and Mihaljevic compared Gender API, gender-guesser, Genderize, NameAPI, and NamSor in 2018, and concluded that Gender API achieved superior accuracy [16]. In 2021, Sebo compared Gender API, Genderize, and NamSor, ultimately reporting that Gender API demonstrated the highest accuracy, followed by NamSor [22].

A commonality between the present study and several previous studies was the lower prediction accuracy of Genderize and Gender API when evaluating names of people in Asian countries, with the exception of Japanese names [16,22]. The higher accuracy achieved for both services in our re-analysis of Santamaria’s dataset indicates that both services have improved since 2015, although Genderize improved by a larger margin. Gender API outperformed Genderize when re-analyzing Sebo’s dataset largely because Gender API handled two-part names that were delimited by spaces as well as names with diacritical marks. In fact, a follow up study [15] by the same author recommended removing diacritical marks and modifying two-part names to improve the accuracy of Genderize.

Tools that infer gender from first names will never achieve perfect accuracy because many names are gender-neutral. Additionally, the gender of certain names correlates with country of origin. For example, Andrea, Gabriele, and Daniele are more likely to correspond to men if they are of Italian origin. In fact, Andrea was the most frequently misgendered name when Genderize and Gender API were called without including countries of origin. Importantly, the higher accuracy achieved after reanalyzing Sebo and Santamaria’s datasets suggests that the accuracy of Genderize and Gender API is likely to increase over time as their respective companies continue to improve their products. Indeed, algorithms such as Genderize and Gender API must be periodically calibrated in order to maintain their accuracy in changing contexts [33,34].

This study’s results should be interpreted with certain caveats in mind. We did not filter out recurring first names during this analysis because the count of names in real-world datasets like ours tends to follow a long-tail distribution [35]. The process for determining the “gold-standard” gender of each of the trialists relied on inference from available information. Affiliation data was missing for a substantial subset of authors, mostly due to the older practice of MEDLINE including only the affiliation of the first author; a substantial number of high-profile oncology journals (e.g., the Journal of Clinical Oncology and Blood) did not include clear 1:1 mappings for author-to-affiliation for a period of time; this issue affects at least 320 (4%) of manuscripts in the HemOnc KB. A subset of authors in the HemOnc KB have not had their gender determined (18.1%), and this subset has some important differences from the set of determined genders. Most notably, the undetermined subset has many more Asian hyphenated names (43.8% vs 3.8%) and authors with a country of affiliation including South Korea, China, Singapore, and/or Taiwan (45.6% vs 3.42%). It is thus likely that our results represent a “best-case scenario” and that automated gender mapping will become increasingly difficult as cancer clinical trials are increasingly conducted in the Asia-Pacific region [36,37]. Additionally, a researcher’s nationality in our data set does not always reflect the cultural origin of their first name as some researchers immigrated to the country of their academic affiliation. Furthermore, the database of gendered researchers in the HemOnc KB is constrained by the diversity of the clinical trialist community.

All tools we evaluated were least accurate with names from Asian countries, excluding Japan. Many Asian names with an identifiable gender in their native script become a gender-neutral representation when converted to the Latin alphabet [16]. Journals should consider publishing Asian names using native characters in addition to English translations. Even so, a 2022 study found that many Chinese characters that are traditionally used in women’s names have become more gender-neutral over time [38]. Similarly, a 2023 study reported that phonological characteristics that indicate the gender of the English spelling of Korean names have also become more ambiguous post-2020 [39]. The ambiguity of gender-neutral names underscores the importance for researchers to have a public presence that includes self-reported demographic information.

In this study, the gender R package was tested with the SSA and IPUMS methods because those name datasets are more recent than the gender R package’s NAPP method, which does not include names post-1910. The gender R package’s Kantrowitz method was also not used because potentially gender-neutral names like Alex, Jamie, and Andrea listed as having a gender of “either”, without a probability of which gender is more likely.

It is important to note that Genderize, Gender API, and the gender R package assume a gender binary. However, a recent survey [40] found that 1.6% of U.S. adults identify as transgender or nonbinary. When evaluating gender ratios using gender inference tools, it is imperative that inclusivity must be considered. However, any tool that predicts the likelihood of an individual being non-binary based solely on first names would remain challenging, due in part to the scarcity of gender-labelled datasets that include non-binary individuals. We suggest that academic journals could facilitate research on gender equity among clinical trialists by including self-reported author gender identities. Critically, methodologies that use tools that infer binary gender must acknowledge the exclusion of non-binary individuals as a limitation. Accounting for the presence of non-binary individuals may not be feasible without self-reported gender name data sets. In response, gender researchers should attempt to develop or locate datasets that are labelled with non-binary identities when conducting meta research gender disparity analyses. For example, a 2022 study evaluated the proportions of men, women, and non-binary corresponding authors in physics journals using self-reported gender [41]. Furthermore, researchers should consider prioritizing accuracy over cost when selecting gender inference tools in order to reduce the number of names that are misgendered.

Both Genderize and Gender API demonstrated high gender prediction accuracy with non-Asian names that were highly normalized without middle or last names or diacritical marks. The cost per name evaluated with Genderize is also several times cheaper than Gender API. However, Genderize loses accuracy compared to Gender API when name formatting becomes less consistent. The SSA and IPUMS methods of the gender R package were less accurate but are open-source alternatives. The results from this study provide a new benchmark for gender inference tools. Replicating the studies of Santamaria 2018 and Sebo 2021 demonstrated that Genderize and Gender API have improved over time. Accordingly we should expect that gender prediction accuracy and features of Genderize and Gender API would continue to be more accurate over time.

Supporting information

Acknowledgments

We would like to acknowledge the efforts of the editorial board of HemOnc.org.