Selection of publications and their authors

We used data from SCImago Journal & Country Rank to retrieve all countries whose researchers authored at least 1,000 scientific publications in 2020 in the field of medicine. SCImago Journal & Country Rank is a publicly available portal that includes scientific indicators for journals and countries developed from information in the Scopus® database [14]. Citation data are from over 34,000 titles and over 5,000 international publishers. Seventy-five countries met the inclusion criterion for the study, as shown in Table 1 (country #1: USA with 277,130 publications, country #75: Cuba with 1,059 publications). We also used data from International Migrant Stock 2020, available on the United Nations Population Division portal, to obtain the percentage of migrants by country in 2020. Data on estimates of the number (or "stock") of international migrants are presented as a percentage of the total population, by age, sex, and country of destination, and are based on national statistics, in most cases obtained from population censuses [15]. We selected the 22 countries for which this proportion was below 2.5 percent (Table 1). We restricted the study to these countries only in order to obtain names of researchers that were as homogeneous as possible and representative of the selected countries. The proportion of migrants for these countries ranged from zero for Cuba to 2.2 percent for Japan and Poland.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. List of countries whose researchers authored at least 1,000 medical publications in 2020, and percentage of migrants per country in 2020.

https://doi.org/10.1371/journal.pone.0294562.t001

Then, using PyMed [16], a Python library that gives access to PubMed, we extracted all publications in 2021 with at least one author affiliated with a university or research institute located in the selected countries (N = 118,897). S1 Table shows the Python program used for authors with affiliations in China. The same procedure was followed for the other countries of affiliation.

We obtained a csv file in which the variable ’authors’ had the following form (example for a publication authored by three researchers):

[{‘lastname’ : ‘x’, ‘firstname’ : ‘x’, ‘initials’ : ‘x’, ‘affiliation’ : ‘x’}, {‘lastname’ : ‘y’, ‘firstname’ : ‘y’, ‘initials’ : ‘y’, ‘affiliation’ : ‘y’}, {‘lastname’ : ‘z’, ‘firstname’ : ‘z’, ‘initials’ : ‘z’, ‘affiliation’ : ‘z’}]

Using Stata, we created the variable ’author1’ (i.e., data for first authors only) and the variable ’country1’ (i.e., country of affiliation for first authors only). As the Python program retrieved all articles with at least one author affiliated with one of the countries selected for the study, we removed the publications for which the affiliation to the selected countries did not concern the first author. To do this, we used regular expressions (‘regexm’ in Stata) to extract the country of affiliation of the first author of each article. For missing data (i.e., publications for which the author’s country of affiliation was missing), we added a manual search using the information provided by PubMed (city or university name). Then, all publications for which the country of affiliation of the first author was not one of those selected for the study were removed from the database. The study database contained data for 88,699 publications. Since the authors of these publications were all affiliated with countries with relatively homogeneous populations, we used their country of affiliation as a proxy for their country of origin.

The database included authors with a single affiliation (N = 68,133) and authors with multiple affiliations (N = 20,566). This second group of researchers could possibly include authors with affiliations in several countries (e.g. USA and China). For these researchers, the country of affiliation used was the one that was part of the countries selected for the study (China and not the USA in the case above). We compared the results of the study using the full sample and the subsample consisting only of authors with a single affiliation to assess if the procedure followed for authors with multiple affiliations was appropriate. We again used regular expressions to identify the two groups of researchers. For a researcher with at least two affiliations, these affiliations were separated in the csv file created with the Python program by the newline character ‘\n’ or a semicolon.

NamSor Applied Onomastics

The authors’ names were classified with NamSor Applied Onomastics, a name recognition software [17]. The software recognizes the linguistic or cultural origin of each name and assigns a gender (male or female) and/or an onomastic class (e.g., China or India). As the estimation is probabilistic, the software also provides a probability for the inference (‘probabilityCalibrated’) ranging from zero to one, with ’one’ meaning 100% accuracy. The way this parameter is calculated is described elsewhere [18].

NamSor operates on the principles of linguistic analysis, probabilistic modeling, and machine learning. By leveraging a vast and diverse database of names from around the world, NamSor identifies patterns in names associated with specific regions and ethnicities. It harnesses linguistic attributes, including phonetics and morphology, to enhance its recognition capabilities. In addition, machine learning techniques empower NamSor to continually improve its accuracy through learning from its database. However, the tool has certain potential limitations. First, names can be highly diverse and may not always accurately reflect an individual’s true origin. For example, people may have names from different cultures due to migration, intercultural marriages, or other factors. Second, the algorithm’s performance can vary significantly by country and ethnicity. It may work very well for some regions but less effectively for others. Third, in cases where a person has a first name from one region and a last name from another, the tool may not perform optimally. Finally, NamSor may not be well-suited for areas with highly diverse populations, such as multicultural urban centers or regions with extensive international immigration. In such places, the algorithm’s accuracy may be challenged.

Names can be classified by NamSor in three different ways: by continent of origin (only three continents: Asia, Africa or Europe), country of origin (e.g., China or India) or ethnicity (e.g., Chinese or Indian). The origin of a name refers to the country or continent where an individual was born, or where the individual’s parents or ancestors came from. According to NamSor, the most likely country of origin for the name “Keith Haring” is the United Kingdom with a probability of 48% (i.e., probabilityCalibrated = 0.48). An ethnicity (or an ethnic group) is a group of individuals who identify with each other on the basis of shared attributes distinguishing them from other groups, such as common sets of cultural traditions, ancestry, language and religion. According to NamSor, the most likely ethnicity for the name “Keith Haring” is German with a probability of 61% (i.e., probabilityCalibrated = 0.61). NamSor can also classify names according to race, a classification that we did not use in our study. This categorization includes six classes and is based on the taxonomy used for the US census: White, Black or African American, American Indian or Alaska Native, Asian, Hispanic or Latino, and Native Hawaiian or other Pacific Islander. According to NamSor, "Keith Haring" is most likely "White" (probability = 73%).

We created two other variables: continent#2 ("Europe" replaced by "Europe, America or Oceania") and country#2 ("Spain" replaced by “Spain or Hispanic American country” and "Portugal" replaced by "Portugal or Brazil"). We added these variables because a preliminary analysis of our data showed that a majority of researchers with Hispanic or Portuguese names who were affiliated with universities or research institutes in Brazil, Mexico or Cuba were considered to be from either Spain or Portugal.

Performance analysis

We evaluated NamSor’s performance by computing three efficiency metrics. These metrics refer to the confusion matrix that contains three components, with ’c’ corresponding to correct classifications, ’i’ to misclassifications (i.e., a wrong continent, country or ethnicity assigned to a name) and ’u’ to non-classifications (i.e., no continent, country or ethnicity assigned). In this study, we considered the “actual country of origin” of the researchers to be their country of affiliation, as extracted from the database listing the authors of publications. The “predicted country of origin” was the country determined by NamSor using the researchers’ first and last names. These definitions also apply to continent of origin and ethnicity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0294562.t002

errorCoded = (i + u) / (c + i + u)

errorCodedWithoutNA = (i) / (c + i)

naCoded = (u) / (c + i + u)

The three performance metrics computed in the study can be interpreted as follows: errorCoded estimates the proportion of misclassifications and non-classifications (this measure therefore penalizes both types of errors equally), errorCodedWithoutNA measures the proportion of misclassifications excluding non-classifications, and naCoded measures the proportion of non-classifications. The same metrics were computed in several recent studies, including some conducted by our research team, to estimate the performance of gender detection tools [13, 18–20]. These tools allow to determine the gender of individuals based on their name.

We repeated the analyses by removing all results with inference accuracy <40% (i.e., probabilityCalibrated <0.4), <50% (i.e., probabilityCalibrated <0.5), <60% (i.e., probabilityCalibrated <0.6) and <70% (i.e., probabilityCalibrated <0.7), respectively. All assignments made with an accuracy level below the selected threshold value were considered as non-classifications. These sensitivity analyses were conducted to determine how the proportion of misclassifications and the proportion of non-classifications changed as the accuracy threshold increased or decreased.

Estimating the proportion of foreign researchers in the study sample

We used the misclassification proportion for "country#2", restricting the analysis to only those researchers for whom the country of origin was determined with an inference accuracy ≥70%, as an indicator of the proportion of foreign researchers in the study sample. Indeed, the level of misclassification can be considered as an indirect measure of the maximum proportion of foreign researchers according to the following formula: "proportion of misclassification with inference accuracy ≥70%" = "proportion of misclassification due to NamSor error" + "proportion of misclassification due to foreign researchers". Even including in the calculation only those researchers for whom their country of origin was determined with an accuracy ≥70%, the proportion of foreign researchers should in fact be lower than the proportion of misclassification for “country#2”, since NamSor is not 100% accurate. As researchers are expected to be more mobile than the general population, we hypothesize that by limiting the countries of affiliation to only those countries with a migrant stock <2.5% the proportion of foreign researchers would be at most 5% in the study. We performed all analyses with STATA version 15.1 (College Station, TX, USA).

Ethical considerations

Since this study did not involve the collection of personal health-related data it did not require ethical review, according to current Swiss law.

Main findings

In this cross-sectional study, we examined the performance of NamSor in determining the origin of individuals based on their first and last names. To this end, we used a database of researchers whose country of affiliation was known. We limited the analysis to researchers affiliated with low immigration countries (i.e., <2.5%). We considered the country of origin of these researchers to be their country of affiliation.

All results obtained in the study were similar using the full sample and the subsample consisting only of authors with a single affiliation. We found NamSor to be accurate in determining the continent of origin, especially when using the modified variable (continent#2) and restricting the analysis to names with an inference accuracy ≥50%. For continent#2, the proportion of misclassifications (i.e., errorCodedWNA) was only 6.3% for the full sample, 5.7% for authors with a single affiliation, and 3.3% for the subsample with inference accuracy ≥50%. However, we found that the risk of misclassification was higher with country of origin or ethnicity, but also decreased when using the modified variable (country#2) and the subsample.

Comparison with existing literature

Several authors used Namsor in the past to estimate the origin of individuals in their research, both in medicine [10, 12] and in other disciplines [21, 22], but our study is the first to our knowledge to have evaluated its performance. We already evaluated NamSor’s performance in determining the gender of individuals from their first and last names, and showed that the tool was accurate in the majority of cases [13]. However, we found that NamSor was much less efficient for some countries, for example for Chinese names [18]. We also found that the use of the accuracy parameter (’probabilityCalibrated’) was not useful to improve the performance of NamSor for gender estimation [23].

The results we obtained in the current study were quite different. Asian names were in general relatively well recognized by NamSor. For example, 76% of the names of researchers affiliated with universities or research institutes in China were correctly classified for “country of origin” (and even 85% for “ethnicity”). These figures were 86% and 85%, respectively, for Japan. The results were similar for authors with a single affiliation (China: 76% for “country of origin” and 84% for “ethnicity”; Japan: 87% for both variables). Furthermore, the use of the accuracy parameter greatly improved the performance of the tool for origin. The best compromise between improving NamSor’s performance and increasing the number of non-classifications was obtained with a threshold value of 50%. With a threshold value of 40%, too few queries were considered as non-classifications (2.6% for “continent of origin” and “country of origin”, and 6.8% for “ethnicity”) to make a noticeable change in performance metrics. For example, for “continent of origin” and “country of origin”, errorcodedWNA decreased only from 16.0% to 15.4% and from 27.3% to 26.1%, while these proportions decreased to 12.6% and 19.5%, respectively, for a threshold value of 50%.

As expected, using “continent of origin” yielded more accurate assignments than either “country of origin” or “ethnicity”. This is a logical finding since “continent of origin” consisted of only three categories, far fewer than the other two variables. For example, if authors with Chinese names were considered to be of Japanese origin, the continent of origin (i.e., Asia) would have been correctly estimated, unlike country of origin or ethnicity. However, if researchers using NamSor needed more precision for their study than simply assigning a continent of origin, the use of “ethnicity” would a priori allow more accurate queries than “country of origin”. For example, for the total sample and for authors with a single affiliation, errorCodedWNA was respectively 20.2% and 18.8% for “ethnicity” and 27.3% and 26.5% for “country of origin”. This difference persisted with the various subsamples.

As expected, it was the joint use of “continent#2” or “country#2” and the various subsamples with threshold values of 50% or more that really improved the performance of NamSor. For “continent#2” and a cut-off value of 50%, the proportion of misclassifications was only 2.6% in our study (vs., for “continent of origin”, 16.0% for the total sample and 15.6% for authors with a single affiliation). For “country#2” and the same cut-off value of 50%, this proportion was 9.3% (vs., for “country of origin”, 27.3% for the total sample and 26.5% for authors with a single affiliation). “Continent#2” led to more accurate assignments than “continent of origin”, as many researchers with Spanish or Portuguese names were actually affiliated with universities or research institutes in Latin America. For the same reason, replacing “country of origin” by “country#2” (i.e., "Spain" by "Spain or Hispanic American country", and "Portugal" by "Portugal or Brazil") was also useful for improving NamSor’s performance.

Anglo-Saxon countries (i.e., UK, USA, Canada, Australia and New Zealand) were excluded from the study, as the proportion of migrants was too high in these countries. However, it is likely that if they were included we would observe misclassifications for the same reason as for names of Spanish or Portuguese origin. It would therefore make sense to use a third variable (country#3) that would add a modification to "country#2", replacing "UK", "USA", "Canada", "Australia" and "New Zealand" with "UK or USA or Canada or Australia or New Zealand".

In recent years, there has been growing recognition of the impact of artificial intelligence and other mechanisms on gender equality and biases across various domains. Three papers by Bao & Huang shed light on this topic. The first paper explored how artificial intelligence (AI) can contribute to creating a gender-neutral learning environment, reducing gender disparities in education [24]. The study compared the results of students in the game of Go with human teachers vs. AI trainers. With human teachers, boys consistently had a higher winning rate than girls, whereas the use of AI trainers led to improvements in the performance of both male and female students. The two other papers highlighted the importance of addressing gender-specific favoritism in scientific recruitment processes, particularly in prestigious scientific committees [25, 26]. Greater female representation in these committees could lead to innovative approaches and managerial effectiveness in shaping research resource allocation and public projects. Bao & Huang discussed the underrepresentation of women in top scientific positions and the need to reform scientific election procedures to foster gender balance, illustrating how gender-specific biases can impact career success and exacerbate gender disparities. These three papers underscore the broader context within which our study of NamSor’s ability to predict individuals’ country of origin and ethnicity takes place. The accurate name-based classification offered by NamSor serves as a critical tool in addressing biases, improving diversity, and promoting equity in research and various decision-making processes.

Implications for practice and research

The performance of NamSor in determining the origin of individuals was probably underestimated in our study, as it was based on the assumption that all researchers affiliated with universities or research institutes in a given country were from that country. This assumption is not entirely correct, as the countries of affiliation could be included in the study up to a threshold in the proportion of migrants of 2.5%. In addition, researchers are a priori more mobile than the general population and the proportion of foreign researchers is expected to be above the 2.5% threshold for a number of countries. However, this proportion was probably less than 5% in the study since the proportion of misclassification for “country#2”, which includes misclassification related to foreign researchers, was 5.0% using the sample consisting of all names for which the determination was made with at least 70% accuracy.

The fact that the estimate of the performance of NamSor is rather conservative means that using this tool in research following the procedure proposed in the study is probably safe. However, finer determinations of the origin of individuals, at the level of country rather than continent, could possibly also be an option. In order to demonstrate the validity of this strategy, further studies would be needed, which could rely for example on self-identification or the expertise of linguists or onomatologists to assess the performance of NamSor for a large number of countries. Unfortunately, such studies are often difficult to conduct if the number of participants is large, which would be necessary in order to have a wide variety of names represented. Future studies could also be used to compare NamSor to other similar tools that estimate the origin of individuals based on their names, for example NamePrism, a name-based nationality and ethnicity classification, and ethnicolr, a name-based race and ethnicity classification.

NamSor’s potential applications extend to various stakeholders, including researchers, institutions, and policymakers. Researchers can harness the power of NamSor to evaluate and address issues related to discrimination based on individuals’ origin in academic collaborations, funding decisions, and research recognition. By leveraging NamSor’s predictions, research institutions and funding agencies can foster a more diverse and inclusive academic environment. For example, NamSor could facilitate targeted initiatives aimed at addressing underrepresentation of certain groups in research. This includes allocating resources to support underrepresented researchers or encouraging collaborations that bridge diverse backgrounds. The tool can also serve as a valuable resource for ensuring equity in research evaluations, such as grant allocations and award nominations. Furthermore, policymakers can utilize NamSor’s capabilities to inform and shape evidence-based policies that promote diversity, inclusivity, and global collaboration in research. By recognizing and addressing disparities, decision-makers can take steps to enhance the international research landscape. Therefore, NamSor’s role in improving the accuracy of origin predictions has wide-reaching implications, contributing to a more equitable and inclusive research community.

Limitations

Our study has a large sample size but has two main limitations. First, we restricted the study to twenty-two countries spread over four continents (Europe, Asia, Africa and America). As the performance of NamSor varies depending on the country examined, our results are not necessarily generalizable to other countries. We therefore recommend some changes in the variables used by NamSor. We recommend the use of "continent#2" (i.e., "Europe" replaced by "Europe/America/Oceania") instead of "continent of origin", and the use of "country#3" (i.e., "UK", "USA", "Canada", "Australia" and "New Zealand" replaced in country#2 by "UK or USA or Canada or Australia or New Zealand") instead of "country of origin" or “country#2”. Second, as already stated above, we considered the country of origin of the researchers to be their country of affiliation. Although we restricted the study to countries with less than 2.5% migrants to obtain the most homogeneous populations possible with names representative of the selected countries, there were inevitably foreign researchers in these countries. The results of our study are therefore probably an underestimate of the real performance of NamSor. It would have been better to determine the (actual) origin of the researchers by self-identification, linguistic analysis, or consultation of experts in onomastics. Furthermore, exploring multiple proxies for “country of origin”, combining various sources of information, could offer a more robust approach to ascertain the true origin of individuals. It is essential to recognize that the complexity of individuals’ origin, identity, and mobility necessitates a multifaceted approach to validation.

Preprint

A preprint of this manuscript was posted on Research Square [27].