2.1. Identity-driven authorship prediction

It is surprising that mature software is not more widely available or used to solve institutions’ need to maintain publication lists at scale. A given institution may be responsible for tracking thousands of publications by thousands of scholars. In addition, scholars frequently adhere to semi-predictable patterns in their record of scholarship, with some co-authoring a hundred or more papers with a certain colleague and others publishing almost exclusively in a certain niche journal or discipline, or on a particular topic.

Any software that predicts article authorships would do well to emulate certain cognitive subroutines of a savvy librarian who is charged with identifying all of that scholar’s publications and has access to significant portions of a curriculum vitae to do so. Librarians generally conclude that a person who received a Bachelor’s degree in 2010 is unlikely to have authored a scholarly paper in 2008 and even far less likely to have authored one in 1998. They know that social workers tend not to contribute to bioinformatics journals. They realize that an article written by an author with a first name of “Terry” is unlikely to be written by a scholar named “Terrie” despite there being a Levenshtein distance of only two between those names. Our shrewd librarian also realizes that students are unlikely to have authored a corpus of several hundred publications while still in medical school, and that the more common a scholar’s name is, the less likely that person is to have authored any one candidate paper. Further, they know that when even one of a group of closely related articles was (or was not) written by a scholar of interest, we can more accurately predict whether the other articles were.

The process of using software to predict publication lists for scholars necessarily starts with a critical mass of information about the scholars themselves. These person metadata include name aliases, email addresses, names of grants, departmental affiliations, institutional affiliations, and names of colleagues who are potential co-authors. While these data are available in a curriculum vitae, they are also stored and regularly updated in many academic institutions’ information systems. We define identity-driven authorship prediction as the process of using information about a scholar, as recorded outside of a bibliographic system, to quantify the likelihood that person wrote a set of articles. See Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Identity-driven authorship prediction systems.

Systems that leverage institutionally available information about a scholar to retrieve candidate articles and predict which of those articles a scholars has written. Curators provide feedback on those predictions, and data is shared with downstream systems.

https://doi.org/10.1371/journal.pone.0244641.g003

Many human-facing algorithms are opaque, failing to explain, for example, why a particular book, movie, or restaurant was recommended.

Qian et al. write [10] that human judgment purely on the basis of algorithms is not suitable because the automatically produced results are not understandable for humans. This is problematic in the case of authorship prediction systems. Even the most advanced algorithm is not 100% accurate, and, at many institutions, the process of maintaining publication lists is the responsibility of departments, librarians, and administrative staff. Absent a clear indication such as a matching email address or ORCID identifier, both of which are infrequent, third parties may not easily recognize an article as being authored by a given scholar. Therefore, it is important that an authorship prediction system plainly explain why a candidate article has been scored as it has as well as provide a notion of relative confidence.

It is likewise practical to track which publications a scholar has not authored both to improve accuracy and to avoid the inevitable questions as to why a particular candidate article is not among the accepted articles.

2.2. Approach used by ReCiter

In this article, we describe ReCiter [11], an open source, identity-driven authorship prediction algorithm. ReCiter retrieves a set of candidate articles from PubMed and, optionally, Scopus (standard institutional license required); clusters those articles; and uses the wealth of institutionally-maintained identity data available to academic institutions to score candidate articles. Candidate articles are output with key contextual information, including evidence, which speak to the probability a given scholar wrote a particular paper and allow a third party like a librarian to quickly decide if an article was authored by a scholar of interest.

ReCiter’s approach is distinct from those pursued by many disambiguation algorithms (e.g., Louppe et al. [12]) in that these applications are largely geared towards creating clusters of all co-authors. If a scholar has potentially written two papers, and both have five authors each, these systems will attempt to create groups out of the ten authors. ReCiter does perform clustering, but that is a relativvely minor subprocess. Also, with ReCiter, the number of potential matches is the number of articles and no more. This design decision is guided by the needs of consumers of our data who are ambivalent as to whether those ten co-authors ultimately represent ten people or five; they only want to know if the scholar in question wrote neither, one, or both of those articles.

The goal for ReCiter is to fully leverage feedback for whether a scholar wrote an article as well as the abundance of identity information institutions commonly maintain in various systems of record. Useful data includes (see Fig 4) a person’s past and present departmental, institutional, and program affiliations; email addresses; years of bachelor and doctoral degrees; inferred or preferred gender; grant identifiers; and the names of known relationships such as mentors, managers, co-investigators listed on grants, mentees, and individuals who share the same Human Resources-maintained organizational unit. Because this is an identity-driven approach, all such information can be employed to predict publications before the system has any confirmation that a scholar of interest has authored a paper. ReCiter outputs: the full citation of a candidate, or potential, article; up to 13 scores, only one of which is the average score of articles in its cluster; and the corresponding evidence for why an article was suggested. ReCiter’s conclusions are not derivative of any other service including ORCID ID, Scopus Author Identifier, etc.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Institutionally maintained metadata about a sample WCM scholar that can be used for programmatic authorship prediction.

Identity sources at academic institutions like Weill Cornell Medicine can provide a range of data. These data can be used to programmatically compute the likelihood a given article was written by a given scholar. For example, any known “relationships” are often the names of potential co-authors.

https://doi.org/10.1371/journal.pone.0244641.g004

2.3. Algorithm

ReCiter identifies, retrieves, and scores candidate records in PubMed for an individual scholar. This entire process is coordinated by ReCiter’s Feature Generator API. The Feature Generator API negotiates with other ReCiter APIs including the ones for identity retrieval, PubMed retrieval, Scopus retrieval (if one so designates), and Gold Standard retrieval. With Gold Standard API, one can download a list of identifiers of articles that have been accepted and rejected by a user. While one can interact with these APIs individually, it is a best practice, at least in a production environment, to allow the Feature Generator API to do this. Feature Generator API can output article metadata, scoring, and qualitative descriptions providing a rationale for scoring (e.g., a candidate record is indexed with a grant matching one in the Identity table). See Fig 5 for an overview of the algorithm’s workflow and S1 Appendix for a sample output of this API.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Overview of the ReCiter algorithm.

The Feature Generator API coordinates the following key steps in the ReCiter workflow: retrieve candidate records, identify target author, cluster articles, score articles, and output results.

https://doi.org/10.1371/journal.pone.0244641.g005

Search queries are constructed by using the name(s) recorded in the Identity table in DynamoDB. The standard syntax for a name search consists of a verbose last name and first initial, e.g., Albert P[au]. ReCiter will also search for additional candidate records in cases where a scholar has multiple name aliases or a compound surname. For example, the scholar, Gabriel Garcia Marquez, would be searched as Marquez G[au] OR Garcia G[au]. This approach identifies potential articles where scholars may have a maiden name or their middle name is accidentally comingled with their surname, which sometimes occurs in the case of Arab and Latin names. If a user has different names in identity source systems (e.g., one captures a maiden name, another does not), ReCiter will look up each of those names independently.

If the count of candidate records exceeds a set amount–the current default is set to 2,000 –ReCiter does not retrieve any records and instead goes into its “strict” mode to narrow down the result pool. In strict mode, the standard last name, first initial search is made in conjunction with each of six additional search strings. These search strategies are: searching by verbose first name, first and middle initial, names of relationships, grant identifiers, departments, affiliated departments, and affiliated institutions. For example, one of the six strict searches for a scholar named “Xu Xu”, who is in the department of Radiology would be: Xu X[au] AND Radiology[affiliation]. ReCiter also uses strict mode against inferred last names. If a scholar’s last name has a space or dash (e.g., Garcia-Marquez, Garcia Marquez), the system will look up "Garcia" and "Marquez" independently but paired with the additional search strings. This mode is known as STRICT_COMPOUND_NAME_LOOKUP.

By default, all candidate articles have a user assertion of null, which reflects the fact that no human curator has asserted whether the article should in fact be assigned to the scholar of interest. A user can provide feedback to accept or reject an article suggestion. They can also revert previously accepted or rejected articles to null. ReCiter records user feedback including accepted and rejected articles in its GoldStandard table. If the user or agent requesting the lookup for a scholar selects AS_EVIDENCE for the use-gold-standard parameter, records from the GoldStandard table are also retrieved. If FOR_TESTING_ONLY is selected, ReCiter will only retrieve and score articles retrieved by the name lookup.

Once a pool of candidate articles is retrieved, additional data about a given article are optionally retrieved from Scopus by doing a DOI search against its API. These data include Scopus institutional identifiers, verbose author name, and citation count.

ReCiter uses metadata from PubMed and Scopus to output a single canonical publication type for each article, and also outputs a standardized/sortable date of publication and a display date. For the purposes of clustering and scoring, ReCiter will standardize all names, removing suffixes, dashes, and diacritical marks such as accents.

For every retrieved candidate article, ReCiter uses a set of heuristics to identify the author who is most likely to be the target author. Our approach has some overlap with the heuristic proposed by Torvik and Smalheiser [13]. According to our analysis, our system is able to identify the correct target author more than 95% of the time. The correct assignment of target author, even in cases of a true negative, can be quite helpful for accurately scoring the candidate articles. Clustering uses target author assignment as do six of the scoring strategies. It is also useful for business reporting. The Office of Research asks these authors to see all the papers in the prior month appearing in “top” journals where a principal investigator was senior author.

In the clustering stage, ReCiter uses an unsupervised rule-based agglomerative algorithm to create groups of articles written by the same person. There are two steps to clustering: “tepid clustering” and “definitive clustering.”

With tepid clustering, we identify features, which are somewhat uncommon (frequency of around 100,000 records or fewer out of a corpus of 40+ million). The features used in tepid clustering are: journal name; co-author name excluding cases where an author is identified as a target author and extremely common co-author names [13, 14]; Medical Subject Headings (MeSH) major term where count in the MeSHTerm table is < 100,000; and Scopus Affiliation ID for target author. By themselves, these features are not positively conclusive. But, if a certain proportion of features between any two articles share these features, we can conclude they were written by the same person, and they are clustered.

Clusters are merged if the cluster-cluster-similarity-score-threshold, as stored in application.properties, is less than the computed value based on raw counts and overlap from those clusters:

cluster-cluster-similarity-score < (cluster1 ⋂ cluster2)² / (count-of-items-in-cluster1 * count-of-items-in-cluster2)

For example, suppose there are two clusters:

{

          Cluster Id: 1

          Journals: Cell;

          MeSH major: Thalassemia, Sunlight, Tamoxifen, Tryptophan, Brain;

          Coauthors: Marshall T, Michaels A;

          Scopus institutional affiliation identifiers for target author: 6007997;

}

{

          Cluster Id: 2;

          Journals: Cell;

          MeSH major: Thalassemia, Tamoxifen;

          Coauthors: Marshall T, Michaels A, Johnson Q;

          Scopus institutional affiliation identifiers for target author: 6007997, 342823053;

}

The system would conclude these clusters should be merged as 0.2 < (6^2) / (9 * 8). The clustering is designed to work even if Scopus affiliation identifiers are not available.

With definitive clustering, we look to combine clusters where features generally occur thousands or fewer times in a corpus of 40 million records. Because they occur so infrequently, we will merge clusters whenever there is a single such piece of evidence. Any article that shares any of these features with another article should be in the same cluster as that other article: email; grant identifiers excluding papers that mention more grants than listed in a threshold in application.properties; cites or cited by papers; MeSH major where global raw count in MeSHTerm < 4,000.

Then, the candidate articles are individually scored. ReCiter is object-oriented and uses a strategy design pattern [15] in which each evidence type is modeled as a different strategy and appears in a separate module within ReCiter. In the first round of scoring, a given candidate article is scored using 12 criteria, which we hypothesized are correlated with whether an article would be accepted or rejected.

• target author’s name—similarity of name recorded in the Identity table compared to target author’s name
• organizational unit—whether any departments, divisions, etc. in Identity appear in the affiliation statement of the target author
• email—if the email in Identity appears in a target author’s affiliation statement
• department-journal category—a score that reflects the extent to which a scholar’s organizational unit (e.g., Orthopedic Surgery) is highly associated with the ScienceMetrix category [16] (e.g., Orthopedics) of the journal
• target author’s institutional affiliation—if affiliation recorded in Identity is consistent with affiliation listed in the author affiliation statement for the target author
• non-target author’s institutional affiliation—if affiliation recorded in Identity is consistent with affiliation listed in the author affiliation statement for any author except the target author
• relationship—if individuals known to have worked with a scholar, including those who are listed on a shared grant or have served as a mentor, match any co-author names
• grant—if a candidate article is indexed with an NIH grant identifier recorded in the scholar’s Identity profile
• education year—effect scholarly age has on the likelihood an article in a given year was written by a scholar; especially penalizes candidate articles that are published far earlier than expected; works on a sliding scale
• person type—factors into account how often different types of scholars (e.g., faculty, postdoc, medical student, etc.) are likely to author an article
• article count—a score for how many candidate articles were identified for a given author; score increases when there are fewer candidate articles and decreases when there are more candidate articles
• gender—a score, which uses Social Security Administration data [17], to infer the likelihood a name in the Identity table shares the gender with that of the name of the target author

In some cases such as email-match-score, the score is a constant stored in ReCiter’s configuration file, application.properties. With the "article count" evidence, consistent with Bayesian insights about probability, we reward individual candidate articles, in which there are few articles and penalize cases in which there are a lot. We do this for each article using three values:

• count-articles-retrieved–count of the articles actually retrieved
• article-count-threshold-score–as stored in application.properties, e.g., 800
• article-count-weight, as stored in application.properties, e.g., 200

The value for article-count-score for each article is equal to:

(article-count-threshold-score–count-articles-retrieved) / article-count-weight

Here is sample output for a scholar’s ("C. Cole"), article count score, where the scholar has 1,151 candidate articles:

"articleCountEvidence": {

    "countArticlesRetrieved": 1151,

    "articleCountScore": -1.755

},

In other scores, the scoring logic is more complicated. The score for a target author’s name is the sum of individual subscores for first name, middle name, and last name. In scoring first name, the algorithm attempts to identify which of the 10 types of match between name metadata in the Identity table and the given name of the target author is highest. Matches that are more tenuous have lower scores. For example, the strongest match would be full verbose name (“Terrie” vs. “Terrie”). Less strong matches include those based on inferred initials (“Paul J.” vs. “PJ”), a fuzzy match (“Michael” vs. “Michel”), and a case where initials are present but in an unexpected order (“JD” vs. “DJ”). The weakest scores correspond to cases where names conflict partially (“Ximeng” vs. “Xu”) or entirely (“Curtis” vs. “Ronald”). The algorithm also looks for various other cases where first and middle name may be inverted in the article metadata or middle name may be combined with last name. S2 Appendix describes this logic in greater detail.

We then compute cluster-score-average, the average score of all articles in a cluster. If the user requesting the lookup for a scholar selects AS_EVIDENCE for the use-gold-standard parameter, the presence of accepted articles in a cluster increases the cluster-score-average by a value set in application.properties while the presence of rejected articles decreases this value. Conversely, when the FOR_TESTING_ONLY parameter is selected for use-gold-standard, there is no such effect.

If the cluster-score-average is higher than that of the target article (total-article-score-without-clustering), the target article’s total score increases. If it is lower, it decreases. We also compute a cluster-reliability-score, in which ReCiter gauges the extent to which a target author’s verbose first name, where it exists, is consistent within a given cluster. If it is inconsistent (e.g., a given cluster contains articles with target authors that have the first name of RockBum, RaeKwon, RulBin, and RyoonHo), we downweight this effect. The sum of all the different evidence subscores is total-article-score-nonstandardized, a value which is mapped to a standardized score (total-article-score-standardized) between 1 and 10.

The Feature Generator API allows users to choose whether to restrict article retrieval to those added since the last time ReCiter retrieved articles for a scholar. Also, administrators can elect whether to return articles of a particular status (e.g., “accepted”, “rejected”, “null” in which any feedback has yet to be provided, or some combination thereof) and which have a minimum score.

For each article, ReCiter outputs one of four designations as seen in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Significance of accuracy designations in Feature Generator API output.

https://doi.org/10.1371/journal.pone.0244641.t001

The following scholar-level measurements of accuracy are computed and output:

precision = TP / (TP + FP)

recall = TP / (TP + FN)

accuracy = (TP + TN) / (TP + TN + FP + FN)

Measurements of accuracy can depend on several factors. Name origin is widely discussed in the literature, but academic age is especially important for algorithms that perform authorship prediction. We could test ReCiter’s accuracy only by using longstanding faculty at Weill Cornell in cases where we have a rich set of identity data and a lot of publications to show relationships. In such cases, ReCiter may perform relatively well. For those scholars early in their careers, the algorithm is generally less forgiving.

2.4. Evaluation

2.4.1 Creating and using a support vector machine-based training model.

The original set of weights used to compute the 13 scores for any given candidate article were no more than the product of developer intuition, and trial and error. To improve the accuracy and overall performance of ReCiter we sought to optimize these constants. To do so, we randomly selected a group of 500 Weill Cornell-affiliated scholars, which consisted of full-time WCM-employed faculty (n = 310), affiliated faculty (n = 82), postdocs (n = 68), and third- or fourth-year M.D. students (n = 40). We purposefully targeted a heterogeneous group including types of scholars such as M.D. students who are unlikely to have authored any more than several publications. If our goal was to maximize accuracy for this analysis, we could have limited ourselves to only principal investigators and other well-established senior scholars. Our approach at least partially guards against the possibility that ReCiter could be overly aggressive in suggesting publications. It also tests the accuracy of an authorship prediction system in cases when there is sparse identity data for a scholar, and there are fewer signals to leverage. We subdivided candidate articles for our 500 individuals of interest into a training group and a test group. The training group represented 80% of all candidate articles, and test group represented 20% of all candidate articles. For privacy concerns, we are not able to share the identifying details of the 500 selected scholars without securing permission from each of them. However, we do want to inspire some confidence that these names have not been selected in a self-serving way. S3 Appendix includes the surnames and first initials of the 86 scholars who have the most common names as defined by anyone who had 2,000 or more candidate articles.

ReCiter estimates the likelihood a given scholar authored a given article by using personally identifiable information including personal email, names of mentees, etc. As such, it is not possible for us to share the raw data.

To create our gold standard of known publications, we started with any assertions about author identity for our 500 individuals available in our legacy system. Starting in 2014 to the present, the publications authored by full-time WCM-employed faculty, affiliated faculty, and postdocs, have been recorded and updated in an institutionally maintained MySQL database containing approximately 185,000 publications. Data from this system is repurposed in a variety of forums and has received feedback through a number of touchpoints. Publication data are published on users’ public-facing VIVO scholarly profiles. They appear in T32 training grants applications [18] and in reports for the Dean, Office of Research, department chairs, Office of External Affairs, Institutional Reporting, and various educational program offices. When full-time faculty start at Weill Cornell Medicine, librarians manually curate lists of all these individuals’ publications. Missing or incorrectly assigned articles frequently elicit feedback sent to the Library and are manually corrected in that database. All full-time WCM-employed faculty are required to review their publications on an annual basis. This feedback may include group authorships (e.g., “HI-TECH Investigators”) in which there is no indication other than an individual scholar’s assertion that a given scholar contributed to a paper. Publications authored by MD students were reviewed by the Office of Medical Education.

All assertions were manually reviewed by two librarians who updated publications lists for all individuals between October 28 and November 21, 2019. A second librarian identified 45 articles that were incorrect. Of these, four were reverted. Any candidate articles that were not accepted were implicitly asserted to be rejected.

Of the 16,330 publications ultimately judged to be authored by our 500 scholars of interest, 79.8% were originally recorded as such in the legacy reporting database. This figure increases to 88.1% when only considering the 310 in scope full-time faculty. We attribute this higher figure due to the greater level of scrutiny in maintaining publication lists for full-time faculty. By comparison, publications authored by affiliated faculty are typically added only by explicit request. While some of the discrepancies between the two systems are the result of curation errors, especially errors of omission, a non-trivial portion of the differences are the result of Scopus incorrectly crosswalking records from PubMed to Scopus. This is particularly common in the case of comments and other types of publications that refer to another publication.

For all in-scope individuals, we used a Bash script (see S4 Appendix) to query ReCiter’s Feature Generator API and output the result including article metadata, evidence, and scoring to a series of JSON files. For the use-gold-standard option, we selected the FOR_TESTING_ONLY parameter. Typically, the Feature Generator API is set to only store articles of a minimum score (this cuts down on storage costs by approximately 85%), so we could not export data directly from our Analysis table. We then used the ML_Model_Test_upload.py (see S5 Appendix) Python script to transform the JSON files into individual flat comma-separated values (CSV) files, which can be loaded into a database (see S6 Appendix).

We considered and experimented with several possible machine learning approaches to optimize the weights of the constants in the application.properties configuration file: random forest, linear regression, and support vector machine. Because our data set has far more rejected articles than accepted articles, we opted to use a support vector machine (SVM) analysis [19], which has a reputation for being more robust in the presence of unbalanced data. SVM can produce results that are explained when using a linear kernel, and we can incorporate the model results into the original linear regression system. Also, it is reputed to be a strong choice for performing binary classification [20]. All the following operations are included in a Jupyter notebook, available in S7 Appendix.

In order to use weights calculated by the model, we used Linear SVM [21]. This library fits the dataset and generates a hyperplane for dividing the data into two groups: those articles the algorithm concludes were written by the scholar of interest and those the system concludes were not. Using Linear SVM, we can also identify the weight and intercept of this hyperplane in the model output, and infer the importance of each feature. By using a decision function, which is calculated based on weight and intercept, we are able to predict confidence scores for samples. The confidence score for a sample is the signed distance of that sample to the hyperplane. With a higher absolute value of confidence score, we are more certain with the prediction of the sample.

Confidence score was computed using the below equation, in which X_test1[i] is candidate articles’ feature score, clf.coef_[0] are the weights generated by the model, and clf.intercept_[0] is the intercept generated by the model. The np.matmul function is the sum of the features multiplied by their respective weights.

It is generally preferable to first normalize data and then feed it into an SVM model. But in our case, we did have an a priori assumption of the importance of features, so we concluded that it would be acceptable to preserve the original scale. Also, we experimented with tuning the regularization parameter of the SVM model, with values from 0.0001 to 100, increasing by a multiple of 10. We determined that the regularization parameter of 1 provides the optimal value.

In order to test the robustness of the parameter weights output by SVM, we used cross-validation. Additionally, we tried the Pearson Correlation and Random Forest feature selection methods. With Pearson Correlation, we computed and output the correlation between each feature to determine if any features were so highly correlated we could conclude they may be redundant. With Recursive Feature Elimination, we used SVM as a model. Finally, we experimented using Random Forest and looked at the embedded feature importance. We tried to adjust the weight generated from SVM manually by incorporating the feature ranking from all of these three models, but these changes did not improve the system performance, so we only used the weight generated by Linear SVM.

We updated the weights in the application.properties file (see S8 Appendix), multiplying the existing values by the values output in this exercise. We then re-ran ReCiter for all candidate articles in our test set. We then sampled the test set and compared the new system’s performance with the old one.

We produced two different approaches to evaluate the system’s ability to predict scholars’ publications. The first was to calculate performance of all candidate articles for all 500 scholars. In this method, articles are pooled prior to analysis. With the second approach, performance is first computed by individual person and results are then averaged. In both cases, we choose a threshold value for total-article-score- nonstandardized that would yield the highest overall accuracy. Going forward, except where otherwise indicated, we will use the term, "article score" synonymously with total-article-score-nonstandardized.

5.1. Key findings

With the latest version of ReCiter, we have successfully created a publication management system, which is capable of predicting the publications authored by a diverse group of scholars at Weill Cornell Medicine with approximately 98% accuracy. Our system can lessen the workload for both administrators and scholars, providing publication metadata that can be used across public-facing profiles, reports, biosketches, and MyNCBI profiles.

ReCiter is a highly accurate, open source publication management algorithm designed for academic institutions with limited need for custom development. It innovatively uses a range of institutionally-maintained identity data to suggest which publications in PubMed a scholar has authored. Each of its components are needed for adoption by institutions. Additionally, because the key components are available as individual services, one could swap out one for another.

ReCiter retrieves and scores candidate articles for a given scholar, including those with unexpected or missing affiliations, as well as cases where article metadata is missing or highly misleading. ReCiter effectively uses the weights of a support vector machine analysis to score the available evidence.

ReCiter uses a wealth of institutionally-maintained identity data to suggest and score articles; provides an evidence-based probability for each of the candidate articles; offers a human-readable explanation for why a given article was suggested; and, reliably identifies a target author’s rank or position.

5.2. Weaknesses

While a significant subset of records in the Gold Standard had been validated by administrators and the scholars themselves including during the College’s annual review cycle, this analysis relies on a gold standard reviewed and updated by third parties. Ideally, all of the publication assertions would be validated by the scholars themselves. Scholars’ reluctance to curate their publications in a consistent and timely manner is ironically the very driver for the creation of this software.

ReCiter uses a variety of institutionally-maintained identity information to score its suggestions. We have attempted to construct a data model generic enough to accommodate the range of identity data maintained by a diverse array of institutions. For example, the relationships attribute could be populated with any variety of cases where a known association is likely to appear as a co-author. Yet, the most valid test of the durability of this model can be proven when this algorithm is used more widely.

One potential shortcoming is that many institutions may not have access to much of the identity data available in its identity data model. As a result, the accuracy of their system’s suggestions could suffer. One idea we contemplated is programmatically identifying and proposing candidate features such as email addresses, ORCID identifiers, name aliases, names of co-authors, etc. and then soliciting feedback from users. As Weill Cornell Medicine assists our Clinical Translational and Science Center in identifying publications for thousands of individuals about whom we have little data, this measure may significantly help.

Another challenge relates to what a fair method should be for computing precision for individuals with very common names. Yi Wang is a faculty at Weill Cornell. As of July 2020, a PubMed search for Wang Y[au] returns approximately 157,000 candidate records, with 3.7% of PubMed articles [33] having a “Y. Wang” as an author. Suppose a Yi Wang has authored only two papers, and we correctly identify those two and erroneously recommend 40 additional publications for this scholar. If ReCiter made a practice of counting those 157,030 records as true negatives, the recall would be 100%, the precision almost 5%, and the overall accuracy would be over 50%. For many, that figure may seem misleading. There may be no ideal approach for reporting ReCiter’s precision for common names. For now, ReCiter only counts publications from articles that are actually retrieved and scored–unless they are false negatives. Such articles are factored into the overall accuracy score even if they are not retrieved.

We note that a subset of publications that we judged to be correctly identified by ReCiter were not recorded in the legacy reporting database. However, this was relatively uncommon, as we discuss in the Methods, occurring for 20.2% of papers that were ultimately accepted. Less than 1% of papers were asserted to be authored by a scholar in our legacy database, but upon review by a librarian, were judged not to be. Ideally, the authority for which scholars wrote which papers would be curated entirely outside of the system being evaluated. We considered excluding all such papers from the analysis, but, we decided against that. We reasoned that we would never have access to both rich identity data and perfectly assigned publication metadata for the more than 16,000 papers authored by 500 scholars.

We recognize that ReCiter could more effectively use machine learning technologies when it comes to institutional disambiguation, assignment of publication type, and citation deduplication (i.e., Article A in PubMed refers to the same object as Article B in Scopus), among others. In particular though, we believe that ReCiter’s accuracy could be greatly improved by using a machine learning approach for clustering. At present, ReCiter does not use keywords from article title, journal title, or abstract. This absence represents a significant opportunity for improvement, especially for clustering.

5.3 Future work

Going forward, our team intends to deploy integrations with additional bibliographic sources. These integrations will also insulate developers from the complexity and the quirks of unwieldy source systems, and return data in a predictable, easy-to-parse format.

We have heard from a number of parties who are interested in generating suggested lists of scholarly articles from bibliographic sources that offer coverage of the social sciences and humanities. At present, ReCiter is optimized for PubMed, so this would require some dedicated effort. A source that had more complete cites and cited by data would be particularly attractive as ReCiter only uses cited by data recorded in PubMed Central, which contains less than 20% of PubMed.

For this analysis, we did not analyze how accepted and rejected publications affect the overall score of other candidate articles. These data are used in the average-clustering-score, described above, but they did not figure in our analyses. Anecdotally, we have noticed this feedback definitely improves the accuracy of scores of articles that were not accepted or rejected. The net effect is that fewer candidate articles have ambiguous authorship.

We realize that the validity of the approach we describe here would be more compelling if we could demonstrate that ReCiter can produce accurate lists of publications at other academic institutions, and show that implementing ReCiter saves administrators’ and scholars’ time, and results in more accurate outcomes. We intend to study these questions in future efforts.