Evolution of scientific publishing

Bibliometric indicators are often used to measure scientific progress, evolution, and research [52, 53]. Fig 2 showcases the use of Alexandria3k to chart a view of scientific publishing evolution in the post-WW2 period. It shows an exponential rise in the number of published works (CAGR—compound annual growth rate = 5.4%; R² = 0.99) and in the number of journals (CAGR = 5.1%; R² = 0.99), in line with what is reported by the trade association study [54, p. 27]. Despite this rise, publications are becoming ever more connected by citing each other. This can be seen in the rises of the references each work contains, the citations works receive, the phenomenal proportion of all works ever published that are cited at least once every year (20%), and corresponding rises to the 2-year and even 20-year global impact factor. Authors appear to be collaborating more and on longer papers with only a slight decrease in the mean number that they publish each year. Papers published in journals that are open access at publication time have risen exponentially (CAGR from 1990 = 0.23%; R² = 0.99) to reach 22% of all journal papers, a ratio that is also mirrored by a published figure [54, p. 8]. The fall in the consolidation/destabilization (CD) index is in line with recently published research reporting that papers are becoming less disruptive over time [55]. There is significant correlation (Spearman rank-order correlation coefficient 0.95; P < 0.001) between the CD₅ index yearly averages obtained using Alexandria3k with those available in the previously published dataset, which was obtained from data that are not openly available. The sharp inflections in the Figure probably stem from artifacts of the underlying data set, and indicate that in some cases obtaining scientifically robust results might require deeper analysis of the data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Evolution of scientific publishing metrics in the post-WW2 period.

https://doi.org/10.1371/journal.pone.0294946.g002

A trend related to this work is the rising number of research synthesis studies. We found about 437 thousand scientific studies published from 1846—the year of the first one we found [56]—onward based on the analysis of previously published primary studies. The number of synthesis studies published each year has risen considerably over the past two decades, particularly for systematic literature reviews (see Fig 3). These were typically identified by their titles through terms such as: “systematic review”, “systematic literature review”, or “systematic mapping study” (secondary studies using methods that help make their findings unbiased and repeatable—251 850 titles); “secondary study”, “literature survey”, or “literature review” (a not necessarily systematic study reviewing primary studies—77 037 titles); “tertiary study” or “umbrella review” [57] (a study reviewing secondary studies—4 039 titles); “meta-analysis” [58] (a systematic secondary study employing statistical methods—92 363 titles); as well as (systematic by definition) “scientometric” (employing quantitative methods to study scientific research—2 769 titles), “bibliometric” (using statistical methods to study written communications—12 361 titles), and broader Science of Science (SciSci—studies of scientific processes based on large data sets) [59] studies [58].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Number of research synthesis studies published each year in the period 1971–2021.

https://doi.org/10.1371/journal.pone.0294946.g003

The size of scientific fields can affect canonical progress [60] while the corresponding size dynamics can be associated with a field’s evolutionary stages [61]. Associating publications with the general scientific field of the journal they were published (according to the Scopus All Science Journal Classification Codes—ASJCs) allows Alexandria3k to generate a view of the evolution of the ASJC 27 fields over the years. Fig 4 shows the ten fields amounting to more than 2% of publications in 2021 that had the largest change in their publication number in the period 1945–2021. Clearly visible is the expected rise in Computer Science, Materials Science, and Physics and Astronomy, as well as the fall of Arts and Humanities and Social Sciences publications. Owing to the exponential rise of published works (tallied on the Figure’s right-hand axis) the falls are only in relative terms: 2021 saw the publication of 291 366 Arts and Humanities articles against 19 873 in 1945.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Evolution of subject coverage and publications 1945–2021.

https://doi.org/10.1371/journal.pone.0294946.g004

COVID-19 research

The bibliometric study of a specific scientific topic, such as COVID-19 [62], can aid progress by providing details and insights on the state of the art and the community working on it [63, 64]. To showcase how Alexandria3k could help study research related to COVID-19 [65] we used Alexandria3k to create and study a data set of publications containing COVID in their title or abstract. Our findings are in broad agreement with studies using different research methods [66, 67].

We counted 491 945 publications from about 1.5 million authors. These covered 331 different topics demonstrating the many disciplines associated with the research. Some noteworthy topics and work numbers among those with more than one thousand publications include General Medicine (rank 1—70 609 works), Psychiatry and Mental health (rank 4—10 404 works), Education (rank 5—9 590 works), Computer Science Applications (rank 18—6 013 works), General Engineering (rank 20—5 942 works), Strategy and Management (rank 42—3 208 works), Law (rank 57—2 557 works), History (rank 62—2 329 works), Cultural Studies (rank 76—1 893 works), Pollution (rank 97—1 549 works), and Anthropology (rank 130—1 032 works). Looking at listed funders, we saw that the top three in terms of associated publications were the National Natural Science Foundation of China (3 506 works), followed by the (US) National Institutes of Health (2 316), the (US) National Science Foundation (1 022), the Wellcome Trust (914), and the (UK) National Institute for Health Research (661). We also examined the affiliations of COVID study authors, propagating them to the highest parent organization (e.g. a university hospital to its university). Through this measure the top five entities were the Government of the United States of America (1 465 works), the University of California System (925), University of Toronto (910), University of London (824), and University of Oxford (660).

We also looked how long it took for COVID articles to start citing each other, building, as it where, “on shoulders of giants”. As can be seen in Fig 5 a citation network ramped up relatively quickly, surpassing ten thousand citations to COVID research by April 2020, and reaching 118 thousand citations on March 2022. The large number of works published in January 2020 appears to be due to journals publishing later in the year volumes with that date. This was, for example, the case with an overview article on COVID-19 and cerebrovascular diseases [68] and an eight month retrospective [69]. (The same phenomenon may also explain the January 2021 rise.) Notably, this backdating practice can distort the establishment of priority over scientific advances based on a journal’s publication date.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Citations from COVID research to COVID research over time.

https://doi.org/10.1371/journal.pone.0294946.g005

Finally, we examined the heightened role of preprints in COVID-19 publications [70]. In the COVID data set from the 491 945 published works, 48 211 (9.8%) were identified as preprints. Contrast this with all the works published in the same period, where from the 16 369 968 published works, 572 200 (3.5%) were identified as preprints.

Impact and productivity metrics

We replicated diverse impact and productivity metrics typically calculated in a proprietary fashion by commercial bibliometric data providers.

The Journal Impact Factor (JIF) and Hirsh’s h-index are often used as (imperfect) citation-based metrics of a venue’s quality and impact [71, 72]. We calculated the 2021 JIF, and compared the 600 journals with the highest JIF available through Clarivate with 581 we matched through their ISSNs, obtaining a Spearman rank-order correlation coefficient of 0.72 with P < 0.001. We also calculated the journal h5-index, and compared the values of the 100 “Top publications” venues listed in Google Scholar [73] against those of Alexandria3k by hand-matching the venue titles. The comparison of the 91 matched elements gave a Spearman rank-order correlation coefficient of 0.75 with P < 0.001. Examining the same metric in a field where considerable work is published in conferences, we compared the h5-index of the 13 common venues between the 20 Google Scholar reports in the “Software Systems” category [74] against a curated list of 32 software engineering venues [75], and obtained a Spearman rank-order correlation coefficient of 0.83 with P < 0.001.

Through the same data we also obtained the most cited articles published in the corresponding period (predictably, the top-one, with 21 426 citations, was on COVID-19) [76] and overall (the top one was a 1996 article on the generalized gradient approximation method [77], which received 39 715 citations from a surprising diversity of fields). A Clarivate Web of Science query performed on May 20 2023 found the same articles ranked 5th and 4th correspondingly in terms of citation numbers.

A widely-cited study has reported a citation advantage of open access articles [78], though followup studies drew a more nuanced picture [79]. We looked at the corresponding advantage of articles published in open access journals. A comparison of the population of citation numbers to articles published from 2011 onward in open access journals and corresponding ones to articles that were not gives a negligible effect size (Cliff’s d = 0.02; Cohen’s d = −0.035), and also indicates that publications in non open access journals are cited more (mean 8.6) than the ones that are published in open access ones (mean 7.1). This finding is in line with the one reported by a longitudinal study that found citations associated with journals funded by article processing charges to have reached the same level as subscription journals, but to be lower for journals funded by other means [80]. On the other hand, a later study examining the relative influence of a journal’s impact factor reports open access journal publications to have significantly more citations overall compared to the rest [81], so clearly more research is needed on this front.

Hirsh’s h-index was originally proposed as a metric to quantify an individual’s scientific research output [82]. We demonstrated this use of Alexandria3k by obtaining the h5-index on authors identified through ORCID. A noteworthy observation was the number of authors with a high metric: the top-ranked author has an h5-index of 76, twelve authors have an h5-index larger than 60, and 100 larger than 38. These achievements appear to be even more difficult to explain than earlier observations of hyperprolific authors [83]. We explored the phenomenal productivity and impact exhibited by the authors at the top of the distribution by examining the clustering coefficient of the graph induced by incoming and outgoing citations of distance 2 for a given work. The clustering coefficients of a random sample of 50 works from authors with an h5-index larger than 50 (median coefficient 0.05) appear to be significantly different from a random sample of other works of the same size with the same number of citations for each one (median 0.03). Specifically, comparing the two populations we obtained a Mann-Whitney U statistic measure of 781 for the coefficients of the top-ranked authors’ works with P < 0.001.

Implementation

Alexandria3k is designed around modules that handle the Crossref, ORCID, ROR, and flat file (journal names, funders, open access journals) data sources. Each module defines the data source’s schema in terms of tables and their columns. The schema’s SQL DDL implementation is used to define the corresponding tables in a populated database. The schema is also used for analyzing and satisfying vertical slicing requests when populating databases with Crossref and ORCID data, and for running Crossref queries without populating a database. Where possible, data are on-the-fly decompressed, extracted from an archive (with a tar or zip structure), and parsed in chunks, thus avoiding the storage cost of the entire decompressed data set (1 TB for Crossref and 467 GB for ORCID).

In its simplest form Alexandria3k can evaluate an SQL query directly on the Crossref dataset, often to perform exploratory data analysis. Results can be saved as a CSV (comma-separated values) file or iterated over through Python code for further processing. Both of these modes have limitations in terms of performance, aggregation of query results, and combination of data from multiple sources.

In most cases Alexandria3k is used to populate an SQLite database [88]. Multiple alternatives were considered for the data back-end. The native Crossref tree format would suggest a JSON-style NoSQL database. Redis [89] would be an efficient back-end, but would limit the amount of data to that that could be processed in memory. MongoGB [90] would address the capacity limitation, at the cost of a more complex installation process due to licensing issues. A client-server relational database system could also be used to offer improved integrity constraints and more advanced query optimization facilities. However, these options require the installation, configuration, connection, and maintenance of a separate database process, which is not a trivial task, especially for researchers outside the computing field. Consequently, the adopted approach uses the SQLite database, which is embedded into Alexandria3k, directly available in Python, easily installable as a command-line tool in all popular computing platforms, and accessible through APIs for diverse programming languages and environments. Transferring a database between computers only involves copying the corresponding file. Consequently, there is no need to setup, configure, and maintain a complex client-server relational or NoSQL database management system. Despite its -Lite suffix, SQLite supports large parts of standard SQL [91] (including window functions and recursive queries), and employs sophisticated query optimization methods; these feature made it ideal for use in Alexandria3k. SQLite’s main downsides—lack of multi-user and client-server support—are not relevant to common Alexandria3k use cases.

Direct queries on the Crossref data set (without populating a database) are implemented by defining SQLite virtual tables that correspond to the offered schema through the Python apsw module. Crossref data are distributed in the form of about 26 thousand compressed containers. Running a query on them is trivial when the query accesses a single table: the Alexandria3k virtual table implementation moves from one container to the next as the table is scanned sequentially.

Direct Crossref queries involving multiple tables are handled differently. First, a dummy query execution is traced to determine the tables and fields it requires. Then, in-memory temporary tables are populated with the required data from each container, and the query is executed repeatedly on the instantiated tables. This approach works for all cases where relational joins happen within a Crossref container (e.g. works with their authors or references). More complex cases, such as relational joins between works or aggregations, require the population of an external database.

The database population design uses two techniques to improve its performance. First, records are bulk-inserted in batches by attaching directly the virtual tables to the database to be populated, and by having the database engine perform the insert operations with a single internal command for each batch. This avoids a round-trip cost of obtaining the data in Alexandria3k and then storing them back in the database. Second, database indices over the containers in which the data are split are implemented and used so as to access each container file in turn for populating all required tables. The correspondingly improved locality of reference [92] is then utilized by caching the decompressed and parsed file contents. Unfortunately, multithreading cannot be easily used to further improve performance because both Python and SQLite make it difficult to obtain the required concurrency.

The performant matching of author affiliations with RORs is based on multiple applications of the Aho-Corasick string-matching algorithm [93]. First, an Aho-Corasick automaton is created with all unique organization names, aliases, and acronyms. Second, the automaton is used to find entries in it that also match other entries (e.g. “ai”, the acronym of the “AI Corporation”, also matches the organization name “Ministry of Foreign Affairs”), and mark them for removal to avoid ambiguous matches. Finally, a new automaton, constructed from the cleaned-up entries, is used to find the longest match associated with each author affiliation string. This is stored in the database as the affiliation’s organization identifier. When the Alexandria3k user specifies that affiliations should match the ultimate parent organization, a recursive SQL query adds a “generation” number to each matched organization, an SQL window (analytic) function [94] orders results by generation, and a final selection query obtains the ROR identifier associated with the most senior generation.

Proof-of-concept studies framework

We structured most proof-of-concept studies we presented as a series of queries that build on each other. This aids comprehensibility, testability, analysability, and recoverability. We specified the corresponding workflow using Makefiles [49] based on the simple-rolap system, which manages relational online analytical processing tasks [50, 51]. The simple-rolap system establishes the dependencies between queries and executes them in the required order. Most studies start with a population phase, which fills a database with the required horizontal and vertical data slices. In many cases, we used the rdbunit SQL unit-testing framework to test SQL queries [51]. We employed a shared Makefile with rules and configurations that satisfy dependencies required by more than one study.

In total the proof-of-concept studies distributed as examples with Alexandria3k comprise 107 SQL query files amounting to 1 876 (commented) lines of code. The query operations are organized by 18 Makefiles (511 lines), which populate the databases and execute the queries in the required order. The queries create 42 intermediate tables and facilitate their efficient operation through the creation of 48 table indices. The charts, tables, and numbers in this report are based on data produced by SQL queries that obtain their results from intermediate tables, from populated databases, or directly from the Crossref data set.

We created a vertical slice of the complete Crossref database, which we used for a number of purposes. The slice contains mainly the primary and foreign keys (including DOIs, and ORCIDs) of all entities, plus the publication year, author affiliation names, and work subjects, which are not normalized. We also ran Alexandria3k to populate the database with the Scopus All Science Journal Classification Codes (ASJCs) [95] and RORs, and linked work subjects to ASJCs and author affiliations to RORs.

We used the graph database to extract most of the database contents metrics provided in the main text and in Tables 1 and 2. This was done through simple SELECT Count(*) FROM table or SELECT Count(DISTINCT id) FROM table SQL queries.

Scientific publishing evolution

In common with other studies [55, 96] we limited our examination to works published after World War II, in order to avoid misleading comparisons with the markedly different scientific and publication environment that preceded it. We calculated most numbers used for plotting Fig 2 from the populated Crossref graph database. We obtained the number of published works and journals through SQL Count aggregations of the underlying data grouped by year. We obtained the ratios of authors per work and references per work by counting the corresponding elements of the associated detail tables and then obtaining SQL Avg aggregations grouped by year. We joined papers with their citations using the document’s DOI as a key and used this to calculate the two-year impact factor, the received citations per work, the twenty-year impact factor, and the proportion of all published works cited at least once each year. a twenty-year rolling sum over the number of works published each year. (To facilitate the required case-insensitive DOI matching, Alexandria3k converts internally all DOIs into lowercase.)

We calculated the number of pages per work by populating a database with the required work and author details, and by extracting the starting and ending page number from Crossref works that contain a dash in their pages field. This process excludes single-page works reported only with a single page number (rather than as a range with the same starting and ending page). We excluded from the data records with a starting page lower or equal to the ending page or those having more than 1000 pages, because the latter (164387 records) were often derived from data-entry mistakes, such as repeated page digits (e.g. 234–2366), as well as unusable data formats, such as 1744–8069–5–32.

We calculated a measure that can be used to track author productivity (works per author) despite clashes in author names, by taking advantage of the fact that we display productivity in relative terms (adjusting it to be 1 in 1950). In absolute terms authors with same names increase the productivity’s absolute value. (An author named Smith, Kim, or Zhu would appear extremely prolific.) However, assuming that the ratio of clashing names in the population does not change, the effect of duplicate names on the relative productivity measurements is cancelled out.

Note that an actual study of author productivity would need to test and control for the assumption we made, because the population’s composition might change over the years to include authors from ethnic backgrounds with more or fewer common name clashes, (for example, about 80% of China’s population shares the 100 most common Chinese surnames [97]) making the phenomenon more frequent or less frequent over time. Using Alexandria3k we obtained frequently-occurring names at the two ends of the examined period and found that in 1950 the three most frequent names were W. Beinhoff (161 works—0.10% of the total names), E. Rosenberg (149—0.09%), and F. De Quervain (115—0.07%); whereas in 2021 they were Y. Wang (63363—0.23%), Y. Zhang (57414—0.21%), and Y. Li (51792—0.19%). The different percentages at the two periods’ ends indicate that further controls for this change would be required, e.g. by measuring name clashes through ORCIDs while also considering variations in their adoption [98].

We calculated the CD₅ index [99] of Crossref publications by populating a database with their publication date and DOI, as well as the DOIs of the corresponding references. Given the available data, we were able to calculate the CD₅ index for six additional years (until 2016) compared to previous results [55], using the remaining five complete years we had at out disposal (2017–2021) to obtain the required citations’ window.

The CD₅ calculation proved to be computationally challenging. The processing for the already-published 1945–2010 range, taking into account even publications lacking reference data, required more than five days of computing and about 40 GB of RAM. Surprisingly, extending the range to 2016 increased the required time to 45 days. To address this we enhanced the original CD calculation algorithm implementation converting it to C++ [100], so at to use more efficient data structures and algorithms as well as multithreading. This allowed us to perform the CD₅ index calculation in 33 hours of elapsed time using 40 hours of CPU time and 49 GB of main memory.

To help other researchers build on this data without incurring the associated high computational cost, we have made the resulting data set containing the DOI and the CD₅ index for 50 937 400 publications in the range 1945–2016 openly available [101]. This improves upon previously available data [102], which extends to 2010 and only provides the time-stamp of each publication, without other uniquely distinguishing publication identifiers.

We calculated the evolution in the number of publications across scientific fields (Fig 4) by propagating the specific fields associated with each work in the Crossref graph database to the more general containing field. For that we used as general fields the Scopus ASJCs that ended in “00”, and as their sub-fields those that started with the same numeric prefix. For example, we allocated publications under the subject of “Catalysis” (1503) to “General Chemical Engineering” (1500). We then calculated total publications in 1945 and 2021, changes in the percentages of a field’s publications in terms of the total at the two time points, and included in the Figure the ten fields with the largest change whose publications amounted to more than 2% of the 2021 total.

We calculated the numbers associated with research synthesis studies by processing the output of Alexandria3k run on the Crossref data with an SQL query that matches specific words in publication titles. The query’s terms are structured to give precedence to the characterization of titles indicating a systematic review as such, classifying the rest as (unspecified) secondary studies. In Fig 3 we combined the plotting of bibliometric (BM) and scientometric (SM) studies, and did not plot figures for mapping reviews (MR), umbrella reviews (UR), and tertiary studies (S3)—6061 publications in total. We also did not plot the 400 identified studies published before 1971 as well as studies published after 2021, as only part of the year 2022 data were available. Note that works containing “bibliometric” or “scientometric” in their title may either employ these methods [103] or refer to them [104]. We used another query with the same terms to list the 30 studies published before 1950 and obtain the earliest one [56] among them.

COVID-19 metrics

To study COVID-19 publications we populated a database with a full horizontal slice of the Crossref data by specifying as the row selection criterion works containing the string “COVID” in their title or abstract (“covid” is not part of any English word). We also linked works to their subjects and author affiliations to the corresponding RORs. We obtained organizations publishing COVID-19 research by assigning author affiliations to works, and by counting both ROR-matched affiliations and unmatched affiliations as simple text.

We calculated the approximate number of researchers who worked on all COVID-19 studies by starting with the number of unique (author given-name, author surname) pairs in the set of all COVID study authors N_an. The number of true authors could be higher if many authors share the same name or lower if the same author appears differently (e.g. through the use of initials) in some publications. We address this by obtaining from the set of authors with an ORCID the number of distinct ORCIDs N_o, which is the true number of authors in that set, and the number of distinct names N_on, which approximates any bias also found in N_an. We then consider the true number of authors as

Note that the employed method’s accuracy is still vulnerable to ghost and empty ORCIDs that seem to be created by paper mills [105].

Journal impact factor

We calculated the 2021 Journal Impact Factor [71] by populating a database with the keys, ISSNs, publications years, and pages of works and their references published between 2019 and 2021. We then created a table associating works with journal ISSNs. From this table we obtained citations published in 2021 to works published in 2019–2020 (the impact factor’s numerator). We further filtered works to identify “citable” items, which Clarivate defines as those that make a substantial contribution to science and therefore do not include elements such as editorials and letters. For this we used as a rough heuristic works longer than two pages. (We also included works lacking a page range.) From the count of citable items per journal we obtained the number of publications published in the 2019–2020 period (the denominator). Finally, to compare our results with the numbers published by Clarivate we associated each impact factor metric with all ISSNs known for a journal (electronic, print, alternative), excluding the “alternative” ISSNs of one journal used as primary for another journal.

Open access publications

We calculated the yearly evolution of open access journal articles and their ratio to all journal articles by using the Directory of Open Access Journals (DOAJ) provided through Alexandria3k. We joined the Crossref works with DOAJ based on each journal’s ISSN to create a database table containing works published in open access journals after each journal’s open access start date. We then joined the open access papers with their Crossref metadata and grouped them by year to obtain the yearly evolution shown in Fig 2. We used the set difference between all journal papers and the previously obtained joined table of the ones published in open access journals to obtain the corresponding ratio. We did not employ the Mann-Whitney U statistic for comparing the two populations, because the Shapiro-Wilk test on them indicated that the citation numbers are not normally distributed (W = 0.115, P < 0.001).

Productivity metrics

We obtained the h5-index [82] productivity metrics by populating databases with data sliced vertically to include the keys of works and references and horizontally to include items published in the period 2017–2021. For the software engineering venue metrics we selected the examined conferences based on the DOI prefix assigned to the conference publications each year.

To study the citation graph of top-ranked authors we obtained a) a random sample of 50 works written by top-ranked authors, and b) a random sample of 50 works from all other publications, paired with the ones selected from the top-ranked authors to have the same number of citations as them. For each publication in the two samples, we created a separate graph containing the work w, the set S of the works w cites and the works that cite w, and then again the set S^′ of the works w^′ that cite or are cited by w ∈ S. The graph’s edges are citations from one work to another. We created each citation-induced graph with a Python program querying the populated database and employed the NetworkX [106] Python package to calculate the graph’s average clustering.

Performance and map of available code

Performance details of the tasks reported here are summarized in Table 3. The reported figures are associated with two parts of each analysis. First, the population of an SQLite database with a horizontal and vertical slice of Crossref (and other) data. The size of this database (together with any indexes created in it) is reported as S_pop and the corresponding time required to populate the database as T_pop. Second, the execution of ROLAP SQL queries to obtain the required results. In many cases the queries generate secondary tables, which are stored in a separate ROLAP (analysis) database. The size of the ROLAP database is reported as S_ROLAP and the time required to run the queries as T_Qtot, T_Qmean, and T_Qmax. The query run time includes time for creating indexes, intermediate tables, and final reports. For tasks where no population figures are listed, the queries are run directly on the Crossref containers. For tasks where no ROLAP size is listed, the queries generate the results directly from the populated database, without creating any intermediate tables. All reported times are elapsed (wall clock) times shown in hours:minutes:seconds format.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance summary.

https://doi.org/10.1371/journal.pone.0294946.t003

The numbers shown on the table involve executions on idle or lightly loaded hosts with processes taking up a single core. The memory use (maximum resident set size—RSS) of population and query tasks averaged 1.2 GiB (minimum 4 MiB, maximum 49 GiB, median 35 MiB, σ = 5 GiB).

In addition to the SQL query times reported in the table, Python programs were used for analyzing the author graph (0:08:38 elapsed time, 55 MiB RSS) and the open source journal impact (0:01:19 elapsed time, 3.5 GiB RSS). Furthermore, a multithreading C++ program was used to calculate the CD₅ (9:24:20 elapsed time on 8 cores, 66:48:37 CPU time, 49 GiB RSS).

The task names in Table 3 mirror the contents of the Alexandria3k source code distribution examples directory. They are associated with what is reported in this work as follows. The author-productivity and yearly-numpages tasks were used to derive the corresponding yearly evolution lines shown in Fig 2. The crossref-standalone task derived the evolution in the number of journals (Fig 2), the yearly availability of abstracts (Fig 1), and the types of works available in Crossref. The open-access task derived the evolution in the number of papers published in open access journals (Fig 2), the reported ratio, and the statistical analysis regarding citations to publications in open access journals. The graph task was used to derive the remaining yearly evolution and yearly availability metrics, the Crossref record metrics (Table 1), and the evolution of publications in scientific fields (Fig 4). The orcid task derived the ORCID metrics shown in Table 2. The research-synthesis task derived the evolution in systematic literature reviews (Fig 3). Finally, cdindex derived the CD₅ index, covid the COVID-19 figures, impact-factor-2021 the 2021 journal impact factor, journal-h5 the journal h5-index, person-h5 the person h5-index, and soft-eng-h5 the h5-index of software engineering venues.

Statistical analysis

For reporting the correlation between the metrics obtained by Alexandria3k and existing ones and for comparing the graph clustering coefficients between two populations we used the functions spearmanr and mannwhitneyu from the Python package scipy.stats. All calculations were performed with “two-sided” as the alternative hypothesis (the default). No other options were provided to the function calls. We calculated CAGR with LinearRegression function of the sklearn.linear_model package. For the analysis and charting we used Python 3.9.10 with the packages cliffs-delta 1.0.0, matplotlib 3.3.4, numpy 1.20.1, pandas 1.2.4, scikit-learn 0.24.1, pySankey 0.0.1, and scipy 1.6.2.