3.1.1 The traditional approach.

Determining the scope and boundaries of a technology using the patent corpus, or organizing patents into clusters, has been a longstanding challenge. A variety of methods have been attempted in order to address this issue. The three main tools that have been utilized are technological classifications, citations, and keywords. While each of these tools can provide useful insights, they are also prone to introducing a significant amount of noise and variability into the analysis. In this section, we provide qualitative intuitions on these limitations. Section 3 will further quantify them. Technological classes are based on technical principles which are only partially related to the concept of technology we are looking for (functional application). Citations between patents have clear limitations in this case as well. Patent-to-patent citations are generated in order to define the scope of the technological monopoly granted to the patentees and to assess the validity of a patent over prior art. Proximity in the sense of functional application is then just one of the many reasons to generate a citation. Besides, the network of citations is very sparse and a large number of patents are never cited [30]. Finally, keywords can help identify patents dealing with a technology. However, language is highly variational: there are many ways to mention the same idea and at the same time a given word can have many different meanings. Hence, one can expect neither comprehensiveness nor accuracy from keywords alone. In this context, following [15], manual patent curation might appear to be the most accurate way to delineate a technology in the patent corpus.

3.1.2 Automated patent landscaping.

That is where the automated patent landscaping introduced recently by [9] makes an important contribution. The authors develop a semi-supervised machine learning framework to emulate human-made technology classification. The algorithm only requires a small set of patents as input—the seed—which must be representative of the technology of interest. The algorithm then expands to “likely related” patents using both technological classes and citations (forward and backward). Specifically, it first expands to technological classes which are overrepresented in the seed and then expands twice on citations. Importantly, at this stage, we know that the resulting expansion set includes patents unrelated to the target technology or “false positives”. The false positives are then pruned out using a classification model, based namely on the patent abstract, applied to the expansion set.

More precisely, the classification model is trained to distinguish between patents that belongs to the seed and a set of patents randomly drawn from the universe of patents, outside the expansion set (so-called anti-seed) and therefore “likely unrelated” to the target technology. This approach ultimately returns a group of patents in the target technology at virtually no cost, except for the curation of the seed patents. Importantly, no human intervention is needed to elaborate the set of rules determining whether a patent belongs or not to the target technology: semantic patterns are learned from the data.

The approach described in [9] has already demonstrated a high level of potential, but it still exhibits certain limitations that are worth noting.

First, the pruning model is trained on “polar” cases while we would prefer to apply it to “intermediary” cases. The seed patents (positive examples) are selected to be at the “core” of the target technology. On the contrary, anti-seed patents (negative examples) are chosen from the complementary of the expansion set, hence potentially very far away from the target technology. For example, when trying to select patents related to the blockchain technology, the anti-seed might contain patents on drugs, car engines and semi-conductors. Hence, even if the algorithm performs well on the validation set, it is not necessarily indicative of its performance when applied to patents in the expansion set, which may contain a significant proportion of “intermediary” examples. These examples may not be directly related to the target technology, but are still relatively close to it in terms of their characteristics or features. Training the model using a large majority of polar cases may therefore affect the overall validity of the classification model and the performance of the algorithm.

Second, the algorithm does not adequately consider the effect of variations in the data, such as the impact of changes to the seed data on the algorithm’s output. The robustness of the algorithm, or its ability to produce consistent results despite variations in the input data, is an important factor to consider when evaluating the reliability of the results and the overall interpretation of the analysis. Robustness is a crucial aspect to consider when assessing the confidence we can place in the results and the conclusions that can be drawn from them.

3.1.3 A new extended approach.

Our extended approach aims to address these two limitations. Firstly, we augment the anti-seed with more challenging examples. These complex examples naturally emerge from the human labeling of the seed patents tailored for each technology. We begin by reviewing existing efforts in the literature to landscape our target technologies using conventional methods. Specifically, we draw from this literature and their stipulated selection criteria (often based on technological classes and/or keywords; see, for example, [31–33] for Blockchain–a comprehensive list of our sources is provided in S2–2 in S2 Appendix). This guides us in generating a set of indicative patents, keywords, and CPC classes to seed for each technology. S2–3 in S2 Appendix elaborates on these criteria for each technology. Leveraging these guidelines, we randomly sample a set of potential candidate patents and manually inspect them by perusing their titles and abstracts, labeling them as either relevant to the technology or not (refer to S2–1 Table in S2 Appendix). Crucially, we retain the patents we exclude as they offer invaluable “hard examples”. Even though these patents met one or more criteria set by prior landscaping attempts, human scrutiny based on abstracts deemed them irrelevant. These typically represent the “intermediate” examples we intend our classification model to learn from, ultimately differentiating them from patents genuinely related to our target technology. We term this collection of examples the augmented anti-seed. Ultimately, our model is trained using both the traditional anti-seed as proposed by [9] and our augmented anti-seed to formulate the negative samples. S2 Appendix delves deeper into our seed construction process.

Second, we address the data variation question by implementing a series of robustness tests based on random variations in the seed. Specifically, we investigate how variations in the seed affect the expansion and the pruning outcomes. Formally, to test the robustness of the expansion, we draw random subsets from the seed, run the expansion using each of these subsets and compare the generated expansion sets. Next, we assess the pruning robustness by iterating over various random train-test splits of the annotated data. Various models are trained on varying sets of training data for each technology. Pruning robustness is ultimately evaluated by looking at models’ agreement on a sample of out-of-training patents. Detailed results are reported in Section 3.

4.1.1 Construction of the seed.

Next, we delve into the algorithm deployment itself. As already discussed in Section 1, our work starts one step before the algorithm described by [9]. This first step consists in the definition of rules to identify a set of candidates. These candidates are picked out of patents which match at least one of the rules that we were able to find in the specialized literature. These rules include technological classes, keywords and patent similarity. Indeed, there are instances in the specialized literature where specific patents are identified as being particularly representative or significant for a particular technology. In our study, we used the patents most similar to these key patents as defined in the Google Patents database and expand our dataset to include these related patents. A random set of candidates are then labeled by humans based on the abstract and detailed annotation guidelines (see S2–1 Table in S2 Appendix). Annotation guidelines guarantee both transparency and replicability. In practice, we labeled candidates until at least 300 were accepted, forming the technology seed. Notably, the rule-based candidates consistently contained a significant proportion of false positives, which we excluded. This excluded set, termed the augmented anti-seed, contains in particular patents that would have been mistakenly included in the target technology set if only a simple rule-based approach had been applied.

4.1.2 Expansion.

Starting from the seed, the following step is the expansion. Regarding this step, we mainly follow to the [9]’s procedure. We first expand to technological classes that were over-represented in the seed and then expand twice using citations (backward and forward). Note however that we had to adapt at the margin to take into account our choice to work at family level rather than publication level. In particular, we expressed citations in terms of the patent family rather than the usual publication format. For each family, we considered all citations received (forward) and sent (backward) by any patent in that family.

4.1.3 Pruning.

Finally, our pruning stage also differs from [9] along 3 dimensions. First comes the composition of the training data. As already discussed, we add an augmented anti-seed to the seed and anti-seed described in their paper. Second, while our predecessors used not only text but also citations and technological classes as input to the classification model, we only restricted to text. In our view, both technological classes and citations imply potential pitfalls at this stage. Using technological classes in both the expansion and the classification model can generate pathological cases. Assuming that all technological classes in the seed are found important, then the anti-seed and the seed have no technological class in common which makes the classification task trivial. Regarding citations, by construction, patents in the second level of the citation expansion (L2) have no citations in common with the seed. Hence, considering citations in the classification task implies a systematic and uncontrolled bias against patents in the part of the expansion which we find undesirable. Third comes the model itself. We implement 3 different neural network architectures popular for text classification tasks: the multi-layer perceptron (MLP), the convolutional neural network (CNN) and a transformer, specifically a pre-trained Bert encoder. We provide an overview of these architectures in the following sub-section. The actual pruning is performed using the Transformer model which exhibits both the highest performance and consistency.

4.2.1 Performance.

We then train all these models. The task is a standard binary text classification. Specifically, we train and evaluate each model on ten distinct train-test sets for each technology. We implement this approach as a cross-validation method to have an estimate of the impact of random variations of the training set on both the performance of the model and its out of (training) sample predictions—later called consistency. Let us first focus on performance before moving to consistency later. We report the median precision, recall and F1-score for each technology and model architecture in Table 1. These metrics were all computed on the test set, that is, on examples not used to train the model. The precision is the share of texts that the model assigns to the seed and which are indeed part of it. The recall is the share of texts in the seed which were indeed predicted to be part of it. The F1-score is the arithmetic mean of the precision and recall. We observe that MLP and CNN architectures tend to exhibit similar F1-score. However, MLP models have higher precision and lower recall than CNN. This relates to the fundamental nature of MLP. As stated earlier, MLP can be seen as a sophisticated keyphrase matcher which usually has high precision but low recall. In any case, the transformer outperforms both of the models and achieves around 90% of median F1-score for all technologies except for self-driving vehicles (79%). This latter technology is indeed harder to classify even for humans. The very same technology can be used to automate driving or to assist human driving. In the former case, we would accept a patent while in the latter it would be rejected. In the rest of the paper, we will use results from this latter model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Models performance.

https://doi.org/10.1371/journal.pone.0295587.t001

4.2.2 Consistency.

As already discussed, although performance per se matters, it is also crucial to understand how variations in the seed data can affect the results of the algorithm. We identify two channels. First, data variations can affect the expansion. The latter depends on the seed and has a critical role. It determines the set of documents which will be considered by the pruning model. Second, data variations can affect the pruning itself. The pruning model depends on the seed, the anti-seed and the augmented anti-seed and ultimately determines which documents in the expansion are to enter the technology or not. Robustness to random variations in the data is then crucial to ensure that algorithm results can be exploited rigorously. To investigate the consistency of the expansion, we generate random subsets of the seed. Specifically, we consider 3 different sizes: 90%, 70% and 50% of the initial seed and draw 10 subsets for each size. We then proceed to the full expansion starting from these distinct seeds and compute the pairwise family overlap of the generated expansion sets for each technology and seed size. Detailed results are reported in Table 2. We find that the average pairwise family overlap exceeds 89% in all cases. This remarkably high number indicates a high level of consistency for the expansion step and reassure regarding the relevant of the delimited technology.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Median pairwise expansions overlap.

https://doi.org/10.1371/journal.pone.0295587.t002

Next, we looked at how the pruning stage is affected by variations in the training data. As discussed above, we trained the same architectures on 10 different train-test splits (of respective size 80%-20%) for each technology as a way to emulate natural variations in the data. We then apply these models to a set of 10,000 out-of-training-sample documents randomly drawn from the expansion. For each technology, we then look at the standard deviation of the ten scores (each score ranging between 0 and 1) for each document and report its median in Table 3. We find that the standard deviation of the predicted scores is usually very low, most of the time below 0.05 which supports the consistency of the pruning step.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Models robustness (Median dispersion in predicted scores).

https://doi.org/10.1371/journal.pone.0295587.t003

To summarize, our evaluation of the performance and consistency of the extended patent landscaping is very encouraging. In the next section, we take a first look at the set of patents that constitute each of the six technologies and consider the external validity of our approach.

4.3.1 Top 10 assignees by technology.

Top panel of Table 4 reports the top 10 assignees for each technology by the number of patents they were granted worldwide.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Top 10 assignees and top 10 inventors.

https://doi.org/10.1371/journal.pone.0295587.t004

A first observation is that most of the obvious players in each technology are present. For the sake of brevity, we focus on some remarkable high-ranked agents for each technology and explain why they were indeed expected. Starting with additive manufacturing, Xerox and Hewlett-Packard are two large companies that traditionally developed printers and which naturally moved to 3D printing technologies. In the field of blockchain, Alibaba, Intel, nChain and IBM are also in the top list of assignees in the expert-based landscaping of blockchain innovation proposed by [32]. The most prolific assignees in the field of Computer vision include firms that build and sell electronic devices, including cameras (Canon, Sony etc…). Interestingly, the top assignees in the field of genome editing are universities such as University of California Berkeley, Harvard University and University of Pennsylvania. As explained in Section 1.3, this technology as the characteristics of being very tightly connected to the academic world and breakthrough advances have been made in the laboratories of famous universities. Nevertheless, the list also reports large companies that develop chemistry and pharmaceutical products like Regeneron and Dupont. Overall, these findings are consistent with results from an overview of patenting in the genome editing technology field proposed by [36]. The field of hydrogen storage technologies is primarily dominated by car manufacturers, reflecting the primary use of this technology to power hydrogen-propelled vehicles. Similarly, the realm of self-driving cars features many traditional car manufacturers, notably including Toyota and Ford, both of which actively publicize their advancements in autonomous vehicle development. The roster of leading assignees also features automotive equipment suppliers like Bosch and Denso Corp. Notably, Toyota, Ford, and Bosch are highlighted as top assignees in the field according to Chapter 3 of [37].

On top of very large firms that spread over a large number of different technologies such as IBM and Samsung, we also note the presence of a number of firms that are much more specialized in a specific field. This is notably the case of Air Liquide for hydrogen storage, nChain for blockchain, ASML for additive manufacturing, Regeneron pharma for genome editing and Denso Corp for self-driving cars.

4.3.2 Top 10 inventors.

Moving from firms to people, the bottom panel of Table 4 reports the top 10 inventors for each technology by the number of patents they were granted worldwide.

As previously, for the sake of brevity we focus on the most emblematic and high-ranked inventors. We can note the presence of Marta Karczewicz in both Blockchain and Computer Vision. M. Karczewicz is a prolific inventor working at Qualcomm Technologies, Inc.. She is famous for having developed many technologies related to data compression which facilitates the transfer of important mass of information. The methods she developed are very central for many computer-related technologies such as computer vision and blockchain. As a recognition for her contributions, the EPO named her one of the three finalists for the award of European inventor of the year 2019 [38]. Considering additive manufacturing, the most prolific inventor in the field is Kia Silverbrook. He is also a famous inventor who holds more than 9,000 patents worldwide [39]. K. Silverbrook founded Silverbrook Research, a company that developed digital printing and 3D printing technologies, among other inventions. In the field of genome editing, our top inventor is Andrew Murphy. He is the vice president in charge of research of Regeneron, a biotechnology company that develops different drugs and recently made important progress in new therapies using CRISPR [40]. We also note the presence of Feng Zhang, a Professor at MIT and researcher at the Broad Institute. He is well known for his role in the development of optogenetics and CRISPR. He is also famous for his ongoing patent dispute with Chemistry Nobel Prize recipients J. Doudna and E. Charpentier over CRISPR-cas9 human application priority. Next, regarding hydrogen storage, Stanford R. Ovshinsky was a prolific inventor and engineer who contributed enormously to various fields, including energy science, and own hundreds of patents. In particular, he developed solid hydrogen storage technologies and founded the company Ovshinsky Innovation LLC at the end of his life to continue to explore alternative sources of power. Finally, in self-driving vehicle technology, Atsushi Tabata is an engineer at Toyota who published several articles related to the automation of driving controls.

4.3.3 Top academic publications.

As a last exercise, we use the PatCit database [35, 41] to look at the most cited academic papers by technology. PatCit is a tool that lists all citations from patents to research articles (also known as Non Patent Literature citations) that were used as a source. We report these articles along with the corresponding journal title in Table 5. To save space, we only report the top 3 for each technology but a longer list of doi is available in Table S3–2 in S3 Appendix. As expected, the most cited articles, i.e. those that were the most pivotal in producing the ideas used in the development of the patents of each technology, are published in journal that are related to the technology. These journal can have a direct and clear link, for example, the International Journal of Hydrogen Energy is mentioned for hydrogen storage and the Proceedings Eighth IEEE International Conference on Computer Vision for computer vision.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. 3 most cited articles by technology.

https://doi.org/10.1371/journal.pone.0295587.t005

However, the links may also seem less obvious, reflecting the complexity of externalities from academic research to the development of innovations. For example, the second most cited article for self-driving vehicle is a 1982 research that discusses CO2 concentration in the atmosphere. Since one of the goals of autonomous cars is to reduce the carbon footprint of transportation, this topic is often mentioned and discussed in the relevant patents.

4.4.1 Comparison with other approaches.

Using our candidate annotation exercise, we can compare those results with the performance that would have been obtained based on the rules used by existing attempts to landscape our six technologies of interest. Specifically, it enables us to obtain performance metrics for rule-based approaches using technological class, keywords and patent similarity. Although our approach does not enable us to compute all the performance metrics reported before, we can compute the precision of simpler approaches (for example using only a set of relevant keywords). We find that rule based candidate selection delivers both low and variable precision performances across technologies. Specifically, precision from CPC-class rule-based patent selection ranges from 0.01 (blockchain) to 0.34 (additive manufacturing). Precision from keyword rule-based selection goes from 0.09 (blockchain) to 0.89 (genome editing) for an average of 0.32. Precision from patent similarity ranges from 0.02 (additive manufacturing) to 0.57 (genome editing). All these metrics are reported in S3–1 in S3 Appendix. It clearly appears that our approach to delineate technologies from the corpus of patents not only achieves good performance but also outperforms traditional rule-based methods. Hence, the set of patents selected using this new approach is both more precise and more complete than those of most existing attempts.

Our approach to constructing the seed and anti-seed for patent classification, though more labor-intensive, offers clear advantages. By meticulously designing the seed and anti-seed, we ensure that our classifier focuses on discriminating between closely related cases, enhancing precision in complex scenarios. This contrasts with methods such as those by [9], which are trained on more distinct, polar cases. Our advanced transformer-based architecture further supports this nuanced approach, enabling more effective and efficient handling of large patent datasets. The comprehensive effort in seed construction, while demanding, facilitates a robust and transparent classification framework, effectively addressing potential biases and ensuring balanced classifications. As such, while our methodology may not lend itself to direct performance comparisons with existing rule-based approaches due to its distinct foundational principles, it represents a significant advancement in the field of automated patent classification, providing a more accurate and comprehensive understanding of technology landscapes.

4.4.2 Use and misuse of this classification.

Our methodology offers flexibility in extending to various technologies, providing researchers a robust framework to delineate technologies through meticulous seed construction, adhering to our “automated with human in the loop” approach. Key to this extension is the precise definition of the set of tasks and stringent rules for patent inclusion in the seed, as these are crucial for maintaining the output’s quality. Specifically, it necessitates selecting technologies characterized by well-defined functional applications (e.g. solar photovoltaic materials, hardware for quantum computing, biodegradable plastic, battery technology for electric vehicle etc…). While our approach is versatile, we recommend caution when applying it to broad and multifaceted concepts such as AI or green innovation. These terms often encompass a diverse range of notions, which might challenge the specificity required for our methodology. However, specific technologies within these broader domains can certainly be effectively delineated using our approach. This recommendation is aimed at preserving the accuracy and relevance of the classification, ensuring that the methodology is applied where it can be most effective and meaningful.

4.4.3 Ethical concerns.

In the context of automated patent classification, it is essential to address associated ethical considerations. Our methodology integrates both automated processes and human oversight. This integration aims to rectify potential biases that might arise from exclusive reliance on automation, ensuring a more robust and transparent classification framework. By utilizing a diverse and representative training dataset, we actively mitigate the risk of unbalanced classifications, which could inadvertently misguide research and development trajectories. Moreover, our approach remains consistent with the public domain status of patents, circumventing potential privacy complications. Thus, our methodology offers not just efficiency in patent landscaping but also a conscious alignment with ethical best practices in academic research.