Generation of relationship networks and biomarker predictions

Watson for Drug Discovery (WDD) biological relationship network extraction has been described previously [19, 20]. Briefly, interaction networks of biological entities are extracted as guided by a catalog of known biological interaction types and relying on natural language parsing of sentences to establish explicit, directional relationships between known entities, including genes and proteins (hereafter, “genes” for brevity), drugs, chemicals, and diseases. Importantly, WDD identifies not only direct relations between two entities, but also indirect relationships; indirect relationship extraction allows for a greater degree of variation in the semantic positioning of the agent, verb and theme of the relationship, including the appearance of other arbitrary words in key roles, for example “bcl-2 is able to modulate transmembrane trafficking of p53” would be considered an indirect relationship between bcl-2 and p53. WDD relies on a rule-based approach for learning syntactic relationships that connect known entities through verbs and other trigger-phrases from a curated dictionary, each of which also has a canonical, or normalized, form. For example, “inhibit” and “inhibits” are trigger words used to identify events associating a drug to a gene, and alongside similar phrases, map to a canonical form “Regulation Negative”. Learned rules are then applied to a MEDLINE corpus (https://www.ncbi.nlm.nih.gov/pubmed/) for extraction of relationships.

The WDD extraction engine is built using IBM InfoSphere BigInsights SystemT [21]–a powerful information extraction system for extracting structured information from unstructured and semi-structured text. SystemT provides basic text analytic capabilities such as sentence splitting, token detection, natural language parse of sentences, etc. Deep-parsing components of SystemT are derived from an English Slot Grammar (ESG) parser [22]. that provides core linguistic analysis. In the following sections, we describe the individual components that are central to WDD relationship extraction: (1) entity identification and normalization and (2) relation recognition and normalization.

Entity identification and normalization

In order to understand mentions of genes according to their context and map them to standardized identifiers, both rule-based [23] and machine-learning-based [24] approaches were integrated [25] for gene and protein extraction and normalization. To ignore mentions of generic protein families, such as “Histone”, and instead focus on mentions of specific genes and proteins such as “Histone H3” can be very challenging. For example, given a mention such as “GRK-1,3 and 5 have strong impact”, WDD needs to extract not only GRK-1, but also GRK-3 and GRK-5. Moreover, WDD needs to normalize genes to their canonical forms (their standard name); a key challenge here is that genes often share synonyms or abbreviations. Thus, WDD must understand mentions of genes according to their context and map them to standardized identifiers. To this end, WDD uses a hybrid model for gene extraction and normalization. The gene annotation process involves three steps: candidate generation, candidate selection, and entity normalization. For purposes of text inference, and specifically for biomarker discovery, distinguishing the gene from its protein product in text is not essential, therefore, WDD does not explicitly differentiate between genes and proteins.

Gene candidate generation

To ensure high recall for the gene extraction task, WDD relies upon a comprehensive dictionary of human genes compiled from NCBI [26], the UniProt KnowledgeBase [27], HUGO [28], and CTD [29]. This dictionary contains approximately 461,000 entries, where each entry is mapped to one of approximately 37,500 gene canonical forms. When generating candidates, WDD employs a combination of dictionary-matching, pattern-matching, and abbreviation-matching rules to maximize recall while maintaining precision; these are summarized below and described in greater detail in a recent technical publication [20].

Dictionary-matching rules: Because it is infeasible for a dictionary to capture all possible gene synonyms used in the literature, WDD supports fuzzy, or proximal matching of dictionary terms, i.e. the identification of phrases in text that are similar to, but not exactly matching, phrases in the annotation dictionary, which for example supports extraction of names with variations in the use of space and hyphen characters. WDD also accounts for different levels of ambiguity in the dictionary terms. During dictionary compilation, each dictionary term is semi-automatically categorized according to length and character complexity into one of three categories: unambiguous terms like “G protein-coupled receptor kinase 2”, ambiguous terms like “ATM” (which could conceivably be used as an abbreviation to refer to any number of non-gene concepts), and very risky terms like “C1” and “C2” (which could additionally stand for table and figure numbers, etc.). The dictionary at time of writing contains approximately 429,000, 17,000, and 500 terms in each category respectively. Each mention that matches some dictionary term is assigned a confidence score of “high”, “medium”, or “low”, depending on the category of the term. Mentions with a confidence score of medium or low are further verified using the context-based classifier described in the Candidate Selection section.

Pattern- and abbreviation-matching rules: Pattern matches concern examples like “ERK1/2”, whereupon detection of “ERK1” as a gene annotation, the immediate local context is inspected and the character pattern “/2” is identified as shorthand for “ERK2” and normalized accordingly. Abbreviation matching concerns dynamic detection of context-specific abbreviations in text, such that in a context like “angiotensin-converting enzyme (ACE)”, the abbreviation phrase “ACE” can be confidently extracted and normalized in the same manner as its definition “angiotensin-converting enzyme”, independent of the confidence level that “ACE” is assigned in any dictionary, or indeed whether it is a dictionary term at all. Dynamic abbreviation phrases identified in this way are then extracted across the entire text of the document so that subsequent mentions of e.g. “ACE” can be understood correctly. Since the meaning of each abbreviation can only be said to be constant within the particular document in question, phrases identified as abbreviations do not carry over into other documents.

Candidate selection

The context surrounding terms provides important information for resolving ambiguities resulting from abbreviation and term overlap. In order to exploit the cues in the surrounding context, we trained a naïve Bayes classifier that uses features derived from the neighboring context of a putative protein mention. WDD uses the sentence containing the discovered mention as the context span and considers all words present to the left and right of the mention, the number of gene mentions in the sentence, and whether or not the two immediate words to the left and right of the discovered mention are gene names. In addition to this model, the classification step involves application of a series of manually curated rules that filter medium and low confidence gene candidates based on key phrases and text patterns that denote a non-gene context.

To evaluate the classifier quantitatively, we compare its output to a sample of documents manually curated with the expected or ground truth annotations. Two ground truth corpora are utilized for this purpose, comprised of 100 MEDLINE abstracts prepared in collaboration with subject matter experts [19] (826 gene annotations) and all 347 MEDLINE abstracts from the BioNLP 2011 gene extraction challenge “GENIA” (https://sites.google.com/site/bionlpst/home/genia-event-extraction-genia) (5,301 gene annotations) respectively. As WDD development operates on agile model of continuous development and deployment [30], evaluation is performed within a testing framework which automatically executes upon every software build. This evaluation is performed after all document pre-processing, annotation, normalization and other post-processing steps have been performed ensuring that the evaluation metrics reflect the final, user-facing result. At the time of writing, gene classification in the collaboratively curated ground truth achieves 91.3% precision, 74.0% recall, and 81.7% F1 score and 77.0% precision, 67.8% recall, and 72.1% F1 score in the BioNLP 2013 challenge corpus. The difference in performance between these two datasets is primarily due to the BioNLP challenge dataset including ambiguous or otherwise non-specific annotations such as “these genes” or “a variety of cytokines”, which are not generally desired in WDD output, but nevertheless affect the annotator’s quantitative performance results in this corpus.

To assess annotation performance at full-corpus scale (28,278,250 MEDLINE abstracts at the time of writing), the classifier is also evaluated qualitatively. This is done using full-corpus scale annotation histograms that relay the most frequently annotated phrases and canonical names which result, in addition to numerous other aggregated statistics calculated across the total MEDLINE annotation count of 33,767,950 gene mentions (averaging 1.19 gene mentions per abstract, including document titles). User-provided annotation feedback is also used as a means of identifying and resolving annotation issues.

Entity normalization

To map a mention using a synonym or alias back to the correct canonical gene, WDD adopts the idea of unigram language models from computational linguistics [31] and builds “context models” for genes. For example, synonym “D1” maps back to both the Dopamine Receptor D1 (DRD1) and Leiomoden 1 (LMOD1). Whenever WDD identifies D1 as a gene mention, it analyzes the surrounding words for clues to identify the correct canonical gene, allowing disambiguation. For example, a mention of “dopamine” or the family name “GPCR” in the abstract would indicate that the abstract was discussing Dopamine Receptor D1 (DRD1); these patterns are identified through training a machine learning model which utilizes data about the other occurrences of each candidate gene across the literature.

In addition to the use of unstructured context information for gene disambiguation, WDD also leverages structured data associated with each of the candidate genes, and the document in which the annotation occurs. The structured data used for candidate genes is sourced from the Gene Ontology (GO) [32, 33] and used to increase the context model score for each candidate gene if any of the GO labels also occur in the document text. The structured data used for documents depends on the corpus in question, for example, MEDLINE abstracts include curated Medical Subject Headers (MeSH); this document metadata is used in the machine learning model in an equivalent manner to the context words extracted from the document text.

A prior technical publication describes the gene disambiguation algorithm in greater detail through a worked example of disambiguating a mention of “Drp1”, which could mean one of many genes depending on the context; the other processes mentioned here are also described in greater detail [20].

Event extraction and normalization

To extract relationships, WDD uses a catalog of gene-gene relation types and other biologically relevant verbs, together with natural language parsing of sentences. The catalog is maintained as a list structure similar to that used in the BioNLP 2013 pathway extraction task. The goal of extraction is to obtain triples, each comprising:

1. A relation: an element appearing in the catalog of relationships
2. An agent: a gene that is causally active in an event.
3. A theme: a gene that undergoes the effects of an interaction.

Entities (agents and themes) and the relationship between them are collectively referred to as events.

Depending on the relation, extracted triples may or may not possess directionality. Binding relationships, for instance, do not represent directional relationships, whereas phosphorylation is an example of a relation that has a clear causal agent and affected theme.

WDD extracts two kinds of triples that capture relationships between genes–direct and indirect. The first is a simple triple where there is a direct relationship between known entities. The following two sentences express direct relationships between known entities Plk1 and RSK1, and between p53 and Notch1: “In addition, studies on HeLa cells using Plk1 siRNA interference and overexpression showed that phosphorylation of RSK1 increased upon interference and decreased after overexpression, suggesting that Plk1 inhibits RSK1.” “These data indicate that p53 negatively regulates Notch1 activation during T cell development.” The second is a relationship where the genes are involved in more complex and indirect relationships. In the following examples we see that WDD extracts relationships between bcl-2 and p53 that are indirect, since different processes are involved in their interaction: “Silencing of Bcl-2 induced massive p53-dependent apoptosis.” “Our data suggests that bcl-2 is able to modulate transmembrane trafficking of p53.”

Learning patterns for event extraction

WDD applies rules over natural language parse trees for relation detection. Each sentence is run through the IBM ESG Parser and the dependency parse structure is examined for syntactic connections between the labeled relationships, agents, and themes. WDD applies complex rules where indirect interactions are involved. For example, a sentence such as “ATM-mediated phosphorylation of p53 at serine 15.” contains an indirect interaction, “mediated,” between the agent and the main relation, phosphorylation, i.e., “ATM-mediated” is an adjective modifying “phosphorylation”:

amod(phosphorylation, ATM-mediated) prep_of(phosphorylation, p53)

A rule is applied to allow for any indirect relationship (from the relationship catalog) to act as a connector between an agent and the relationship.

WDD applies rules where agents and themes are not simply genes but rather processes or events involving a gene. For example, in the sentence “Our data suggests that bcl-2 is able to modulate transmembrane trafficking of p53.” the relation “modulate” is connected to an entity “p53” via a process that is captured in the phrase “transmembrane trafficking of p53”. Parse structures for every gene-gene relation in the labeled training data are examined in the above manner to collect a set of rules that describe how relationships and entities might relate to one another. Although the set of rules is not exhaustive and will not identify every possible relationship, the high quality of the rules yields precision and recall sufficient to create a network well suited to relationship prediction [20].

Inference from networks

Collaborative Filtering [34] is the process of using known connections in a network to predict possible new connections. While this approach has been applied extensively to predict human preferences, the application of this method for predicting new biological connections is less well studied. Our approach to doing prediction over the generated network of gene relationships uses matrix factorization (MF) [35]. This technique can use the discovered triples directly to simultaneously predict many kinds of relationships. The process of evaluating the consistency of a relationship between entities, or between entities and a property, starts with extracting the known relationships from publications, as described in earlier sections. These relationships are then represented as a binary matrix. The score in this matrix for the relationship to be evaluated is set to zero. This matrix is then factored into H*W, where H and W represent smaller, dense matrices of a fixed, lesser dimensionality. The score of the relationship is its value in the product matrix of H*W. This score is compared to a set of comparable existing relationships evaluated in a similar manner in order to obtain a ranking of the relationship.

Given a set of agents (M) and a set of Targets/Properties (N), and directional relationships extracted from publications of the form m → n, where m ∈ M & n ∈ N, calculate the relative consistency of any particular m,n pair as follows:

1. Build a binary matrix, X, of dimension M x N containing a 1 in the mth row, nth column iff m→n exists in the scientific literature.
2. Set X[a,b] = 0, for the relationship a→b, whose consistency is to be evaluated.
3. Create a factorization of X, such that:
   • L_ij(W_i*,H_*j): loss at (i, j)
   • We assume the input is non-negative.
   • Find best model:
   • In our implementation we use Spark ALS (Alternating Least Squares Matrix Factorization) for implementation of matrix factorization. More details about the ALS document in Spark:
     1. –. https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/recommendation/ALS.html
     2. –. http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html#collaborative-filtering
   • Here are some brief explanations about the parameters along with default values.
     1. –. numBlocks is the number of blocks used to parallelize computation (set to -1 to auto-configure). (-1)
     2. –. rank is the number of latent factors in the model. (20)
     3. –. Iterations (20)
     4. –. lambda specifies the regularization parameter in ALS. (0.1)
     5. –. implicitPrefs specifies whether to use the explicit feedback ALS variant or one adapted for implicit feedback data. (implicit)
     6. –. alpha is a parameter applicable to the implicit feedback variant of ALS that governs the baseline consistency in preference observations. (0.1)
4. The resulting H,W matrices can be multiplied to produce a new matrix X₂. The value of X₂[a,b] is the relative consistency score for the relationship a→b.
5. Repeat this process for a set of comparable relationships to use as a background population. This can be some representative sample of all existing relationships in the literature.

Biological networks and performance evaluation

The WDD BTK network consisted of 64 gene upstream of BTK and 103 genes downstream of BTK. This network was compared to a BTK network from Clarivate Analytics MetaBase (https://clarivate.com/products/metacore/). MetaBase provides manually curated high quality systems biology content with nearly 1.7 million molecular interactions from over 1,600 pathway maps, and more than 230,000 disease-gene associations. A comparative study made by Shmelkov et. al. for transcriptional regulatory pathways in humans for seven well-studied transcription factors indicated that MetaBase had significantly high overlap with 10 other commonly used pathway databases [36].

The BTK network from MetaBase consisted of genes that were up to four links from BTK, upstream or downstream, with each link representing a directional physical interaction such as phosphorylation, receptor binding, ubiquitination, etc. As described, WDD extracts information on direct as well as indirect relationships. Since direct relationships are more likely to be transient changes upon BTK inhibition which were not in the focus of this study, we removed directly interacting proteins after ranking.

Data and code availability

All data generated or analyzed during this study are included in the published article and its supplementary information files (S1 and S2 Tables). A free 30-day trial of WDD is available at https://content.mkt.watson-health.ibm.com/whls-2018-wdd-free-trial.html.