Corpus characteristics and annotations

In order to assemble a corpus suitable for pattern development, all fields of Medline articles were searched with a text query ‘Alzheimer disease AND hypothesis’ using the PubMed search engine that retrieved 3007 abstracts as on 26-7-2012. From this initial collection of retrieved abstracts, a preliminary corpus containing 150 randomly chosen abstracts was generated (referred to as HYPO-DEV-1). All abstracts in HYPO-DEV-1 were annotated with one class named Speculative pattern (see next section for annotation examples). This corpus was manually annotated for the presence of speculative text patterns that characterize scientific hypotheses. In parallel, annotation guidelines for the annotation of speculative statements were developed. First, an annotator (referred to as the principal annotator) participated in the annotation and guideline development process. The corpus was iteratively annotated by him along with the standardization of the annotation rules. Textual patterns annotated in the HYPO-DEV-1 were collected to form an initial version of a dictionary containing speculative patterns.

In the next step, a secondary corpus containing 200 randomly chosen, but non-overlapping, abstracts were generated (referred to as HYPO-DEV- 2). HYPO-DEV- 2 was annotated by two different annotators (referred to as the junior annotators). These annotators individually annotated HYPO-DEV-2 corpus based on the previously developed annotation guidelines and their annotations were used to calculate the inter-annotator agreement (IAA). The IAA determines the quality and acceptability of the notion of ‘hypothesis or speculation’ among annotators and provides a rationale for measuring the quality of prior developed annotation guidelines. Novel patterns annotated in the HYPO-DEV-2 were used to enrich the initial version of the dictionary (i.e. based on patterns collected from the HYPO-DEV- 1 corpus).

The IAA kappa between the two junior annotators was calculated as high as 0.81, which indicates an acceptable agreement, given the complicated nature of the annotated patterns. Similar to the developmental corpus, a test corpus (referred to as HYPO-TEST) was generated by searching PubMed using the keyword ‘Alzheimer’ and then randomly selecting 200 abstracts from the 58922 retrieved abstracts (as of 26-7-2012). Two new ‘independent annotators’ used the previously developed annotation guidelines for annotating the HYPO-TEST corpus. In case of any possible conflicts occurring between the annotators concerning ‘non-overlapping patterns’, the principle annotator was involved in resolving them. The following guidelines were applied for resolving the conflicting annotations between the independent annotators.

1. No entirely new annotations were added by the principle annotator. This means that if they were not marked earlier by either of the annotators, the principle annotator would not add any new annotation.
2. None of the annotations made by the annotators were removed by the principle annotator if both of the annotators marked them earlier.
3. Annotations were added or removed if they were marked by either of the annotators provided that they both agree with the ‘principle annotator’ on the consensus decision complied thereafter.
4. In case of partially overlapping annotations, only the conflicting words were removed. For instance, the first independent annotator marks “might be involved” whereas the second annotator marks “might be”, then the decision will be made to resolve the annotation by removing the word “involved”.

The same workflow as mentioned above was used in annotating the “Remote corpus” mentioned later in the manuscript.

HYPO-TEST corpus serves as our gold standard for the performance evaluation of different hypothesis detection approaches (pattern matching and ML based) mentioned in the manuscript.

To address the concern of data bias that might result from our search and corpus assembly strategy, we also evaluated the performance of our model for speculation detection on Bioscope, an independent and expert-curated corpus. This corpus consists of medical and biological text annotated by experts for speculation and their linguistic scope. The corpus is a good resource for comparison and independent assessment of Natural Language Processing (NLP) systems [12]; hence, we used this corpus as a surrogate system to independently assess performance of our model (mentioned under result section).

An overview of the methodology described in this section is provided in (Figure 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. An overview of HypothesisFinder development approach.

The workflow for the development of HypothesisFinder shows how the model was trained, optimized and on what data sets its performance was evaluated.

https://doi.org/10.1371/journal.pcbi.1003117.g001

Annotation of speculative patterns

Words and phrases such as ‘may be involved’, ‘might regulate’, ‘seems to play’, and ‘it appears as’ typically render statements as speculative, and are thus called speculation keywords or speculative patterns. These words are strong indicators of speculation in text. Particularly in scientific abstracts, whose purpose is to summarize the scientific knowledge presented in full text articles, these speculative patterns act as a linguistic marker that guides readers in detecting proposed hypotheses within text. In this work, annotations were performed in a way to capture not only speculative patterns (typically a phrase or a span of text) but also hypothetical sentences (sentences that also indicate a scientific hypothesis).

Based on our experiences obtained from the annotation process, speculative patterns can be further classified as strong, moderate and weak patterns. Further to back up this claim, we parsed a new Alzheimer's related corpus of 58,922 abstracts into 707,946 sentences. Previously identified speculative patterns were searched for their presence in this ‘sentence corpus’ and a sentence count (i.e. total number of sentences with specific speculative pattern) was obtained for each pattern. Additionally, two independent curators manually curated the retrieved sentences (sentence count up to 50) for each specific pattern, hence calculating the ‘percent efficacy’ of each pattern to indicate a sentence as speculative. In most cases when the sentence count exceeded more then 50, we randomly chose 50 sentences for our analysis. An example of utilizing a strong, moderate, and weak pattern categorization is shown in (Figure 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Classification of speculative patterns.

Figure presents examples of strong, moderate and weak speculative patterns along with their estimated ‘percent efficacy’ or ability of pattern to cast a sentence as speculative.

https://doi.org/10.1371/journal.pcbi.1003117.g002

Based on the results, we concluded that phrases containing modal verbs such as ‘may’, ’might’, or ‘could’ linguistically represent class of strong speculative identifier. Other words such as ‘potential’, ‘possibility’, ‘should ‘, ‘would’, and similar ones are not necessarily used in speculative context and may produce false positives. For instance, consider the following sentences:

1. Herein, we discuss the evidence supporting the critical role of tau oligomers in AD, the potential and challenges for targeting them by immunotherapy as a novel approach for AD treatment..(PMID: 21605039)
2. For some forms like vascular dementias, the possibility of prevention is the most valuable approach that should be enforced more aggressively. (PMID: 17012946)

To overcome this problem and to increase the accuracy of our model, we combined weak speculative patterns either with additional speculative words or with additional auxiliary verbs or adjectives before including them in the final dictionary. Revision of annotated patterns after introducing these changes to the dictionary showed improved performance and hence increased the quality of annotations as exemplified by following sentences:

1. These data suggested that GHE could be a potential agent for preventing or retarding the development or progression of Alzheimer's disease. (PMID:19221423)
2. These results suggest that some SAD may involve alternative processing of multiple γ-secretase substrates, raising the possibility that the molecular pathogenesis of SAD might involve γ-secretase dysfunction. (PMID:21681798)

In both of these cases, combining ‘potential’ and ‘possibility’ with additional speculative patterns such as ‘could be a’ and ‘raising the … that’ respectively turns these weak indicators to linguistically stronger identifiers of speculation, resulting in less likelihood of producing false annotations. Furthermore, to decrease false annotations and to better define what a speculative sentence is, annotation guidelines were also enriched with examples of patterns and sentences (Negative controls) that can conflict with speculative patterns and sentences.

The followings are some examples of sentences expressing inferences, results and conclusions, arguments and open questions, which cannot be considered speculative.

‘Inferences’

Taken together these data demonstrate that this transgenic AD model can serve as a powerful tool for the identification of AD therapeutic interventions. (PMID:21673973)

‘Results and conclusions’

Galantamine reduces behavioral symptoms in patients with mild to moderate AD, leading to reduced caregiver burden. (PMID:21615354)

‘Argumentative sentences’

There is increasing evidence that the consumption of flavonoid-rich foods can beneficially influence normal cognitive function. (PMID:21982844)

‘Open questions’

Whether T2D can cause late onset Alzheimer's disease (LOAD) remains to be elucidated. (PMID:22433668)

Following the recommended annotation guidelines, abstracts in HYPO-DEV- 1, HYPO-DEV- 2, and HYPO-TEST datasets were annotated for speculative patterns and hypothetical sentences in text. Speculative patterns appearing in HYPO-DEV- 1 and HYPO-DEV- 2 were extracted to generate a dictionary. Synonymous patterns were manually detected and grouped together to represent constitutive patterns (see the next section for details). Synonymous patterns denote different permutations or combinations of a speculative pattern that can occur within text. For example, the representative speculation pattern ‘appear to’ could have the following synonymous patterns: appeared to be|appears to be|appear that|appears that|appeared that |appearing to|appears to|appears to play|appear related|appeared related |appears related.

Dictionary characteristics and automatic pattern recognition

The dictionary used for recognition of speculative patterns in text comprises 156 representative and 392 synonymous patterns (Dataset S1). Efforts have been dedicated to cover all possible variations of speculative patterns appearing in developmental corpora. For automated identification of patterns within the HYPO-TEST corpus, ProMiner [13], a dictionary-based named entity recognition (NER) system, was used. In addition to dictionary-based NER, ProMiner also applies pre-defined rules and its search algorithm is geared towards handling the recognition of ambiguous multi-word terms in text. Previous scenarios are present where ProMiner has been used to perform NER by pattern matching or regular expressions in text [14]–[16]. The ProMiner search was performed using case-insensitive, word order-sensitive and the longest string exact match search constraints.

To study real-use case scenarios, ProMiner along with the speculative pattern dictionary was applied to the complete MEDLINE abstracts for the identification of speculative patterns. The recognized patterns were indexed and visualized using SCAIView, a scalable indexing and retrieval platform that has exhibited successful information retrieval scenarios from MEDLINE [14], patents [15] and e-health records [16]. SCAIView supports document retrieval as well as entity extraction. Speculative statements are indexed and made searchable within SCAIView and can be searched in combination with other biomedical terminologies and ontologies. Thus, the hypothetical space related to a particular question of interest can easily be retrieved using SCAIView. An example of such search scenarios using SCAIView is shown in (Figure 3). Moreover, ‘HypothesisFinder’ has been integrated into SCAIView and is freely available for usage and testing at http://www.scaiview.com/scaiview-academia.html

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Example showing usage of HypothesisFinder integrated in SCAIView for extracting hypotheses related to Alzheimer's disease.

Figure shows how HypothesisFinder is used within SCAIView in conjugation with other pre-indexed terminologies and ontologies to retrieve Alzheimer-specific hypotheses. Presented example shows how a hypothesis positioning Tau and Amyloid-beta as potential biomarker candidates in relation to AD is identified by HypothesisFinder in scientific abstracts.

https://doi.org/10.1371/journal.pcbi.1003117.g003

Training a machine learning model for hypothetical sentence detection

The application of ML-based approaches for sentence classification has demonstrated considerable success in the past [17]. To test whether dictionary-based or ML-based approach performs best for the identification of speculative statements in scientific text, we did a comparative assessment of an established ML-based approach against the pattern-based approach described above.

First, the HYPO-DEV-1 and HYPO-DEV-2 datasets were segmented into sentences. As required for ML training, sentences that represented hypotheses, i.e. contained speculative pattern annotations, were labeled as “POSITIVE” whereas those that did not were labeled as “NEGATIVE”. The overall training set contains 483 sentences labeled as POSITIVE and 2049 sentences labeled as NEGATIVE.

Sentences present in HYPO-TEST corpus formed an independent test set over which the performance of the trained model was validated. Similar to the training data, POSITIVE and NEGATIVE instances were generated for the test set, resulting in 246 sentences labeled as POSITIVE and 1194 sentences labeled as NEGATIVE.

A ML-based sentence classifier developed by Gurulingappa et al [16] was applied to train a sentence classification model (i.e. over HYPO-DEV-1 and HYPO-DEV-2 sentences). The sentence classifier facilitates switching between different classification algorithms that include Naïve Bayes (NB), Nearest Neighour (NN), Decision Trees (DT), Maximum Entropy (MaxEnt), and Support Vector Machines (SVM). The performance of sentence classification system was tested under three conditions using baseline features, speculative features, and lexico-syntactic features. Baseline features use all words appearing in sentences as features for classification. Speculative features were formed by occurrences of hand-crafted patterns (mentioned in Section Annotation of speculative patterns) that are potential indicators of hypotheses and Lexico-syntactic features were formed by the following:

1. Lemmatized tokens appearing in sentences
2. Bigrams of lemmatized tokens appearing in sentences,
3. Trigrams of lemmatized tokens appearing in sentences,
4. 2,3, 4 character prefixes and suffixes of nouns appearing in sentences, and
5. Stanford parser generated token dependencies occurring in sentences.

During training and testing the model with lexico-syntactic features, the above-mentioned textual features were extracted from HYPO-DEV-1, HYPO-DEV-2, and HYPO-TEST sentences.

Dictionary-based performance evaluation

The performance of HypothesisFinder was first evaluated on the HYPO-TEST corpus addressing the following two aspects:

1. The ability to recognize speculative patterns in text using a predefined pattern dictionary. The evaluation was performed by systematically comparing human annotated speculative patterns in the HYPO-TEST set with patterns identified by ProMiner.
2. The ability to classify sentences as ‘Hypothetical’ or ‘Non-hypothetical’ based on the recognition of speculative patterns. If ProMiner recognizes one or more speculative patterns in a sentence, the sentence is labeled as ‘POSITIVE’ where as sentences that do not contain speculative patterns are labeled as ‘NEGATIVE’. This classification also provides a rationale for the comparison of the pattern-based approach against the ML-based approach.

Evaluation metrics used for speculative pattern recognition and hypothetical sentence classification were Precision, Recall and F-score. The following formulas were used for the computation of Precision, Recall and F-score values [18].where true positives are the number of entities/sentences that were annotated by ProMiner and further matches with the human annotated entities/sentences that serves here as our gold standard. False positives are the number of entities/sentences that were recognized by ProMiner, but were not matched to annotations in the gold standard. False negatives are the number of entities/sentences that were not found by ProMiner when compared with the gold standard annotations. The results of the evaluation are listed in (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Performance of HypothesisFinder on the HYPO–TEST corpora.

https://doi.org/10.1371/journal.pcbi.1003117.t001

For ‘Pattern recognition’, HypothesisFinder achieved a precision of 0.84 and a recall of 0.86 in comparison to the manual annotation in the HYPO-TEST corpus. The major reason accountable for a 14% loss in coverage seems to be deletion of very weak speculative patterns in their original form to avoid chances of false annotations, as shown with the examples in “Annotation of speculative patterns” section.

Performance evaluation of sentence classification by machine learning

The performance of the maximum-entropy based sentence classification model was evaluated on a corpus composed of HYPO-TEST sentences. Precision, Recall and F-score metrics used for sentence classification over the class ‘POSITIVE’. Initially, experiments were performed by switching between different classification algorithms provided within the sentence classification system. Among the different ML classifiers tested, the Maximum entropy classifier (MaxEnt) provided the best performance whose results are considered here.

Comparison of the performance of ProMiner and MaxEnt classifier in their ability to classify sentences in HYPO-TEST as hypothetical is shown in Table 1. The MaxEnt classifier was applied for sentence classification using baseline features, speculative features, lexicosyntactic features, and their combinations. The MaxEnt classifier achieved the F-score of 0.50 and 0.62 when using simple words or lexicosyntactic features, respectively. Adding the speculative features boosted the ML classification performance with the F-score of 0.88. This indicates the value of hand-derived speculative patterns in assisting the development of a robust machine-learning model. Classification using speculative features alone was not possible since these features do not appear in all sentences whereas the ProMiner-based classification resulted in the F-score of 0.92.

The sentence classifier used here was applied with default input parameters (as defined by Gurulingappa et al). Since the goal of this work was to evaluate the adaptability of pattern-based approach to hypothetical statement detection, no extensive optimization of features for the sentence classifier was performed. Nevertheless, the observable difference in performance of pattern-based approach as compared to ML-based approach drives the interest in applying ‘easy-to-mold’ patterns for identifying hypothetical statements particularly in scientific abstracts where scientific knowledge presented in articles is summarized.

Performance assessments over Bioscope and Remote corpus

Evaluation of our model on Bioscope corpus (Table 2) in comparison to HYPO-TEST corpus show a relatively lower recall (0.73) while a comparably high precision (0.91) is maintained. A possible reason for the recall decrease is the lack of “weak patterns” (i.e. general terms such as ‘should’, ‘would’, ‘potential’) in HypothesisFinder's dictionary; these weak patterns are marked as speculative within the Bioscope corpus. Such weak patterns were integrated into the dictionary only in combination with other terms (e.g. ‘would likely’, ‘should possibly’, ‘could be potential’) so that their viability for identifying speculative expressions compared to their basic form is improved. Final assessment of our model on Bioscope corpus indicated a good performance with the F-score of 0.81, further confirming the ability of our methodology to detect speculations in text without corpus bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance of HypothesisFinder over Bioscope and Remote corpus.

https://doi.org/10.1371/journal.pcbi.1003117.t002

Performance of HypothesisFinder was also checked on a so-called ‘Remote corpus’ comprising AlzSWAN, Parkinson and Epilepsy corpus. AlzSWAN corpus comprises 143 abstracts presented as primary reference of hypothetical statements quoted in the AlzSWAN knowledge base. Again for calculation of evaluation metrics, each of the sub-corpora was manually annotated for speculative statements based on the previously defined annotation guidelines (mentioned under section corpus characteristics and annotations). The results of this evaluation, listed in Table 2, show that HypothesisFinder was able to detect speculative statements with 90% accuracy and 97% sensitivity in AlzSWAN corpus.

In order to show domain-independent application of HypothesisFinder, we also tested our model for speculation detection on a sub-corpus of randomly derived abstracts from PubMed related to Epilepsy and Parkinson's disease. Again, the high accuracy and sensitivity performance of HypothesisFinder (Table 2) demonstrates that our approach can be extrapolated to any given topic of interest.

Application of HypothesisFinder: A case study

Since our initial motivation for developing ‘HypothesisFinder’ was to establish an automated approach for capturing the breadth of hypotheses existing in the scientific literature around AD, we tested and benchmarked our automated approach against the expert-curated AD-specific hypothetical knowledge (inferred from the literature-derived speculations) in the AlzSWAN database. For this purpose, hypothetical statements were extracted automatically using HypothesisFinder and the results were compared to the AlzSWAN hypothesis browser. We sought to explore the power of HypothesisFinder in a semantically enhanced environment such as SCAIView by including a human gene dictionary and the Alzheimer's disease ontology [19] in the search. Through the combination of these dictionaries, a link between hypothetical statements and molecular (e.g. proteins) and clinical (e.g. disease stage) features of Alzheimer's disease can be established.

Figure 4 (A) illustrates the comparison between Alzheimer's stage-specific hypotheses linked to their corresponding biological entity (i.e. genes or proteins) extracted by HypothesisFinder as integrated in SCAIView (See Dataset S2, S3, and S4) and hypotheses with extended annotations (i.e. linked to genes and proteins) present in the AlzSWAN database. It is evident from our analysis that HypothesisFinder collects a large number of speculative statements, significantly more than the number of hypotheses listed in AlzSWAN. Even after curation of the hypotheses identified by HypothesisFinder, the count of normalized genes and proteins linked to these hypotheses (Figure 4 (B)) is higher than those in the AlzSWAN database. Additionally, we were also able to link extracted hypotheses to clinically established AD stages. In order to analyze the coverage of relevant genes and proteins linked to the hypotheses, we also compared the lists of genes and proteins mentioned in AlzSWAN with the list of entities retrieved by SCAIView. The comparison showed that all genes or proteins represented in AlzSWAN are actually a subset of the list generated by the HypothesisFinder approach, which indicates that HypothesisFinder is extending the hypothetical knowledge space not only at the level of speculative statements but also at the level of molecular entities being linked to disease hypotheses. Nonetheless, it should be noted that our automated workflow is not a replacement for expert-curated knowledge inference such as the one used by AlzSWAN but rather it is complementary to extensive manual curation efforts by domain experts. Therefore, systematic assembly of domain specific speculative knowledge present in Medline as well as full text documents using automated approaches will improve the knowledge aggregation and discovery process, leading to an accelerated growth of knowledge bases such as AlzSWAN. However, for making inferences over the gathered hypotheses, human intervention and expert curation is mandatory, as it is done in AlzSWAN.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Comparison of information densities: HypothesisFinder vs.

AlzSWAN. A- The statistical comparison between the numbers of hypotheses related to AD captured by HypothesisFinder within SCAIView (stage-specific retrieval) and the hypotheses with extended annotation derived from citations mentioned in the AlzSWAN database. B- A comparison between biological entity retrieval using SCAIView and relevant entries in AlzSWAN.

https://doi.org/10.1371/journal.pcbi.1003117.g004

Chronological analysis of hypothesis-related knowledge on AD

New scientific findings and discoveries that are expressed in the form of speculative statements represent the category of so-called ‘transient articles,’ whose citations peak within a short period of time [20]. These articles usually contain emerging trends of a specific knowledge domain, which evolves over time. To show how the intellectual landscape of scientific hypotheses on AD has evolved, we mapped the chronological order of retrieved hypotheses linked to five genes that are frequently speculated in the literature to be involved in the disease. As is shown in (Figure 5), the AD knowledge domain is currently dominated by speculations around APP (Amyloid beta (A4) precursor protein) and MAPT (Microtubule- associated protein tau) across all disease stages. Moreover, it is evident that the number of proposed hypotheses across all stages has been increasing since 2005.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Chronological order of hypotheses proposed in Mild (A), Moderate (B) and Severe (C) AD.

Figure shows a schematic representation of how AD stage specific hypothesis related to top five genes that are high-frequently investigated in the literature has evolved in number over time Abbreviations mentioned stands for Amyloid beta (A4) precursor protein (APP), Apolipoprotein E (APOE), Microtubule- associated protein tau (MAPT), Choline O-acetyltransferase (CHAT), Brain-derived neurotrophic factor (BDNF), Beta-site APP-cleaving enzyme 1(BACE 1), Galanin prepropedtide (GAL).

https://doi.org/10.1371/journal.pcbi.1003117.g005

Construction of hypothetical stage-specific disease networks for AD

In order to show the application value of HypothesisFinder in disease modeling, we used the BioNetBuilder plugin [21] in Cytoscape to build hypothetical protein interaction networks from the set of genes and proteins linked to disease stage-specific hypotheses. (Figure 6) depicts the connected components of three networks representing the three stages in AD. Such stage-specific disease networks visualize the disease progression at the molecular level and can provide a framework for further integrative analyses; e.g. integration of stage-specific gene expression data into network models could help to analyze gene expression perturbations across the three stages of AD.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Stage specific AD networks.

Figure presents protein interaction networks for Mild(A), Moderate(B), Severe(C) stage of Alzheimer's disease. These stage specific networks have been generated by using BioNetBuilder plugin in Cytoscape, which was given genes and proteins, associated to stage-wise hypotheses as input.

https://doi.org/10.1371/journal.pcbi.1003117.g006