Evaluation metrics for precise relation extraction.

The main metrics used to evaluate the effectiveness of this research are precision Eq. (1), recall Eq. (2), and the F1 score Eq. (3), which are defined as follows:

(1)

(2)

(3)

In this study, triples manually extracted and annotated serve as the ground truth. Each triple extracted by the LLMs is compared with every triple in the ground truth. If it is equivalent to a triple in the ground truth, it is categorized as a true positive; if it is not equivalent to any triple in the ground truth, then it is counted as a false positive. All extracted triples are merged according to their MESH value pairs. The number of false negatives is determined by subtracting the number of true positives from the total number of annotated triples.

In the preceding description, “equivalent triples” are defined as follows: if the chemical entity and the disease entity in two triples have the same MESH values, and if the relation types are identical for both triples, then the two triples are considered equivalent. For example, the triples (“vigabatrin,” “induced,” “visual field loss”) and (“vigabatrin,” “induced,” “visual field constriction”) are considered equivalent, as both involve the same chemical entity (MESH: D020888) and the same disease entity (MESH: D014786) with the relation type “induced.” Despite the different disease names used in the triples, they are recognized as equivalent.

Evaluation metrics for comprehensive relation extraction.

For this task, the BERTS score was selected for evaluation [37]. The BERTScore was designed to automatically compute the precision, recall, and F1 score by calculating the cosine similarity between two sentences. The evaluation based on similarity is more in line with the linguistic complexity of the results of comprehensive extraction.

Entropy calculation.

To measure the information complexity of each abstract text, Shannon entropy is employed to characterize the amount of information within the abstract. For a text containing multiple characters, the information entropy Eq. (4) is calculated based on the probability of occurrence of each character in the text. The formula is as follows:

(4)

wherein H(X) represents the entropy of the text, is the i-th distinct character in the text, and p() Eq. (5) is the probability of occurrence of the character. This probability is calculated as the number of times the character appears in the text divided by the total number of characters N in the text, expressed as:

(5)

where is the frequency of occurrence of the character in each abstract text, N is the total number of characters in that abstract text, and the unit of entropy is bits.

In this study, we first count the characters in each abstract, including letters, numbers, spaces, and punctuation marks, and calculate the frequency of occurrence for each character. Subsequently, we divide the frequency of each character by the total number of characters in the abstract to obtain the probability of occurrence for each character. Then, using the formula for Shannon entropy Eq. (4), we calculate the contribution of each character to the total entropy. Finally, we sum the entropy contributions of all characters to obtain the overall entropy of the abstract. The entropy value reflects the complexity of the character distribution within the abstract; a higher value indicates a more uniform distribution of characters and greater uncertainty in the information.

To compare the differences in entropy between two groups of abstracts (high F1-score abstracts and low F1-score abstracts), we employed the independent samples T-test. The independent samples T-test assumes that the data from the two groups come from different populations and compares their means. The null hypothesis () posits no significant difference in the mean entropy between abstracts with high F1 scores and those with low F1 scores. The alternative hypothesis () asserts a significant difference in the mean entropy between these groups. The formula for the independent samples T-test Eq. (6) is as follows:

(6)

where and represent the sample means of the two groups, and are the sample variances, and are the sample sizes of the two groups. The denominator is the square root of the pooled standard error of the two samples.

The computed T-value reflects the magnitude of the difference between the two means relative to the variability within the data. Subsequently, the T-value and degrees of freedom are used to find the corresponding p-value, which determines whether to accept or reject the null hypothesis. If the p-value is less than the set significance level (0.05), the difference between the two means is considered statistically significant.

Analysis of extraction effect.

As depicted in Fig 4a, executing solely the relation extraction step b results in suboptimal performance across all three models, with F1 scores of approximately 50%. By adjusting different modules in the workflow, we can effectively improve the results. The analysis of the results leads to the following conclusions:

1. (1) After adding the semantic disambiguation module (b+d), the F1 values of all three models increased significantly by about 20%. This indicates that the world knowledge of the LLM can effectively perform disambiguation tasks in this domain.
2. (2) After incorporating the LLM’s entity extraction step (LLM_NER+b+d), the F1 values of the three models showed no significant change. This suggests that splitting relation extraction into two steps—entity extraction and relation extraction—does not noticeably enhance the results. The LLM follows its inherent patterns during relation extraction, and specifying extraction steps does not alter its fundamental extraction approach.
3. (3) After adding the traditional model ConNER for entity extraction (ConNER+b+d), the F1 values of the three models improved by about 10% from the b+d baseline, reaching or nearing the highest F1 values. This is because the traditional model ConNER extracts biomedical entities more efficiently than the LLM, providing a high-quality list of entities. This highlights the importance of high-quality entity recognition about extraction. Additionally, removing the Note_prompt in relation extraction resulted in no significant changes in the F1 values of the models, indicating that requiring the model to establish one-to-one correspondence for entities does not have a noticeable impact on the results. It is again proved that relation extraction of LLMs follows an inherent pattern, and artificial steps are of little significance.
4. (4) After adding the follow-up Inquiry step, there were no significant changes in the F1 values of the models, suggesting that the models are quite confident in their extraction results.

In summary, in our tests, by establishing appropriate workflows, three LLMs all achieved commendable results in precise extraction tasks. Among these, the GPT-4 model reached the highest F1 score of 87%, demonstrating a strong comprehension of chemical-disease relationship types. This indicates substantial potential for the application of LLMs in this field.

Analysis of bias.

Fig 4b reveals that by adjusting different modules in the workflow, all three LLMs consistently demonstrate superior extraction performance for “induced” relationships over “treated” relationships. This observation suggests an inherent bias in relation extraction towards certain types of relationships within LLMs. Based on the analysis of the results, the following conclusions can be drawn:

1. (1) LLMs exhibit a markedly enhanced extraction capability for “induced” relationships when compared to “treated” relationships. Initially, across various workflows, the F1 score fluctuations for “induced” relationships in the three models are significantly less pronounced than those observed for “treated” relationships. The range between the maximum and minimum F1 scores for “induced” relationships is approximately half that for “treated” relationships, indicating that the models are more consistent in extracting “induced” relationships. Furthermore, the incorporation of follow-up inquiry leaves the F1 scores for “induced” relationships largely unaffected, whereas those for “treated” relationships exhibit a discernible decrease. This suggests that the models are less certain about “treated” relationships compared to “induced” relationships.
2. (2) Low-quality entities may compromise the model’s comprehension of relationship types, particularly for “treated” relationships. Furnishing a high-quality entity list can significantly mitigate bias. Following entity extraction using the traditional model ConNER (ConNER+b+d), the F1 scores for “treated” relationships demonstrate substantial improvements across all three models, thereby minimizing the bias discrepancy. This indicates that the difference in understanding between the two relationship types is not significant, especially for more capable models. Providing a high-quality entity list enables the models to focus on relationship types, leading to better performance.
3. (3) Among the three models, The Claude-opus model exhibits superior balance across relationship types, whereas GPT-4 displays greater variability in performance for “treated” relationships, possessing the most pronounced bias gap among the models.

In summary, we can conclude that LLMs exhibit certain biases. This bias can be mitigated to some extent by adjusting the design of the workflow.

Analysis of extraction effect.

Fig 5a illustrates that the comprehensive extraction performance across the three models is remarkably uniform, with GPT-4 exhibiting marginally superior performance relative to the others. Additionally, for each model, the performance differences under workflows composed of different modules are also minimal. Text structuring, follow-up inquiry, and the Note_prompt that requires strict adherence to the text content do not significantly influence the outcomes.

In this section, the highest F1 score achieved was 74% using the A+C workflow on the GPT-4 model. We consider this result to be somewhat unsatisfactory; therefore, we introduced a chain of thought (CoT) in this scenario (“Let’s think step by step”). After incorporating the CoT, the F1 score slightly decreased to 73%, indicating that, similar to precise extraction, comprehensive extraction by LLMs also adheres to their inherent patterns.

In summary, we can conclude that the LLMs exhibit robust understanding capabilities in comprehensive extraction, yielding consistent results across diverse module combinations.

Analysis of bias.

It is worth noting that according to the results in Fig 5b, the three models show significant differences in performance across different extraction content. Furthermore, under various extraction contents, the three models also exhibit notable differences in performance within workflows composed of different modules. This indicates that the results shown in Fig 5a are merely superficial. From the detailed analysis of the results in Fig 5b, the following conclusions can be drawn:

1. (1) LLMs demonstrate a pronounced proficiency in extracting side effects. All three models consistently outperform in the extraction of side effects compared to other contents, revealing an inherent bias toward side effects, attributed to their more distinct and readily extractable presentation in the text.
2. (2) In the extraction of accelerating and mitigating factors, both Claude-opus and GPT-3.5 exhibit substantially better performance in extracting accelerating factors over mitigating factors, suggesting a predisposition towards accelerating factors. This phenomenon likely stems from the training data for LLMs, which predominately comprise medical literature and texts where descriptions of disease onset, progression, and exacerbation are more prevalent than those of alleviation and mitigation. This data imbalance leads to the model being more sensitive and accurate in extracting conditions like “worsening ... certain diseases” or “increasing ... risk.” Additionally, the F1 scores do not vary much across different workflows, indicating that LLMs have an inherent extraction pattern and are less influenced by instructions.
3. (3) For the extraction of accelerating and mitigating factors, GPT-4 exhibits considerable fluctuation. In the a+c (No prompt) workflow, GPT-4’s F1 score for accelerating factors is about 20% higher than for mitigating factors. After adding the Note_prompt (a+c workflow), GPT-4’s F1 score for accelerating factors drops significantly, while its F1 score for mitigating factors rises substantially, exceeding the F1 score for accelerating factors by approximately 4%. Text structuring (a+b+c workflow) results in a significant decline in performance for accelerating factors, with an F1 score decrease of about 9%, but has little impact on mitigating factors.
4. (4) Under various workflows, the Claude-opus model exhibits minimal variation and a small range of differences in extraction performance across different scenarios. Therefore, Claude-opus exhibits commendable data equilibrium.
5. (5) LLMs display considerable confidence in their results, and the inclusion of follow-up inquiry generally has minimal impact under various conditions.

In summary, we can conclude that a notable bias towards the “accelerate factor” is observed under relative conditions. This bias can be partially alleviated by adding a “Note_prompt.” In addition, The extraction performance of GPT-4 under relative conditions is markedly unstable, whereas Claude-opus demonstrates comparatively stable performance.

The relationship between entropy and extraction effect.

In our study, we first calculated the entropy values of abstracts to analyze the relationship between information complexity (entropy) and extraction performance (measured by the F1 Score). Information entropy reflects the uniformity of character distribution in the text, with higher entropy indicating greater information complexity. Theoretically, information entropy could affect the readability and structure of abstracts, thereby influencing the extraction results.

For this purpose, we classified the abstracts based on their F1 Scores, calculating the entropy distributions for high F1 Score abstracts (F1 Score 60%0.8) and low F1 Score abstracts (F1 Score 60%0.8), and plotted a comparison of the entropy distributions (as shown in Fig 6). The graph indicates that the entropy values for both high and low F1 Score abstracts range between 4.19 and 4.91. Although the high F1 Score abstracts tend to cluster around 4.35 to 4.43, the distribution of low F1 Score abstracts is relatively more dispersed. Overall, there is considerable overlap in the entropy distributions of the two categories, and no significant correlation between entropy values and extraction performance is evident.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The relationship between model extraction effect and entropy.

https://doi.org/10.1371/journal.pone.0320123.g006

Further analysis through an independent T-test (p = 0. 23) revealed that the difference in entropy between high and low F1 Score abstracts is not statistically significant. This suggests that information entropy is not a major factor affecting relationship extraction performance. In summary, there appears to be no direct significant association between the entropy of abstracts and their relation extraction outcomes.

Statistical analysis for errors.

Subsequently, we examined the wrong data extracted by LLMs for their precision and comprehensiveness in identifying chemical-disease relationships. As shown in Fig 7, we identified several key factors that contribute to extraction errors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The results of Statistical analysis for errors.

(a) the reasons for errors observed in GPT-4.0 data; (b) the factors contributing to incomplete extractions by GPT-4.0; (c) the reasons for errors observed in Claude-opus data; (d) the factors contributing to incomplete extractions by Claude-opus; “E -” represents errors in entity recognition; “R -” represents errors in relation recognition.

https://doi.org/10.1371/journal.pone.0320123.g007

Through manual analysis of the various types of errors in Fig 7, we found that: Due to the complexity of processes and phenomena involved in biomedical texts, there are diverse forms of hidden meanings within the texts. Although the error forms can be superficially classified into multiple types, they essentially all stem from the LLMs’ inadequate understanding of the hidden meanings in biomedical texts. Here are three typical examples: (1) “Treatment for 2 weeks with Warfarin caused massive focal calcification of the artery media in 20-day-old rats and less extensive focal calcification in 42-day-old rats.” This sentence implies that the focal calcification induced by Warfarin is age-related, but LLMs failed to recognize age as a relevant factor, resulting in a context sensitivity error. (2) “Repeated transient anuria following losartan administration,” In this sentence, LLMs did not realize that adding the modifier "transient" to "anuria" changes the severity of the disease and still treated it as “anuria,” leading to an entity boundary ambiguity error. (3) “The administration of warfarin caused significant bleeding in the patient, and the bleeding was controlled after discontinuing the drug.” In this sentence, the model failed to recognize that “the bleeding was controlled” meant the cessation of bleeding and instead mistakenly interpreted it as the drug discontinuation causing the bleeding, resulting in an expression error.