2.1 Knowledge Base Question Answering Systems (KBQA)

Past educational question answering systems primarily relied on Knowledge Base Question Answering Systems [20–23] (KBQA), which respond to user queries using per-constructed, structured knowledge bases (such as Freebase [24], Wikidata [25], etc.). These systems are typically built upon Information Retrieval [17] (IR) techniques or Knowledge Graphs [26] (KG). The system first parses the user’s natural language question, identifies the entities, relationships, and attributes within it, and then queries the constructed knowledge graph based on this information. In this manner, the system can extract relevant information from the predefined knowledge base and generate answers by incorporating the context of the query. In specific domains (such as medicine, law, corporate knowledge, etc.), KBQA systems are generally able to provide relatively accurate answers, particularly when dealing with structured and standardized questions, showcasing high accuracy in responses [27].

However, traditional KBQA systems exhibit significant limitations. First, the construction and maintenance of the knowledge base often rely on substantial manual labor and rule-making. Especially in dynamic fields, the update cycle of the knowledge base typically cannot keep pace with real-world changes [28]. For example, when new academic research or industry developments emerge, the relevant knowledge bases may fail to reflect these changes in time, causing the system to be unable to provide the most up-to-date information [21]. Second, methods based on structured knowledge bases still face considerable challenges in dealing with the diversity and complexity of natural language [22]. Natural language contains a large number of ambiguous, uncertain, or polysemous words, and in open-domain questions, traditional knowledge bases often struggle to handle this diversity effectively. When confronted with complex contexts or phenomena such as polysemy and metaphor, traditional KBQA systems tend to show insufficient accuracy and relevance in their answers. Moreover, traditional KBQA systems lack flexibility and adaptability, especially when addressing cross-disciplinary, cross-domain, or novel questions. [23] In such cases, the system’s performance is often unsatisfactory, unable to provide comprehensive and timely answers. In conclusion, these issues limit the effective application of traditional knowledge base question answering systems in broad educational contexts, particularly when handling complex, variable natural language queries.

2.2 Retrieval-Augmented Generation (RAG)

With the rise of large language models [4] (LLMs), particularly driven by models such as GPT-4, Llama-3, and Gemma-2, new opportunities have emerged for educational question answering systems. LLMs, through deep learning techniques, are capable of understanding and generating natural language, significantly enhancing the fluency and context relevance of question answering systems. The advantages of LLMs in text generation allow them to generate high-quality answers, enabling them to respond flexibly to various natural language queries. However, despite LLMs’ outstanding performance in language understanding and generation, they still face challenges when dealing with tasks that require complex reasoning, rigorous logic, or precise calculations. LLMs primarily rely on pattern matching and probabilistic generation, lacking true logical reasoning capabilities, making them ill-equipped to handle problems that involve multi-step reasoning, strict calculations, or problem-solving. In such cases, LLMs may generate answers that do not align with actual knowledge, or they may produce “hallucinations” i.e., incorrect or fabricated information.

To address the limitations of LLMs in reasoning and accuracy, the Retrieval-Augmented Generation [29] (RAG) method has been introduced, as shown in Fig 2. RAG combines the strengths of information retrieval and generative models. Before generating an answer, it first retrieves relevant information from external unstructured textual data related to the user’s query and uses this information as context input for the generative model, thereby improving the quality of the generated answer. Specifically, RAG combines a retrieval module with a generation module: it first performs information retrieval to identify the most relevant documents or knowledge fragments and then passes this retrieved information to the LLM to help it generate more precise and informative answers. This mechanism of RAG effectively reduces the risk of LLMs generating false information, especially in open-domain question answering, where it can dynamically leverage the latest information from external knowledge bases, enhancing the system’s adaptability to dynamic knowledge domains.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. How RAG work.

https://doi.org/10.1371/journal.pone.0337361.g002

Although RAG has made progress in alleviating the “hallucination” problem of LLMs, it still faces certain limitations when handling complex mathematical computations and logical reasoning tasks [30]. Since the generative model itself does not possess strong mathematical computation capabilities [31], it may still generate inaccurate or erroneous results when confronted with tasks that require multi-step calculations or precise answers [32]. For example, when processing mathematical problems, although RAG can provide related formulas and methods, it cannot perform actual computations, potentially leading to inaccuracies when dealing with complex reasoning or precise numerical calculations. While RAG can mitigate this issue by leveraging external information to some extent, systems that rely on external textual information still face challenges in terms of precision and reliability in certain tasks.

2.3 Large Language Models (LLMs) and Code Interpreters

To address the limitations of LLMs in reasoning and computation, researchers have gradually introduced the Code Interpreter component [16,33]. A Code Interpreter is a tool that can generate and execute code, allowing it to handle complex tasks such as mathematical operations [34], data manipulation [35], and logical reasoning [36]. Unlike traditional text-based generation methods, a Code Interpreter directly processes computation tasks using programming languages, thereby overcoming the shortcomings of LLMs in complex computational tasks, as shown in Fig 3. By generating code and executing it in real-time, the LLM can obtain precise calculation results and detailed reasoning processes. For example, when a student presents a complex mathematical problem, the system can generate code snippets (such as Python code) and execute them in a controlled environment to obtain accurate answers and provide them to the user. In this way, the Code Interpreter not only improves the system’s performance in handling complex tasks but also enhances the transparency and venerability of the answers. Users can view the code executed by the system and verify each step of the calculation process, thereby increasing the system’s credibility and interoperability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Code interpreter with LLM.

https://doi.org/10.1371/journal.pone.0337361.g003

Additionally, the Code Interpreter plays an important role in logical reasoning tasks. Many problems involving logical reasoning (such as mathematical proofs, algorithm design, etc.) require multi-step reasoning to reach a conclusion. Traditional LLMs often rely on guessing and probabilistic generation when handling such tasks, which can lead to errors in the reasoning process. The Code Interpreter can directly execute logical code to verify each step of the reasoning process, thereby reducing the risk of generating inaccurate answers and improving the reliability of the reasoning process. This method not only enhances the system’s ability to handle complex tasks but also provides stronger computational and logical verification capabilities for educational question answering systems.

Traditional knowledge base question answering systems perform well in specific domains, but their limitations in dynamic knowledge updates, natural language understanding, and complex reasoning restrict their application in broad educational contexts. The introduction of large language models has significantly improved question answering systems’ abilities to generate and understand natural language, but challenges remain in reasoning and computational tasks, particularly with the issue of “hallucinations” The RAG method, by combining information retrieval with generative models, has alleviated some of these hallucination issues, but its performance in complex mathematical computation and logical reasoning tasks is still insufficient. The introduction of Code Interpreters further enhances the system’s computational and reasoning capabilities, enabling more accurate and reliable answers by generating and executing code. Based on this, the educational question answering system proposed in this research, which combines RAG and LLM Code Interpreters, aims to leverage the strengths of both technologies to address the shortcomings of existing systems and improve the overall performance of educational question answering systems in knowledge acquisition and reasoning tasks, providing a more comprehensive and intelligent solution for question answering systems in the educational domain.

2.4 Evaluation datasets

In order to comprehensively and objectively evaluate the performance of the proposed educational question-answering system based on RAG and LLM Code Interpreter in areas such as knowledge updating, reasoning, and computation, this study selects five representative public datasets that cover different difficulty levels and domains. These datasets include AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA. Each of these datasets has distinct characteristics in terms of question types, subject areas, and difficulty levels, which can help thoroughly validate the system’s adaptability and performance across diverse scenarios.

2.4.5 ScienceQA.

The ScienceQA [41] dataset consists of 21,208 multiple-choice questions from elementary and high school science courses, with the majority of the questions combining text and image content. The questions in this dataset are divided into three categories: questions with image context (10,332 questions, accounting for 48.7%), questions with text context (10,220 questions, accounting for 48.2%), and questions containing both image and text contexts (6,532 questions, accounting for 30.8%). Additionally, the ScienceQA dataset provides detailed explanations for answers and related instructional videos, which help the system understand the reasoning process behind the questions. Evaluation on the ScienceQA dataset can assess the model’s reasoning ability when handling Multi-modal information, especially the combination of images and text, and the role of RAG in integrating external knowledge, as shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Datasets To EVAL.

https://doi.org/10.1371/journal.pone.0337361.t001

These five datasets vary in question difficulty, subject areas, assessment focuses, and data formats, each offering unique strengths. They provide multiple perspectives to evaluate the comprehensive capabilities of the proposed educational question-answering system. By conducting unified experiments and comparative analyses across these datasets, we can quantitatively measure the system’s performance in mathematical reasoning, scientific knowledge understanding, multi-step logical deduction, and real-world educational applications. Additionally, this will help better explore the synergistic effects of RAG and Code Interpreter in handling complex educational question-answering scenarios.

3.1 Description of the LLM Models under evaluation

This study selected five representative and commonly used large language models for evaluation, chosen for their distinct characteristics in model size, inference capabilities, and their role in both commercial and open-source ecosystems. The five models are: GPT-4o, an optimized commercial version of GPT-4 [5] by OpenAI, known for its strong general-purpose natural language processing and reasoning capabilities; Claude-3.5-Sonnet [42], a conversational language model by Anthropic, which emphasizes context understanding and logical reasoning abilities; Gemini-pro-1.5, Google’s closed-source advanced version of the Gemini series, especially strengthened in mathematical reasoning and domain-specific knowledge. Gemma2-9B, an open-source model by Google with a smaller parameter size (9B) that prioritizes efficient inference and real-time deployment, capable of being deployed on a single A100 40G GPU; and Llama-3.5-8B, the latest iteration of Meta’s open-source Llama series, also with 8 billion parameters, striking a balance between computational efficiency and inference performance, suitable for resource-constrained deployment scenarios. These models were selected due to their unique advantages across model scale, inference properties, and both commercial and open-source ecosystems, providing a robust basis for comparison in this study.

3.2 Experimental design and comparison setup

To comprehensively assess the impact of different technological combinations on the performance of the educational question-answering system, this study designed four experimental setups. The first setup is the original LLM based Q&A system [43](Baseline), where the five models (GPT-4o, Claude-3.5, Gemini-pro-1.5, Gemma2-8B, and Llama-3.5-8B) were directly applied to the test sets from the five datasets without any external retrieval or code interpretation capabilities. This baseline setup serves to measure the basic performance and limitations of “pure LLMs” in handling educational question-answering tasks. The second setup integrates RAG (LLM + RAG), where a retrieval mechanism is added to the original LLM. In this configuration, the model first retrieves relevant document segments from an external knowledge base and then combines the retrieved information with the user query to generate more accurate, up-to-date, and controllable answers. This setup primarily compares the impact of retrieval on the model’s performance, particularly in knowledge updating and hallucination reduction. The third setup integrates the Code Interpreter with the original LLM (LLM + Code Interpreter), without the use of external retrieval. In this setup, for tasks requiring complex computations, multi-step reasoning, or programmatic inference, the model generates executable code (e.g., Python) during the answering process, which is then run in a sandboxed environment to produce accurate computational results or verify logical processes. This configuration evaluates the role of code execution in enhancing the model’s mathematical reasoning and logical inference accuracy. The fourth and final setup (LLM + RAG + Code Interpreter), which represents the complete system proposed in this study, combines both retrieval and code execution. In this setup, the model first retrieves relevant knowledge, then uses the Code Interpreter to perform complex calculations or logical inferences, generating answers based on the latest retrieved information. This combined approach is explored to determine whether it leads to superior and more reliable results in educational question-answering tasks.

3.3 Datasets and task settings

The datasets used in this study, AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA were introduced in related works section. Each dataset covers different subject domains, difficulty levels, and types of questions. The study evaluates the models from several perspectives, including basic comprehension and language generation, multi-step logical reasoning and calculation, cross-domain knowledge and contextual relations, as well as interoperability and venerability. For tasks involving concept explanation and simple question-answering, the models’ ability to accurately respond to questions was assessed. For tasks requiring multi-step mathematical operations and scientific reasoning, the ability of the models to correctly derive intermediate steps and results was evaluated. The study also examines how effectively the RAG mechanism supports cross-domain, up-to-date knowledge retrieval to enhance accuracy for such questions. Additionally, the ability of the Code Interpreter to generate and execute code to transparently demonstrate the reasoning process and reduce errors in computational tasks was evaluated. Since all five primary datasets (AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA) consist of multiple-choice questions, accuracy was chosen as the primary evaluation metric. Only the multiple-choice portions of these datasets were selected for evaluation to maintain consistency across metrics.

3.4 Implementation details of the RAG retrieval module

The RAG mechanism was implemented using the following approach, including knowledge base construction, retrieval and text selection, as well as fusion and input. The knowledge base was constructed by integrating publicly available text sources, such as Wikipedia excerpts, popular science websites, and digitized versions of various textbooks in both Chinese and English to ensure linguistic consistency in retrieval. The original text was segmented into sentences or paragraphs, and vector-based retrieval techniques [44] (e.g., Faiss or Milvus) were used to index the text. Embedding vectors for each text fragment were computed using per-trained language models like Sentence-BERT [45], facilitating similarity-based retrieval. For the retrieval process, the user query was converted into a vector representation, and cosine similarity was computed between the query vector and the vectors of text fragments in the knowledge base to retrieve the top-k most relevant document segments.

When a user query qqq arrives, it is converted into an embedding vector . Each document fragment in the knowledge base is also represented by an embedding vector . To find the most relevant fragments, we calculate the cosine similarity:

(1)

The top-kkk fragments with the highest similarity scores are retrieved. Retrieved fragments are then filtered to remove noise or irrelevant content and re-ranked based on similarity scores, document quality, and any predefined exclusion lists. Retrieved fragments were filtered to remove noise or irrelevant content and re-ranked based on similarity scores, document quality, and any predefined exclusion lists. The selected document fragments were then concatenated with the user query to form a prompt, which was input into the LLM, enhancing the model’s ability to generate high-quality and timely responses.

3.5 Implementation details of the Code Interpreter component

To enhance the system’s reasoning and computational abilities, the Code Interpreter component was responsible for generating and executing code to handle complex tasks. The execution environment was designed with a sandbox mechanism to isolate and limit the code’s access to system resources (e.g., runtime, memory, network) to prevent any security risks from malicious code or unintended actions. Python was chosen as the primary programming language for code generation due to its general-purpose nature and extensive libraries for scientific computing (e.g., NumPy, SymPy, pandas), which support multi-step mathematical operations and data manipulations. The code generation strategy used Chain-of-Thought [46] (CoT) prompts to guide the model to “think step-by-step [47]” and output executable code. In multi-turn dialogues, when programming was necessary for precise calculations or logical validation, the system would automatically generate the relevant code blocks and execute them within the sandbox environment. Once the execution results were returned, any errors (such as syntax errors or irrational results) were automatically debugged and corrected by the system, with the code being regenerated until a correct or reasonable outcome was achieved. The results were then synthesized into the final answer, and the entire execution process, including the code, logs, and output, was presented to the user to ensure traceability and variability of the reasoning process.

3.6 Prompt engineering

To maximize the synergistic effects of the LLM, RAG, and Code Interpreter components, careful prompt engineering techniques were employed [48–50]. The system prompt was designed to set the context clearly: “You are an educational question-answering system capable of answering questions across various subjects and performing multi-step reasoning and computation.” The prompt further specified: “For complex reasoning or calculations, please display the intermediate reasoning steps and validate using executable code,” and “If additional knowledge is required, first perform retrieval and then use the retrieved information to answer.” For multi-step reasoning, the Chain-of-Thought (CoT) prompt encouraged the model to decompose complex problems into smaller, manageable tasks, using instructions such as “Let’s break down the problem” or “Now think step-by-step” to help the model approach the problem systematically. Preference directives, such as “If calculations are needed, use Python code,” were used to activate the Code Interpreter. Additionally, a self-consistency mechanism was employed, where the model generated multiple reasoning paths and compared them to identify the most consistent and accurate solution. If discrepancies were found, further retrieval or code generation was triggered for verification.

3.7 Experimental procedure

The experiments for each dataset (AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA) followed a consistent procedure. First, the data were prepossessed by standardizing each question and its answer choices or descriptions, ensuring compatibility with the input formats for retrieval and LLM processing. Four experimental setups were tested in parallel: Setting A (original LLM), Setting B (LLM + RAG), Setting C (LLM + Code Interpreter), and Setting D (LLM + RAG + Code Interpreter). Each model was run through the same test set under these four configurations, with answers, reasoning processes, and time costs recorded. Automated batch testing and log recording were used, where each test case triggered the corresponding LLM interface to generate interactive dialogues or log files. In RAG scenarios, retrieved document fragments and their similarity scores were logged, while in Code Interpreter scenarios, generated code and execution results were recorded. The results were then collected and evaluated using the official metrics of the datasets, with accuracy as the primary evaluation metric for multiple-choice datasets (AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA). The final step involved error analysis and visualization, where errors were categorized and analyzed based on retrieval issues, LLM misinterpretations, or code execution errors, and the reasoning process for multi-step inferences was visualized to assess whether RAG and Code Interpreter combinations significantly improved interoperability and answer quality.

3.8 Evaluation metrics

Given that the primary datasets (AI2_ARC, OpenBookQA, E-EVAL, Textbook Question Answering (TQA), and ScienceQA consist of multiple-choice questions, accuracy was chosen as the primary evaluation metric. The same evaluation approach was applied to the multiple-choice portion of the E-EVAL dataset.

3.9 Implementation and environmental setup

The experiments were conducted on high-performance servers or cloud services equipped with GPU clusters. A secure sandbox environment was set up for executing code, with resources strictly limited. The retrieval database (Milvus/Faiss) was hosted on a separate server or in a Docker container on the same server. Both Gemma2-9B and Llama-3.5-8B models could be deployed on a single A100 40G GPU. The software environment included Python 3.9+ and various scientific libraries (e.g., NumPy, SymPy, pandas), vector retrieval tools (Sentence-BERT, FAISS, Milvus), and APIs or offline inference frameworks for the LLMs. As shown in Table 2, for models like GPT-4o and Claude-3.5, Gemini-pro-1.5 cloud-based commercial APIs were used, while Gemma2-8B, and Llama-3.5-8B were deployed locally on GPUs. The Code Interpreter was integrated with each LLM via conversational protocols, allowing real-time code generation and execution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Open source model and closed source model in this study.

https://doi.org/10.1371/journal.pone.0337361.t002

Through these methodologies, this study systematically evaluated the integration of RAG and Code Interpreter in educational question-answering systems, aiming to provide both theoretical and practical insights for future system optimizations.

3.10 Details of relationship management

In order to ensure the reliability and reprehensibility of code execution, a multi-level error handling mechanism is implemented in the system with the following flow.

First, a scientific sandbox environment architecture was constructed to encapsulate the Python 3.10 runtime environment using Docker containers for containerized isolation.

Second, per-checks are performed using AST parsing to carry out static per-analysis and signal capture at the dynamic detection layer.

Third, in terms of error classification and processing, five cases are handled separately according to syntax errors, infinite loops, memory leaks, numerical computation errors, and missing third-party library dependencies.

Finally, the use of error type-guided correction is carried out to implement a self-correction strategy.

4.1 AI2_ARC experimental results

AI2_ARC is a dataset for scientific reasoning, focusing on the model’s ability to integrate comprehensive knowledge, understand complex language, and perform reasoning tasks. The table below shows the performance of five models on this dataset under different settings.

Analysis: From Table 3 and Fig 4, it is evident that the performance of all models improves progressively with the introduction of prompt engineering (such as 1-shot, 3-shot, and CoT). This indicates that providing more context significantly enhances the model’s reasoning ability. The introduction of RAG and Code Interpreter further boosts the accuracy, especially in scientific reasoning tasks. RAG and Code Interpreter help the models handle cross-domain knowledge retrieval and complex reasoning tasks. The GPT-4o model performs the best across all settings, with its accuracy reaching 92.1% when combined with RAG and Code Interpreter.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. AI2_ARC experimental results.

https://doi.org/10.1371/journal.pone.0337361.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. AI2_ARC Accuracy Comparison Chart on different LLM.

https://doi.org/10.1371/journal.pone.0337361.g004

4.2 OpenBookQA experimental results

OpenBookQA is a dataset focusing on scientific reasoning and the use of external knowledge sources. This dataset requires models to demonstrate strong multi-step reasoning capabilities.

Analysis: From Table 4 and Fig 5, it can be observed that the introduction of RAG and Code Interpreter leads to an overall improvement in accuracy. Particularly, the GPT-4o and Gemini-pro-1.5 models achieve accuracy of 93.2% and 91.6%, respectively, when RAG and Code Interpreter are used. These results highlight that in complex multi-step reasoning tasks, RAG’s knowledge retrieval and Code Interpreter’s code execution significantly improve the accuracy of the answers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. OpenBookQA experimental results.

https://doi.org/10.1371/journal.pone.0337361.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. OpenBookQA Accuracy Comparison Chart on different LLM.

https://doi.org/10.1371/journal.pone.0337361.g005

4.3 E-EVAL experimental results

E-EVAL is a Chinese K-12 educational dataset with various types of questions, particularly focusing on spelling errors, sentence understanding, and cross-disciplinary knowledge integration. As shown in Fig 6 and Table 5, EVAL Accuracy Comparison Chart on different LLM.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. E-EVAL experimental results.

https://doi.org/10.1371/journal.pone.0337361.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. E-EVAL Accuracy Comparison Chart on different LLM.

https://doi.org/10.1371/journal.pone.0337361.g006

Analysis: On the E-EVAL dataset, all models exhibit more balanced performance compared to the other datasets. Gpt-4o demonstrates strong reasoning and cross-disciplinary capabilities once again. With RAG and Code Interpreter, the accuracy shows significant improvement, particularly in handling spelling errors and sentence comprehension. RAG’s knowledge base retrieval and Code Interpreter’s multi-step reasoning help the model better understand complex educational scenarios.

4.4 Textbook Question Answering (TQA) experimental results

TQA is a dataset based on high school science textbooks, containing text, images, and videos, designed to assess a model’s ability to process multi-modal information.

Analysis: From Table 6 and Fig 7, in the TQA dataset, all models show significant improvement in accuracy. Notably, GPT-4o and Gemini-pro-1.5 perform exceptionally well, achieving accuracy rates of 70.0% and 69.0%, respectively, when combined with RAG and Code Interpreter. These results suggest that, for multi-modal tasks involving text, images, and videos, RAG effectively helps the model retrieve relevant information from external knowledge bases, while Code Interpreter assists in multi-step reasoning and computation, thereby enhancing the model’s ability to process complex educational tasks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. TQA experimental results.

https://doi.org/10.1371/journal.pone.0337361.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. TQA Accuracy Comparison Chart on different LLM.

https://doi.org/10.1371/journal.pone.0337361.g007

4.5 ScienceQA experimental results

ScienceQA is a dataset for scientific question answering that tests multi-modal reasoning with both text and image inputs.

Analysis: From Table 7 and Fig 8, in the ScienceQA dataset, GPT-4o excels once again, with accuracy rising to 89.4% when RAG and Code Interpreter are combined. This result demonstrates the model’s superior performance in multi-modal reasoning tasks, particularly when RAG enhances knowledge retrieval and Code Interpreter supports computation in multi-step reasoning tasks involving both text and images.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. ScienceQA experimental results.

https://doi.org/10.1371/journal.pone.0337361.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. ScienceQA Accuracy Comparison Chart on different LLM.

https://doi.org/10.1371/journal.pone.0337361.g008

Through comprehensive analysis of five distinct datasets (AI2_ARC, OpenBookQA, E-EVAL, TQA, and ScienceQA), we find that the educational question-answering system, when integrated with RAG and Code Interpreter, exhibits significant advantages across various tasks. Overall, the system substantially improves accuracy in tasks involving scientific reasoning, open-book question answering, multi-step computation, and cross-disciplinary knowledge integration. Specifically, RAG effectively enhances the model’s ability to handle dynamic knowledge updates and cross-domain reasoning, while Code Interpreter further improves its performance in mathematical reasoning, complex computation, and logical verification. GPT-4o show the best performance in most experiments, particularly when RAG and Code Interpreter are used together, showcasing their exceptional reasoning abilities and precise knowledge retrieval capabilities.

4.6 Error analysis and bottlenecks

A deeper analysis of the experimental results reveals several major sources of model errors, which can be categorized as follows:

4.6.5 Statistical significance testing.

The hypothesis testing framework consists of the null hypothesis and the alternative hypothesis as follows.

Null hypothesis: The differences in performance indicators between configurations are due to random factors.

Alternative hypothesis: The combined approach performs significantly better than the baseline configuration (one-sided test).

In the multiple comparisons correction phase, FDR was controlled using the Benjamini-Hochberg method with a confidence level set at 0.05 and a statistical efficacy of 0.8, the specific experimental results are shown in Table 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Statistical significance test results.

https://doi.org/10.1371/journal.pone.0337361.t008

Through this multi-level statistical validation, we can demonstrate that the performance improvement of the integrated approach at the p < 0.01 level is statistically significant and practically meaningful.

5.1 Key findings

The experimental evaluation provided compelling evidence for the effectiveness of integrating RAG and LLM Code Interpreters in educational contexts. The system’s ability to retrieve relevant, up-to-date knowledge was significantly enhanced by the RAG framework, making it particularly valuable in educational environments where knowledge is constantly evolving. By supplementing the language model’s inherent knowledge with real-time retrieval from external sources, the system ensures more accurate and contextually appropriate answers, especially for scientific and technical questions that require up-to-date information. Additionally, the Code Interpreter played a crucial role in enabling the system to handle more complex reasoning tasks, such as multi-step calculations, logical reasoning, and algorithmic execution. By generating executable code to support reasoning, the system not only provided more precise answers but also allowed users to trace and validate the reasoning steps, enhancing transparency and trust. Furthermore, the system showed improved performance across a wide range of educational tasks, from scientific reasoning to cross-disciplinary knowledge integration. It excelled in tasks that demanded not just factual recall but also logic, computation, and the synthesis of information from multiple domains. Its robustness was evident in the system’s adaptability to diverse educational scenarios, as demonstrated by experiments on datasets like AI2_ARC, OpenBookQA, and E-EVAL. The system consistently produced accurate and reliable answers, even when tasks involved multiple logical steps or required external knowledge.

5.2 Challenges and limitations

Despite these advancements, several challenges remain. While RAG significantly boosted the system’s knowledge retrieval capabilities, occasional failures were noted, particularly when external sources lacked coverage of recent developments or niche topics. This limitation underscores the need for continuously updated knowledge sources and more robust mechanisms to handle knowledge gaps in real time. Additionally, despite the potential of the Code Interpreter to facilitate multi-step reasoning, errors in code execution were sometimes observed. These errors, stemming from misinterpretation of tasks or syntactically incorrect code, affected problem-solving accuracy. To address this, further work is needed to enhance the system’s code generation and debugging capabilities, ensuring more reliable execution of complex tasks. Another challenge lies in cross-disciplinary integration. While the system showed promise, it occasionally struggled with questions that required knowledge spanning multiple fields, especially when intricate relationships between concepts were involved. Future improvements should focus on enhancing the system’s ability to synthesize knowledge across disciplines, ensuring more seamless and reliable cross-disciplinary reasoning. Furthermore, as educational content expands and diversifies, ensuring the scalability of the retrieval system while maintaining speed and accuracy will be crucial. Generalizing the system to handle a broader range of topics and question types, beyond those tested in the current experiments, will require further refinement.

The deployment of LLM in educational environments, despite their great potential is accompanied by significant ethical risks and challenges. These risks may not only affect student learning outcomes, but may also have far-reaching implications for educational equity, cognitive development and even social values. First, there is the problem of bias due to data bias. Large-scale language models are trained based on a large amount of data, and if there are biases in these data, such as over-representation or under-representation of certain groups of people, one-sided emphasis on certain points of view, etc., then the model may learn these biases and then show bias towards specific groups and ideas in the generated content, which may adversely affect the students’ values and reinforce their stereotypes of certain groups of people Impression. Second, the problem of bias in the algorithm itself, the algorithmic structure and design of the language model may also introduce bias. For example, the attention mechanism of the model may unconsciously give higher weight to certain types of information and relatively ignore others, leading to unfairness in the generated results. Such algorithmic biases may not be easily detected, but can subconsciously affect information transfer and student cognitive formation in educational settings. Third, inertia of students’ thinking. If students rely excessively on large-scale language models in the learning process and seek answers from the models directly when they encounter problems instead of thinking, analyzing and exploring on their own, they will gradually develop thinking inertia, which is not conducive to the development of their critical thinking, creativity and problem-solving skills. In the long run, this will affect students’ overall quality and future development. Finally, educator dependency: educators may also rely excessively on large-scale language models for teaching, for example, by directly using model-generated lesson plans, teaching content, and so on, while neglecting their own professional knowledge and teaching experience. This may lead to a decline in the quality of teaching and a limitation in the professional development of educators, as well as an inability to tailor teaching to the actual situation of students.

5.3 Future directions

Several exciting avenues for future research and development emerge from our findings. To address retrieval limitations, future systems could integrate more sophisticated knowledge base management techniques, such as the continuous incorporation of new content from diverse and authoritative sources. Machine learning models for knowledge discovery and automated source validation could play pivotal roles in ensuring dynamic and comprehensive retrieval. The Code Interpreter could be further enhanced by developing advanced error-correction capabilities and tools for recognizing common computational mistakes, ensuring more reliable multi-step reasoning. Future work could also explore integrating debugging tools or training models to autonomously correct errors in logic or syntax. Cross-disciplinary knowledge synthesis remains a critical challenge. Future iterations of the system could include dedicated modules designed to recognize and process interdisciplinary links more effectively, which would be particularly beneficial for subjects like history and social sciences, where insights from multiple domains often need to be integrated. Professionalization is another promising direction. By tracking individual students’ progress and tailoring explanations, difficulty levels, and content to their learning trajectory, the system could offer more engaging and effective educational experiences. Integrating real-time feedback loops would further enhance adaptability. Finally, while the system has shown improvements in performance, optimizing its efficiency remains a priority. Developing more lightweight models or hybrid solutions could reduce computational overhead, enabling real-time applications in resource-constrained educational environments.

5.4 Conclusion

In conclusion, the integration of Retrieval-Augmented Generation (RAG) and LLM Code Interpreters represents a significant advancement over vanilla Large Language Models (LLMs) in developing educational question-answering systems. By combining dynamic, real-time knowledge retrieval with powerful multi-step reasoning and precise code execution capabilities, our approach effectively addresses critical limitations commonly found in traditional LLMs, such as knowledge hallucination, insufficient reasoning depth, and inadequate computational accuracy. Experimental evaluations demonstrate that this integrated framework achieves an average accuracy improvement of 10–15 percentage points compared to vanilla LLMs across diverse datasets, underscoring its potential as a robust educational tool. Although certain challenges remain—particularly related to knowledge retrieval efficiency, cross-disciplinary reasoning integration, and error handling in code execution, our proposed system provides a solid foundation for future research. As these technologies continue to mature, they hold great promise for enhancing learning outcomes, fostering personalized education, and promoting more effective, salable, and equitable education globally.