Use of AI in legal affairs

There are various NLP methods that have been utilized for most of the legal documents, such as contracts, case laws, statutes, etc. Legal personnel deal with huge amounts of texts that are legal documents, and therefore, AI becomes useful in improving the way legal tasks are handled [15]. Early research brought out the prospect of AI technology replacing basic legal operations like document review and clause identification [16]. From BERT, legal texts have been fine-tuned for legal text analyses that include named entity recognition, contract novation clause classification, and semantic similarity analysis [17]. Many of these improvements have cut the cycle time and effort needed to legally analyze documents while increasing the quality and standard of the work. Contract review and management has received a facelift from the advancement of AI with the automation of text mining for relevant clauses, commitments, and risks. Furthermore, Kira Systems and LawGeex can use machine learning to analyze critical clauses and check for anomalies and compliance with certain laws [18]. AI-assisted solutions were unveiled that can decrease the human factor influence and speed up contract analysis by 80% [19]. In addition to that, AI contracting platforms can track contract specifics, due dates, and renewals in real time, improving organizational productivity. AI-based applications such as ROSS Intelligence and CaseText operate by presenting lawyers with results that include case laws and legal analysis based on natural language requests [20]. Recent research has tried to apply graph-based artificial intelligence models to analyze citation graphs and assess the outcome of cases to support legal reasoning and judgment [21].

Decision support systems developed from past case data to try to make future predictions have also been used in the prediction of judicial behavior and evaluating associated risks of lawsuits [22]. For example, using the AI technique, it is possible to predict the outcome of the case in the European Court of Human Rights with better than 75% outcomes [23]. Such systems empower lawyers with information that helps improve their ability to reach better decisions and, therefore, come up with proper legal approaches to suit various cases. However, the integration of AI into legal documents has the following ethical and regulatory issues. The problems like responsibility for algorithm selection, data protection, and causality of AI’s decisions have become critical and topical [24]. Further, a problem arises about how AI makes decisions and the subsequent lack of accountability on the part of technology. There have been specific demands for ethical guidelines concerning the use of AI and the creation of a regulatory framework [2].

The study was composed of labeled U.S. case documents, and they established that CNNs, when applied, had improved outcomes of deep learning and high classification accuracy of more than 90%, opposed to conventional classification techniques. This, to some extent, illustrates the ability to deliver great potential for the deep learning ability in dealing with the complex syntax of legal language [25]. Another work was devoted to the text’s classification using CRFs and CNN in the legal sphere. The authors stated that, in general, there was 90% accuracy in CNNs, but there was 97% accuracy in the cases, as stated by the authors, which proved that extensive care should be taken to choose the relevant algorithms depending on the nature of the legal texts to be classified [26]. This result highlights the need for tailored approaches in using ML methods in the classification of legal documents. An investigation introduced an ensemble approach, which combines CNNs with the aim of exploiting the benefits of both. This work led to improved classification accuracies and sturdiness. This shows that it is possible that hybrid model methodologies yield increased performance compared to stand-alone methods [27].

Furthermore, an empirical study pointed out that variability in form and terminology of different legal documents posed a major problem in the classification of legal documents. The researchers presented a new model, which entails the use of domain knowledge to enhance the results of the ML models, thereby enhancing classification [28]. This approach, therefore, emphasizes the need to have a contextual approach when developing AIOps for integration with ML for legal solutions. There is also discussion about explainable AI (XAI) in the classification of legal documents. A paper presented techniques [29] for making the models used in the classification of legal texts more transparent. For practitioners, this will enable them to better understand the decisions of the ML model and gain trust in the intelligent systems. Moreover, new developments in NLP technology [30] have allowed for text analysis to progress to a point where semantic similarity within the text of a legal instrument can be understood and compared. There are other chiefly intricate texts, including legal documents, where state-of-the-art techniques like BERT have been used to classify the texts with considerable precision, revealing how NLP models can be used in a legal context [31].

Deep learning and word embedding-based studies

The results of recent studies show that deep learning and advanced embeddings work well for sorting legal cases in many different areas. In the beginning, the most common practice was using only static word embeddings and classical neural networks. For this example, the predictions of criminal charges and fines using text CNN, RNN, and DenseNet models were applied to BDCI 2017 data. The researchers observed that a deep TextDenseNet network was more effective than RNN models, making it clear that feature hierarchies learned can be better than manually designed features based on static embeddings but are less accurate than the newly introduced fine-tuned contextual embeddings [29]. Furthermore, the CNN classification model already led to greater results on Indian Supreme Court judgments, and more improvements were made using ensemble methods. They considered a variety of embeddings, such as domain-specific ones (InLegalBERT, etc.) and generative transformers, for labeling their case documents and achieved very high accuracy by combining an encoder and the T5 model together with SMOTE and feature selection. This proves that static embeddings only capture some patterns, whereas contextualized representations (like BERT and T5) provide much better performance in this task for legal text classification [32].

Some researchers also bring in information about law and how documents are linked into neural models. Several approaches include structural or relational methods to better support the reasoning found in law for classification [33]. In this area, graph-based approaches receive attention. A framework with multiple modules finds the key legal information in the facts and hands that information over to a predictor model. The improved model outperformed other techniques on the main Chinese legal benchmark, proving that including legal features is beneficial. Some scientific investigators take advantage of unlabeled case materials for learning [34]. Furthermore, the PILOT approach starts by finding similar cases, uses them for prediction, and shows much better results than when previous information is disregarded. By using this strategy, common law’s unique issues are addressed (finding and making sense of previous rulings), unlike the earlier techniques made for civil law. Other approaches to classification arrange and organize parts of legal judgments. The authors apply a BiLSTM to analyze court sentences in child custody disputes, identifying sentences based on their role as either the request, decision, or legal reasoning. Their model would allow legal analysts to quickly find the crucial parts by spotting lengthy rulings with an F1 score over 0.90. It means deep models are not just forecasting court outcomes but can additionally spot legal issues and assign rhetorical roles in the judgments [35].

By comparison, using traditional embedding models such as Word2Vec or GloVe with simple classifiers performs much more comparatively low. An optimum RNN could not perform as well as transformers in making judgment predictions. Generally, the sector now mainly uses contextual embeddings, and today’s studies focus on tweaking or enhancing these for law-related details. Recent observations show that while small, fine-tuned models are good at solving clear tasks, they provide stronger, ready-to-use performance on complex classifications and need fewer large data sets [36]. In essence, recent research covers deep models for specific tasks as well as models that can be easily transferred across many tasks. The greatest success in case classification by law has been achieved by adapting modern NLP approaches to legal data, resulting in far better descriptions of different topics, outcomes, and other features in law [37].

Recent research on research paper classification shows the increase in the importance of metadata and semantic representations as the means of overcoming the drawbacks of traditional methods. In [38], the authors highlighted the limitations of single-label and multi-label classification techniques with the help of Word2Vec to obtain and use semantic and contextual information, which leads to better results on popular datasets without using manually determined thresholds. In a similar way, [39] focused on the importance of classification based on metadata and showed that using title and keywords together substantially improves performance in comparison to either feature, and that classical models like Random Forest, K-Nearest Neighbor, and Decision Tree provided good F-measure performance. In the same line of thinking, [40] took the next step by combining high-level semantic modeling with BERT to create high-dimensional contextual embeddings from metadata attributes, then optimizing them using genetic algorithms to minimize dimensionality and computational cost. The experiment also confirmed that the best metadata sets, especially title and keywords, are always better than the other sets of features. All these pieces illustrate the transition of the traditional statistical representations to semantic-rich embeddings and optimized use of metadata to enhance the performance of research document classification.

Despite the advantages of applying AI to examine legal cases, there is a need to acknowledge the limitations of using these methods. There are privacy risks related to sensitive legal information being processed, concerns about the reliability of model predictions in high-stakes legal settings, and the problem of AI hallucinations—where models may generate plausible but incorrect or unverifiable information. These challenges underline the necessity of an intelligent AI model that guarantees the confidentiality of the data, explainability, and accountability in making decisions, and validation ingredients for mitigating misinformation. Continuing work on using advanced approaches for further development and improvements in techniques for advanced NLP will probably lead to the legal domain, which will benefit the practice of legal research.

Data collection and preprocessing

The data set utilized in this study consists of 25,000 legal cases, with four primary columns: Case ID, Systemic Outcome, Case Title, and Case Description, freely available at an online repository [42]. To facilitate the classification of citations based on citation treatments and case outcomes, the dataset was constructed. A sample description of the corpus is presented in Table 1 to provide insight into the nature and linguistic patterns of the text used in model training.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Sample sentences from the dataset showing case categories.

https://doi.org/10.1371/journal.pone.0350673.t001

The most frequently used terms in the dataset appear as shown in Fig 2. A word cloud formed from major terms extracted from legal documents, which are prominently the key terms with more frequent appearance within court cases, judicial decisions, and legal proceedings. With highly characteristic words including ‘cases,’ ‘court,’ ‘tribunal,’ ‘decision,’ and ‘matter,’ it can be indicated that there is a very strong interest in legal adjudication, dispute resolution, and case law. Terms include ‘applicant’ (or ‘claimant’) as an official party in legal proceedings, ‘appeal’ or ‘respondent’ against whom the action is taken, ‘evidence’ as a legal document, ‘principal’ or ‘principle’ about which an issue is raised in a court ruling, and ‘provision’ or ‘law’ in the context of a legal ruling. This is an indication that the words HCA and CLR are abbreviations for certain legal reports or jurisdictions. In general, with the help of this word cloud, one can visualize well the critical terminology in legal texts for further analysis and legal document classification.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Most frequent words in dataset.

https://doi.org/10.1371/journal.pone.0350673.g002

The text data in these columns was first reviewed for missing values and coherence. Due to the large amount of data, missing values were treated systematically and limited the entries with no textual information. In the textual data preprocessing process, some important steps were taken to eliminate noise and prepare the text for analysis. First, non-alphanumeric characters, such as special symbols, unnecessary punctuation, and digits, were eliminated using the following formal expression as in 1:

(1)

where is the original text, is the cleaned text, and represents each character in text.

In addition, raw words, such as those that are meaningless when the article is analyzed (e.g., ‘the,’ ‘and’ ‘is’), were excluded from the text [43]. A list of stop words was obtained from a standard repository and removed from each document using the following process, as in 2:

(2)

Consequently, all the URLs and web addresses referenced in the texts of the cases under analysis were omitted because they are irrelevant to the legal context. This was done through regular expressions designed to match URL patterns, following in 3:

(3)

Embeddings and modeling

In this paper, the application of expanding sentence embeddings with LSTMs in the classification of legal case text has been introduced. Each document of legal cases transforms into a vector into a dense real space with a fixed dimension, which is supposed to reflect meaning of sentences, where is the dimensionality of the embedding space, working shown in Fig 3. This approach proves to be useful as the model is likely to capture on the context and semantics of the legal text needed to categorize the legal cases by their outcomes and citation treatments [44]. Further, used pre-trained sentence embedding models where to convert the text of each legal case into a relatively numerical format, formalized as in 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Working of sentence transformer.

https://doi.org/10.1371/journal.pone.0350673.g003

(4)

Where denotes the embedding function, and is a transformation function that converts each word into its corresponding word embeddings . The final sentence embeddings is obtained by combining the embeddings of individual words, often by averaging words, defined in 5.

(5)

Recurrent Neural Network (RNN), i.e., LSTM, was then used on the embeddings of the input sentences. LSTMs are good for sequential data, for example, text, since they’re capable of tracking long-term relations and dependencies in a sentence [45]. This is particularly the case when the legal texts are being translated, and often, the meaning of a whole sentence greatly depends on previous sentences. This LSTM model had the ability to input the sentence embeddings from the text, acquire patterns and relational information from the text [46], and then classify each of the cases. At each time step , forget gate controls the extent to which the previous cell state forgotten on the objective flow of bias term as, . Then input gate controls how much new information from the current input stored in the cell state as .

For new information, candidate cell potentially be added for the cell satte as . This state further linked with the previous one to produce new cell candidate as , to produce output as hidden state these states jointly using both semantic representation from the sentence embedding method and sequential dependencies from LSTM, as shown in Fig 4, the classification accuracy in legal case text would improve over simple sentence embeddings.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Architecture of Applied LSTM Model with Sentence Embeddings, showing the process of legal text into semantic vectors using embeddings, processing the sequence with LSTM to capture contextual dependencies, and using a fully connected layer to classify cases.

https://doi.org/10.1371/journal.pone.0350673.g004

Performance measures

To fully assess the performance of the proposed model, several evaluation measures were used, such as accuracy, precision, recall, and F1-score, as shown in Table 2. Although accuracy gives a general idea of the right predictions, it might not accurately represent the model in the multi-class classification tasks. Thus, precision and recall were adopted to measure the accuracy and fullness of predictions, respectively, and the F1-score is a balanced metric, as it is a combination of both precision and recall. These measures provide a more accurate and in-depth measure of classification performance, especially with complex textual legal data [47].

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance evaluation measures.

https://doi.org/10.1371/journal.pone.0350673.t002

Model settings

To achieve reproducibility and transparency, Table 3 shows the overall experimental environment and the hyperparameter set employed in this study. The dataset contains 25,000 legal cases that are designed as a multi-classification problem with three classes: applied, followed, and considered. An eighty percent (80) to twenty percent (20) hold-out strategy was used to split the data into a training and test set to have a uniform evaluation protocol. To represent features, sentence embeddings were used to learn the semantic structure of legal text, with a 1000 input dimension. The suggested LSTM model will include a hidden layer of 128 units, and this layer will oversee learning sequential dependencies and contextual patterns in the embedded text sequences. The LSTM produces the hidden states, which are then fed to a fully connected (dense) layer, which projects the learned representations to the output space that represents the three legal classes. The output layer has a SoftMax activation to give class probabilities to be used in the final prediction. The model was trained at a batch size of 32 (100 epochs) using the Adam optimizer and a learning rate of 0.001. The cross-entropy loss function was used to guide the optimization process for multi-class classification. These hyperparameters were chosen to balance between the speed of convergence and generalization performance. Along with the proposed model, several baseline classifiers such as Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and Extra Trees Classifier (ETC)) were trained using standard hyperparameter settings in addition to the proposed model to allow a fair comparative analysis. The popular metrics were accuracy, precision, recall, and F1-score, which were used to measure model performance. It should be mentioned that the research is based on a supervised learning paradigm; thus, the parameters associated with active learning, including iterative labeling rounds, query strategy, and termination criteria, which are determined by annotation cycles, cannot be applied. The overall design provides a stable and strong experimental design to test the effectiveness of the proposed approach.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Experimental setup and hyperparameter configuration.

https://doi.org/10.1371/journal.pone.0350673.t003

These parameters are defined to train the proposed model. The main aim is to develop an LSTM algorithm that effectively categorizes legal documents by leveraging sentence embeddings and the sequential characteristics of an LSTM network. This algorithm is intended for legal texts, which present highly context-dependent information, and encodes each sentence to a dense vector representation using the concept of sentence embeddings. These embeddings signify the meaning of the sentences and pass it to the LSTM network to understand the sequential nature between those two sentences. Finally, the last hidden state of the LSTM yields the document-level representation of the vocabulary that is then fed to a fully connected layer for classification of the input into some defined classes. The model, based on the algorithm, uses a cross-entropy loss function to optimize the prediction parameters via backpropagation. It is especially appropriate to use legal text analysis since such aggregation and context significance are vital in classification.

Algorithm 1: LSTM Model Training Process

1: Input: Legal document dataset with sentences represented as embeddings

2: Output: Predicted class labels for legal documents

3: {Sentence Encoding and Batch Preparation}

4: For each document in the dataset:

5:  Extract sentence embeddings

6:   Batch embeddings into groups for processing

7: {LSTM Forward Pass}

8: For to :

9:    For each sequence in batch

10:     Initialize hidden state and cell state

11:     For to :

12:      

13:     End For

14:     Extract the final hidden state for classification

15: End For

16: {Classification}

17: For each document representation :

18: Pass through the fully connected layer

19: End For

20: {Loss Computation and Weight Update}

21: Compute cross-entropy loss

22: Back propagation gradients though model

23: Update weights .

24: End For

25: {Prediction}

26: For each test document:

27:    Repeat steps 5–19 for prediction

28: End For

Results with ML models

The performance results of the four models averaged in the following analysis clearly depict a significant difference among the models in terms of their ability to classify the legal cases. The SVC reveals very reasonable and balanced grades in all evaluated criteria, such as accuracy, precision, recall, and F1-score, which is 86%, reflecting its ability to make correct classification, as displayed in Table 4. Regarding the DT model, the accuracy, precision, recall, and F1 score were 81%, which, like SVC, is also relatively reliable with its predictions. In terms of the RF model, the accuracy is 84%, which is best at identifying positive cases. However, the ETC model stands out from all others with total accuracy and a throughput of 89% among all metrics, indicating the best balance and high accuracy in all criteria. The stable high accuracy in all reported criteria suggests that it is the best model for this data set.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Machine learning results in percentage.

https://doi.org/10.1371/journal.pone.0350673.t004

The confusion matrices provide detailed insights into how the models performed across the three case outcome classes in Figs 5–8. Consequently, this study uses the terms “applied,” “followed,” and “considered.” SVC in Fig 5, as presented with lower misclassifications among models, presented almost equally improved predictions. For example, there was a higher accuracy of the “considered” class with 258 true positives, but some confusion with the “applied” and “followed” classes. DT in Fig 6 had a relatively balanced distribution of the probability predictions; its misclassification rate is relatively higher in all classes, especially for the “followed” and “considered” ones, which suggests that it might be overfitting on some patterns. RF shown in Fig 7 was accurate in identifying “applied” (367 true positives), whereas there was a tendency to misclassify “followed” into other classes, hence the lower recall. ETC presented in Fig 8, though it uses the same structure as the RF, lost the capability to distinguish between classes, as can be seen from the confusion matrix, where Low, Medium, and High classes were interchanged more frequently. Overall, ensemble learning models (RF and ETC) are better than individual learning classifiers (SVC and DT), with ETC having the most accurate predictions overall, which demonstrates that ensemble learning is a stronger choice to learn with text in the legal field.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Confusion matrix of SVC model.

https://doi.org/10.1371/journal.pone.0350673.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Confusion matrix of DT model.

https://doi.org/10.1371/journal.pone.0350673.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Confusion matrix of RF model.

https://doi.org/10.1371/journal.pone.0350673.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Confusion matrix of ETC model.

https://doi.org/10.1371/journal.pone.0350673.g008

In all terminology models, the ETC takes the superior performance among ML and Ensemble models in terms of accuracy, precision, recall, and F1-score, recorded at 89%, as shown in Fig 9, which is the highest among all the various models and thus qualifies it as the best model for this kind of classification.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of all applied ML models.

https://doi.org/10.1371/journal.pone.0350673.g009

Results with DL models

The results of the LSTM model with Sentence Transformer highlight its strong ability to fit the training data but reveal significant challenges in generalizing to unseen cases, as displayed in Table 5. They also posed a very low loss equal to 0.13 with very high accuracy, precision, recall, and F1-score of 0.96, which demonstrated that the training model identified patterns in the training set effectively. On the other hand, the validation gave a loss of 1.79, and the performance metrics of accuracy were 0.54. The confusion matrix in Fig 10 further analyzes the performance of the model on each of the classes. For the “followed” class, the model was able to predict 262 of the correct classes that belonged to the “followed” and showed relatively poor performance while classifying the other classes.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Results with deep model in percentage.

https://doi.org/10.1371/journal.pone.0350673.t005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Confusion matrix of LSTM model.

https://doi.org/10.1371/journal.pone.0350673.g010

In the “applied” cases, 273 were correctly classified, although a good number of them were classified as “considered” or “followed”. Likewise, while for the “considered” class the model correctly classified 171 cases, as can be observed, the number of cases misclassified into the other two classes tells a different story. Such an aspect shows that although the model has the capability of making predictions, it does not differentiate between different classes, which creates confusion. The training graph in Fig 11 underpins these observations with increased training accuracy and the corresponding decreased training loss over epochs, indicating the good learning of the function on the training dataset. This difference in the training and validation accuracy means that a model has overfit, that is, it has built into its work only the training dataset, and the function does not generalize any information to other cases.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Training graph of LSTM model.

https://doi.org/10.1371/journal.pone.0350673.g011

Comparison of the proposed model with ML

Differences in effectiveness and response of the models are observed, as in Table 6, when comparing the ML models with the DL model. The specific analysis of the SVC model ranked well among the ML models. As for the LSTM model, it was able to train at a higher level of accuracy for the tuning data. This difference implies that the LSTM model exploits detailed features in the training dataset. In addition, the performance of the occurring ML models, primarily RF and SVC, was less effective for the overfitting session to both the training and validation datasets, as compared to the LSTM.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Comparison of ML models with the proposed model (%).

https://doi.org/10.1371/journal.pone.0350673.t006

Comparison with existing studies

The proposed LSTM model, combined with enhanced sentence embeddings, outperforms all the benchmark models in the classification of legal documents with an amazing Training accuracy of 96%. This result sets our proposed model as a promising improvement based on past techniques, as displayed in Table 7. Earlier works using conventional machine learning techniques, including RF and SVM, were indicated to predict specific contexts with accuracies of over 81% to 94%. However, these methods depend heavily on the handcrafted features and do not possess the ability to model the complex semantic structures, which are found in legal documents. While deep learning models such as BiLSTM have been considered in prior work, they have not yet been able to fully solve for the richness of legal document classification as demonstrated by our LSTM-based method. In addition, modern models, including BERT and T5, have promising performance across 79–91% accuracy.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Comparison of proposed with existing studies’ results in %age accuracies.

https://doi.org/10.1371/journal.pone.0350673.t007

Thus, the lightweight LSTM model focuses on the use of sentence embeddings, which enables a better understanding of the fine semantic features that the main aim is to capture to better comprehend the complex legal language and context. This capability not only outperforms traditional word embedding and LSTM-based work but also sets a new state of the art within the domain. This allows achieving a robust ability of mapping sequential dependencies and recognizing contextual meanings, which are especially important for the classification of legal documents – this work’s focus. Therefore, the proposed model makes it possible and shows the necessity to integrate the state-of-the-art procedures for further development of this research area.