1.1 Aims of the research

Albanian is a member of the Indo-European language family but is categorized as a separate subgroup. It is a unique and understudied language with a rich literary tradition and a growing body of online content. However, to the best of our knowledge, there currently needs to be a corpus designed explicitly for the study of AA in Albanian.

This paper presents the first-ever reference corpus of authorship attribution in Albanian, explicitly compiled to develop and evaluate AA models. The corpus consisted of newsroom columns and literary works and was created through a combination of web scraping and manual book scanning. We describe the corpus creation process, including the selection of texts, the annotation process, and the data collection approach. We also conduct experiments for AA on this corpus, using a range of features and machine learning (ML) techniques, and discuss the implications of our findings for future research in this area.

By newsroom columns, we refer to the written content produced by journalists and editors in a newsroom, including news articles, opinion pieces, analysis, and other forms of editorial content. While literary works are written works considered to have artistic or literary merit, which can include various forms of fiction, poetry, drama, and non-fiction, examples of literary works include novels, poems, short stories, and essays.

1.2 Research questions

Given our examination, we have carefully crafted several research questions to further investigate the problem at hand. These research questions are as follows:

1. RQ 1: What are the most effective linguistic features for accurately identifying the authors in Albanian texts?
2. RQ 2: Can machine learning approaches be effectively used for authorship attribution in Albanian language texts, and if so, which algorithms perform the best?
3. RQ 3: What are the effects of multiclass classification strategies on authorship attribution?

1.3 Contributions of the paper

This study has contributed significantly in the following ways. First, we provide a novel resource for studying AA in Albanian, which can be a reference for future research in this field. Second, we demonstrate the usefulness of this corpus in developing and evaluating AA models. Our work aims to advance the understanding of AA in Albanian and facilitate the development of more effective methods for this problem.

1.4 Organization of the paper

The remaining sections of this paper are organized as follows. The next section provides an overview of the existing research on AA. Section 3 compiles the first-ever extended Albanian authorship attribution corpus (A3C-e). In Sections 4 and 5, we report our experiments and the resulting findings, addressing the research questions. Finally, we conclude with a summary of our findings and suggestions for future work in Section 6.

2.1 Feature extraction

One of the critical challenges in AA is the extraction of features and patterns that can be used to distinguish the writing styles of different authors. Researchers have used a variety of linguistic traits for the AA problem, including lexical, morphological, syntactic, semantic, and structural features.

Lexical features [17, 25] include the use of specific characters, words, and n-grams that capture the distinctive word usage patterns in an author’s writing. The text can be analyzed at the character level and word level. These features include some basic measures that can be calculated for any text, such as word length [7, 9, 17], frequency of specific words [25], and the number of words per sentence [19]. Khatun et al. [33] claimed that character-level features (the number of characters [9, 11, 12, 17, 34], letters [10, 12, 27, 34], the use of digits [10–12, 34], and white spaces [11, 12, 34]) are sufficient for AA. Additionally, techniques such as co-occurrence of word pairs [19], vocabulary richness [10–12, 24, 34], and n-gram analysis [7, 9, 11, 12, 14, 32, 35, 36] provide further insight into the author’s writing style.

Morphological features focus on the grammatical structure of a text and include part-of-speech (POS) tags and inflectional forms. Researchers have used POS tagging to identify the frequency of specific tags [18], such as nouns, verbs, and adjectives [29]. Additionally, function words [17, 27, 37], stop words [5, 10, 14, 24], and POS n-grams [3, 15, 27], have also been used as a feature, with the most frequent n-grams being selected for analysis. In addition to POS tagging, morphological features include inflectional forms, such as verb tense and word pluralization [7]. Morphological features are an essential aspect of stylometry, as they have been used to identify patterns in how authors use language.

Structural features, such as the organization and structure of texts, have also been used in AA. This includes the analysis in sentence- [3, 19, 28, 34, 38], paragraph- [19, 28], and document-level [12] as reported in studies by Romanov et al. [25] and Hriez et al. [17]. Belvisi et al. [2] and others have analyzed features such as greetings and farewells in emails and the average length of paragraphs and sentences [27]. Islam et al. [7] also analyzed the length of words and sentences.

Syntactic and semantic features [9] can be used together to provide a comprehensive understanding of an author’s language and writing style that captures the syntactic and semantic structure of texts [17].

Syntactic features focus on the syntactic structure [25, 26] of a text, including elements such as syntactic phrases [9, 36], relations [27], and writing rules [6, 9]. Wu et al. [10] introduced syntactic structures and dependencies [10, 27] as additional features in the feature extraction process, in addition to words. Researchers can use these features to identify patterns in how authors construct sentences.

Semantic features, on the other hand, require a deeper level of linguistic analysis focusing on the meaning of the text. Researchers have employed active and passive voice verbs [17] in a text to classify authors. On the other hand, Hossain et al. [19] used semantic similarity measures on word pairs. However, the authors noted that the document-level similarity between semantics and syntactic could also contribute to misclassification in AA. The reliability of semantic analysis may need to be higher in under-resourced languages, leading to limited use of high-level semantic features in stylometry, particularly in AA.

Stylometry, the study of linguistic style, uses stylometric features to analyze and identify an author’s writing style. These features, such as readability measures [12, 26], the use of capitalization [3], punctuation [9, 12, 24, 25, 27, 28, 36], sentence structure [25, 26] and word choice [10–12, 27], are statistical properties of the author’s writing style. Combining different feature categories has been shown to improve the performance of AA, as demonstrated by Hriez et al. [17], Wu et al. [10], and Ramezani [9].

Researchers use these features to study writing styles and identify author-specific language characteristics. AA techniques have been extended in various languages by the use of ML-based algorithms for text analysis.

2.2 Classification methods

In recent years, there has been a growing interest in developing more effective methods for AA. Several studies have explored the use of these methods for AA in different languages.

Researchers have focused on developing automatic methods for AA through the effectiveness of ML-based techniques in automated classification, using various algorithms to classify text samples based on authorship.

Some popular ML-based algorithms used in AA include Support Vector Machines (SVMs) [6, 32, 36], Bayesian classifiers [12, 18], k-Nearest Neighbors (k-NN) [11, 34], Random Forests (RF) [16], Stochastic Gradient Descent (SGD) [19], Tree-based classifiers [14, 16, 17, 28], and Logistic Regression (LR) [10, 12, 18, 35]. In most studies, the authors used a combination of these algorithms to classify the text. A diverse set of features and different classifiers with ensemble learning methods [4, 29, 38] has been used to classify authorship. Other ensemble techniques, such as gradient boosting and a voting classifier, were used by Singh et al. [27] and Taha et al. [39]. In this way, Misini et al. [40] have presented various techniques used in authorship-related tasks in different languages.

In [2], researchers use a new technique to short-text authorship detection. They calculated the similarity of n-gram vectors using three different distance metrics [3]. Furthermore, recent work includes deep learning (DL) techniques for AA.

DL techniques, such as convolutional neural networks (CNNs) [12, 19, 33], recurrent neural networks (RNNs) [41] (long short-term memory (LSTM) networks [8, 15, 25], gated recurrent unit (GRU) [8]), have been used to analyze text data and identify authors based on their writing style. A different model for author classification has been proposed recently. A self-attention mechanism is employed in [10, 26] to model word dependencies in texts, improving model performance when combined with CNNs or transformers [20, 25].

Pre-trained language models [42] have become essential in NLP, providing significant improvements in various tasks. Prominent models such as BERT [43, 44], RoBERTa [45], XML-RoBERTa [46] have set new models for performance and efficiency in NLP applications.

In AA, word embeddings have been used in ML models to identify the author of a text. The distinct distribution of words a specific author uses in the vector space allows the model to distinguish the author’s writing style. Popular word embedding models include word2vec [19, 41], GloVe [12, 19, 41], Skip-gram model [33, 41], and fastText [12, 19, 33, 41]. Researchers actively study AA, and the field continues to evolve with new methods and approaches.

2.3 Authorship attribution corpora

One of the key elements in research on authorship attribution is using the dataset, or collections of texts, to evaluate the performance of different techniques. A variety of corpora have been used for AA in other languages, including English [30, 37, 47, 48], Spanish [30, 48], French [48], Chinese [20] among others. However, there is a scarcity of annotated corpora for under-resourced languages, such as Urdu [11, 12], Bengali [19, 24, 31], Arabic [17, 32], Albanian [13], and Turkish [6]. The use of appropriate corpora is crucial for AA research, as the corpora characteristics and features extracted from the dataset can influence the study results.

Our focus in this study is on corpora that consist of either newsroom columns or literary works. These are closely aligned with the composition of our novel corpus, providing a relevant and applicable context for our research. Later in this section, we will also consider other types of documents, further broadening our understanding of AA across different genres and styles.

To provide an overview of the research on AA in different languages, we have compiled a table of corpora used in previous studies (Table 1). The table includes information on the language, the types of texts (literary or columns), and the corpus size. This table highlights the diversity of corpora available for this purpose.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Authorship attribution corpora in different languages.

https://doi.org/10.1371/journal.pone.0310057.t001

As demonstrated in Table 1, several studies have used different corpora for AA in various languages. These corpora have been used to evaluate the performance of AA techniques. However, the need for more standard datasets, especially for low-resource languages, makes AA challenging. Corpora, such as UNAAC-20 [12] in Urdu, and BAAD16 [19, 31, 33] in Bengali, were created using a semi-automatic approach of manual validation and automated scraping, and are significant contributions to the field.

In the field of AA, researchers have used a variety of corpora (collections of texts) to study the writing styles of different authors. These corpora consist of different types of documents, including literary works [13, 17, 18, 20], newsroom columns [1, 7, 12], tweets [3, 32], blog posts [25, 26], and other short written texts [5, 10, 27, 34, 36].

Researchers have developed several corpora specifically for performing AA on literary works, such as novels [15, 16], poems [20], and other written works [28]. Some common features include lexical features [15, 17, 31] (character- [16, 20] and word-level [28]), n-grams [15, 17], POS tags [15, 18, 28], syntactic features [15, 17], and structural features [17, 28]. Also, the use of function words [17], rhythmic features [16, 17], and topic modeling [20] gave favorable results. ML-based algorithms used include SVM [16, 18, 28], Decision Trees (DT) [16, 17, 28], and Neural Networks (NN) [24, 41] (CNN [15, 20], LSTM [15, 31]). The use of transformers using multi-head attention [20, 31] also has been proposed to advance the AA research.

To perform AA on newsroom columns corpora, researchers have utilized techniques such as ML-based algorithms such as SVMs [12], k-NN [11], and LR [12]. Also, neural networks [7, 12] including CNNs [12] and RNNs (LSTM [8, 12, 31], GRU [8]) have been applied. Other techniques that have been employed include lexical analysis [1, 9, 10, 31], morphological analysis which involves POS tags [9], and stop words [10], and structural analysis [7, 12, 25].

In the AA domain, researchers have made progress in identifying writing styles and attributing anonymous texts, but there are still challenges to be addressed. One key challenge is the limited information available for analysis, making it difficult to identify and analyze the linguistic patterns and structures indicative of authorship.

Albanian has received limited attention in previous research as a resource-poor language in terms of NLP. To address this challenge, we have developed a novel reference corpus for attributing authorship in Albanian. It is the first reference corpus for the Albanian language. The complete workflow used in building the corpus is described in the following section.

3.1 Data collection methodology

Creating a corpus of text samples attributed to specific authors is critical to AA. When selecting a corpus, it’s essential to consider corpus size, type of texts and authors included, and alignment with the research questions as discussed in [59, 60]. By collecting a large and diverse set of texts from various sources, the corpus enables to capture the full range of variation in language use and evaluate the performance of different AA techniques under various conditions.

The data collection methodology involved scraping multiple news websites to gather various columns. Book scanning was also carried out to collect literary works from the contemporary literature period.

3.1.2 Book scanning for literary works.

The book scanning process in the Albanian AA task was crucial for collecting the literary works by various authors. This process involved digitizing physical books by capturing images of their pages and converting the images into text using Optical Character Recognition (OCR) technology. This process aimed to create a representative corpus of literary works written in Albanian.

To obtain a comprehensive and diverse dataset, we chose to scan books from the University of Prizren Library, which has a rich collection of contemporary literary works in Albanian.

The first step in the book scanning process was to select the books to be digitized. It was done by reviewing the available books in the University of Prizren library and selecting those written by well-known authors from contemporary literature in Albanian. To ensure the best quality of our corpus, we decided to focus on authors with many books in the library. The authors selected from contemporary literature represented diverse styles and genres.

Once the authors were selected, we began the process of book scanning using the Viisan book scanner and its accompanying Cambook application. The Viisan book scanner is a high-quality device for digitizing large quantities of printed books and manuscripts. The Cambook application is a user-friendly software that allows for easy and efficient scanning of books. It utilizes a high-resolution camera to capture the pages of a book, which is then processed to produce digital images. The selected books were then placed on the Viisan book scanner and scanned in high-resolution grayscale mode, resulting in high-quality digital images of the pages.

Approaching the book scanning process with caution was necessary due to copyright considerations, and selected pages were scanned to avoid copyright infringement. A systematic approach was adopted to achieve this, where every fifth page of the book was scanned, alternating between scanned and unscanned pages. This method ensured that the content collected was limited under copyright laws while still allowing for the collection of a representative sample of the author’s writing style.

The scanning process was carried out in time sequences over approximately one month, ensuring that all the selected books were scanned with high quality and accuracy. The scanned books were then saved as digital files, with each scanned page being saved as an image file.

As a result of the careful scanning process, high-quality text outputs were produced that accurately reflected the original content of the author’s writing. The next step was to convert the image files into text format, which was then saved in a UTF-8 encoding compatible with Albanian text. This step was necessary to make the text data usable for AA analysis.

The scanned images were processed using OCR technology to convert the images into text. It involved analyzing the scanned images to recognize the text and convert it into machine-readable form. The OCR process was done using the integrated software of the CamBook application designed to recognize text accurately in various languages, including Albanian. The resulting text outputs were saved in text files, allowing easy integration with our corpus.

To build a robust and comprehensive corpus for AA, we took great care to ensure the quality of the data collected through the book scanning process. The manual validation and improvement of the scanned files were critical in ensuring that the corpus was accurate, complete, and suitable for use in the AA task.

Once the OCR process was complete, the resulting text files were carefully reviewed to ensure high quality. The manual validation process involved checking the scanned pages for any missing text, incorrect OCR outputs, or other errors that might have occurred during the scanning process. Any such errors were corrected manually by rescanning the affected pages or manually editing the scanned files. Efforts were made to ensure the text files were well-formatted and free of any issues that could negatively impact their use in the AA task.

Finally, the extracted text was pre-processed to remove irrelevant information and properly format the text for use in the corpus. It included removing page numbers, footnotes, headers, and other non-textual elements and ensuring the text was formatted correctly and error-free. The pre-processed text was then used to create a comprehensive corpus of Albanian literary works that accurately represented the original authors’ writing.

The resulting corpus of literary works serves as a valuable resource for research in Albanian AA, ensuring its availability for future study and analysis.

In this study, we used the Viisan book scanner and the Cambook application to digitize a collection of literary works from the University of Prizren library. Collecting and compiling a corpus of literary works for AA in Albanian required automatic and manual techniques. By carefully selecting authors, using high-quality scanning equipment, and performing manual validation and improvement, we created a corpus suitable for use in the AA task. We compiled our corpus using a hybrid methodology, combining automatic and manual steps to ensure reliability. Fig 1 illustrates the complete workflow used in the corpus building.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Process of building the A3C-e corpus.

https://doi.org/10.1371/journal.pone.0310057.g001

The icons included in the graphic illustration of corpus development for each dataset are obtained from the Noun Project.

This corpus will be a valuable resource for researchers working on Albanian AA. It will contribute to a better understanding of the challenges and opportunities of this task in this under-resourced language.

3.2 Specifications of A3C-e corpus

This section outlines the specifications of the A3C-e corpus, including its size and summary statistics for several NLP features, to understand the data and its potential uses better. The proposed A3C-e corpus is a comprehensive and diverse collection of newsroom columns and literary texts, and its specifications are detailed in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The corpus size and authorship statistics of the A3C-e corpus.

https://doi.org/10.1371/journal.pone.0310057.t002

This table presents three crucial levels of linguistic information about the corpus, including corpus size, authorship statistics, and POS tags. The first level provides a comprehensive overview of the size of the corpus, while the second level provides insights into the distribution of text data among different authors. Finally, the last level showcases the POS-related information in the corpus.

Despite previous work on POS tagging for the Albanian language [61–63], the POS-related information in this corpus was generated using an internally developed POS tagger that was trained on a large dataset of annotated texts in the Albanian language. The tagger is developed based on deep learning and achieved a high F1 score of 97%, making it a reliable tool for extracting POS-related statistics from the corpus. The tagset used in the corpus contains 58 POS tags, including abbreviations, adjectives, adverbs, articles, conjunctions, interjections, nouns, cardinal and decimal numbers, prefixes and suffixes, pronouns, prepositions, particles, verbs, special characters, and punctuations. The tags were assigned to each token in the corpus to provide information on the grammatical category and function of words in context.

The proposed A3C-e corpus represents a rich and extensive data collection for research and analysis. With 12,406,976 words and a vocabulary size of 4,631,714.9 unique tokens, it is a comprehensive collection of 7,684 literary works written by ten authors and 8,234 newsroom columns by 50 different authors. The corpus is a valuable resource for those researching AA in the Albanian language.

Fig 2 presents a breakdown of samples by author, showing the number of samples collected from each author and how they are distributed. This information provides a clearer picture of the dataset’s composition and helps understand the data’s variability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of samples by author.

https://doi.org/10.1371/journal.pone.0310057.g002

Overall, creating this novel corpus represents an essential step toward developing automatic AA models for the Albanian language and opens up new opportunities for research in this area.

4.1 Data preprocessing

The data used for our experiments consisted of a corpus of newsroom columns and literary works written in Albanian.

The newsroom column corpus consists of articles from various Albanian news sources, while the literary corpus includes texts from published books. The dataset covers a wide range of topics and genres, providing a diverse set of samples for authorship attribution analysis. The corpus includes a total of 15,918 samples, with 8,234 samples from the newsroom columns and 7,684 samples from the literary works. The total word count for the newsroom columns is 7,609,771, while the total word count for the literary works is 4,797,205. The average number of words per author is 206,783, with the detailed breakdown of samples per author presented in Fig 2.

4.2 Text representation models

This subsection presents the term frequency-inverse document frequency (TF-IDF) text representation model we used in our experiments. The raw text data was transformed into numerical representations using this model, which is then employed for feature extraction and classification. The TF-IDF model [4, 9] is a popular method that represents each document as a vector of term frequencies weighted by inverse document frequency.

4.3 Feature extraction and selection

One of the critical challenges in AA is extracting relevant features from preprocessed text data that can help distinguish between different authors.

While performing research in a low-resource language like Albanian, identifying the appropriate features for AA might be challenging. This is mainly due to the limited availability of NLP tools and resources. However, carefully, we identify relevant features that can be extracted despite the limited resources available. A combination of lexical, morphological, and structural features has been used to capture different aspects of an author’s writing style. Without specific NLP tools and resources for Albanian, selecting these feature categories is a pragmatic choice that can yield valuable insights into the language’s structure and usage patterns. We extract features at multiple levels—character, word, sentence, and document—to provide a comprehensive and robust feature set. The complete set of extracted features is presented in Table 3 and was carefully selected based on their relevance to the AA task.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Proposed feature sets and description.

https://doi.org/10.1371/journal.pone.0310057.t003

Lexical features refer to the author’s vocabulary and word usage patterns. Frequency analysis of certain words or term types can help reveal patterns in how authors use language [11, 12]. This study extracted lexical features at the character- and word levels. To extract lexical features, researchers commonly use character n-grams, where they identify and analyze groups of characters to determine patterns of language use. Word n-grams, on the other hand, are groups of words that are analyzed to determine the relationship between different words in the text. Other lexical features extracted include word length, word frequency, and diversity of terms used in the text.

Moving on to morphological features, they can help identify patterns in word morphology [7], which can be particularly useful in languages like Albanian, where inflection plays an essential role in expression meaning. This study used POS-related information [18] and inflectional word forms to extract morphological features. POS information refers to the syntactic category of a word based on its grammatical function in a sentence, such as a noun, verb, adjective, etc. Inflectional word forms involve identifying variations of words based on their person, gender, number, and other related factors. These features can capture language usage patterns characteristic of different categories of text.

Finally, this subsection discusses the extraction of structural features, which refer to the text’s organization and arrangement of words and sentences. Structural features can capture the text’s overall structure, such as sentence length and punctuation usage [7, 28]. These features can be significant for identifying patterns in how different authors organize and structure their writing. This study extracted structural features at the sentence- and document levels, such as sentence and document length. Additionally, we calculated the readability index of the documents. These features provide insight into different text categories’ organization and structure characteristics.

We employed a combination of imperative and reflective programming paradigms to obtain a complete set of extracted features. By utilizing these two paradigms’ functionality, we could identify all the user-defined functions present in the code. This process involved obtaining a symbol table of the global variables and using reflective techniques to retrieve information about the user-defined functions.

The extracted features are then used to train an ML-based model for the classification task. In addition, we present the feature selection part in the next section, which involves selecting the most relevant features that could capture important information related to AA.

4.4 Authorship detection methodologies

This subsection presents the authorship detection methodologies used in our experiments to perform authorship attribution. We compared two multiclass classification strategies commonly used in multiclass classification problems, specifically one-vs-one and one-vs-rest.

4.4.1 Multiclass classification strategies.

In machine learning and classification tasks, multiclass refers to a classification problem with more than two possible classes or categories that an input can be classified into based on its features. Multiclass classification involves classifying data into more than two classes, where each sample can only be assigned to a single class label.

Two popular multiclass classification [64] strategies are used to handle multiclass problems in ML with binary classifiers: one-vs-one (OvO) and one-vs-rest (OvR). A diagram illustrating the ML-based algorithms grouped by strategy is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Multiclass classification strategies.

https://doi.org/10.1371/journal.pone.0310057.g003

OvO involves training binary classifiers for each pair of authors and combining their predictions using a voting scheme. While, OvR consists of training a binary classifier for each author and predicting the author with the highest confidence score.

OvO and OvR have their strengths and weaknesses, and the choice of algorithm will depend on the specifics of the dataset, the classifier used, and the problem that is solved. Table 4 summarizes the pros and cons of the OvO and OvR strategies for multiclass classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Pros and cons of one-vs-one and one-vs-rest strategies for multiclass classification.

https://doi.org/10.1371/journal.pone.0310057.t004

5.1 Experimental results

This subsection presents the classification results obtained from our extensive experimentation on the new A3C-e corpus. This comprehensive corpus enabled us to test various aspects of AA, including different types of texts, such as literary works and newsroom columns. Moreover, we considered a combined setting encompassing both types of texts to evaluate our proposed methods’ robustness further.

Our experiments encompassed five ML-based algorithms, utilizing two distinct multiclass training strategies. We investigated the impact of nine different feature sets on the classification performance to ensure a broad exploration of potential feature combinations. In addition to traditional ML approaches, we experimented with two DL classification methodologies, aiming to leverage the power of neural networks for the AA task.

The A3C-e corpus was employed in three separate settings: literary works, newsroom columns, and a combination. By evaluating the performance of our algorithms in these distinct contexts, we aimed to assess their adaptability and effectiveness across a diverse range of texts. Furthermore, we conducted experiments to identify the minimum feature set required for successful AA. It will help refine the process and reduce the computational complexity of the models without sacrificing accuracy.

In the following paragraphs, we will delve into the details of our experimental results, comparing the performance of various ML-based algorithms, feature sets, and training strategies. We will also discuss the effectiveness of DL methodologies in the context of AA and explore the implications of our findings for identifying the most efficient feature sets. This comprehensive analysis aims to provide valuable insights into the state-of-the-art methods for AA in Albanian texts.

Table 5 presents the results of the first series of experiments. In it, we compare the F1 scores of different classification algorithms and various sets of features. Regarding the input data in this first experiment, we use the subset of our original corpus, namely the literary works.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Classification results of ML-based algorithms for literary works—A comparison of accuracy.

Best results for each category of experiments are given in bold.

https://doi.org/10.1371/journal.pone.0310057.t005

When the standard TD-IF schema was used for feature extraction, the best results were archived with SVC. The highest score of 0.99 is archived with the following parameters: 1) a multiclass training strategy is one-vs-rest, 2) the TF-IDF features without stop-words removal, and 3) the algorithm is SVC. It is well known that stop word removal helps in increasing the accuracy of text classification models. However, removing stop words may not improve classification results in some cases. It could be due to several reasons, such as the stop words significantly impacting the meaning or context of the text, or the classifier can handle them effectively. Additionally, the frequency of stop words may need to be higher in the dataset to impact the TF-IDF scores significantly. We strongly believe that the stop words in the case of AA positively impact distinguishing the authors, which is also supported by results presented in Table 5.

On the contrary, experimental outcomes employing manually engineered attributes yield the most favorable results when the complete array of features is implemented. In this instance, the XGBoost algorithm utilizing the OvR multiclass training methodology, combined with a combination of lexical, morphological, and structural attributes, achieves the highest accuracy rate of 0.982.

Analogous to the results delineated in the preceding table, Table 6 exhibits the same experimental configuration, albeit with the AA task applied to the newsroom columns subset within the original corpus. We discern comparable outcomes in this scenario, particularly concerning the TF-IDF attribute set. The most striking results are discerned in the context of hand-crafted features. Rather than employing a comprehensive set amalgamated with all extracted attributes, which amounts to over 1500 attributes, solely the subset of lexical and morphological features trained with XGBoost performed optimally in terms of accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Classification results of ML-based algorithms for newsroom columns—A comparison of accuracy.

The best results for each category of experiments are given in bold.

https://doi.org/10.1371/journal.pone.0310057.t006

We construe this discovery based on our domain-specific expertise. Given that newsroom columns predominantly focus on facts and information, the AA model would derive the most advantage from the utilized vocabulary, as opposed to writing style or analogous signals more closely related to literary work.

In the last experiment of this series, the entire corpus is analyzed, with results presented in Table 7. These findings are consistent with the outcomes delineated in the preceding two tables.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Classification results of ML-based algorithms for literary works and newsroom columns combined—A comparison of accuracy.

The best results for each category of experiments are given in bold.

https://doi.org/10.1371/journal.pone.0310057.t007

In the following experiments (Table 8), we evaluated our recently developed corpus on the training of DL-based models. For this purpose, we incorporated various DL training models, fastText, CNN, and Transformers-based pre-trained models like BERT-multilingual and XLM-RoBERTa-base, to demonstrate their efficacy in AA.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Classification results of DL-based models in three separate settings (literary works, newsroom columns, and a combination).

https://doi.org/10.1371/journal.pone.0310057.t008

The F1 scores achieved using fastText, CNN, and BERT models were observed to be marginally lower than those attained with TF-IDF and manually engineered features when classifying literary works. It indicates that DL-based models may still need to outperform traditional ML techniques in literary works classification.

In the case of newsroom columns, the DL classification models, particularly the CNN model, reported significantly lower performance metrics than the previously obtained measurements. This outcome suggests that the CNN model is not optimally suited for the classification of authorship in the newsroom columns corpus, potentially due to the data’s unique nature and the texts’ underlying structure.

When the entire corpus is considered, the fastText model demonstrates commendable performance, with results nearly on par with those reported in earlier experiments. This outcome highlights the potential of the fastText model in capturing and generalizing across diverse textual datasets. BERT also achieved notable accuracy improvements in specific scenarios. In contrast, the CNN model exhibited the least favorable F1 scores, which could be attributed to their architectural limitations or the need for further optimization of hyperparameters to improve their performance in the AA domain.

As the XGBoost algorithm consistently provided better results across the three separate corpus subsets and various feature combinations in our experiments, we used it as the classifier for subsequent experiments.

5.2 Feature analysis and selection

This subsection analyzes features used to classify authorship in Albanian written texts. Using SHAP values [66] and generating waterfall plots, we examine the importance of individual hand-crafted and model-generated features and their impact on the author classification task. Each bar in the SHAP waterfall plots represents a feature, with its length indicating its impact on the model’s output for a specific example. Classification models rely on an ensemble of features and their linear combinations, with no single attribute determining the classification outcome. Figs 4 and 5 show that sentence-level structural features, morphological features represented by POS tags, as well as certain nouns and adverbs, had the most significant impact on authorship classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. SHAP waterfall plot for a specific example in the combined dataset using XGBoost algorithm (hand-crafted features).

https://doi.org/10.1371/journal.pone.0310057.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. SHAP waterfall plot for a specific example in the combined dataset using XGBoost algorithm (TF-IDF features).

https://doi.org/10.1371/journal.pone.0310057.g005

Considering the extensive array of manually crafted features extracted to enhance model accuracy, we sought to examine the influence of feature selection on the performance of the trained XGBoost model in terms of classification results in the subsequent series of experiments.

Feature importance scores generated by the XGBoost model help identify the features with the most significant authorship information, guiding the selection process. We performed feature selection analysis on the combined dataset using the XGBoost classifier with hand-crafted and TF-IDF features.

This subsection describes the iterative feature elimination process [67] and its impact on model performance. The x-axis represents the number of features, while the y-axis shows the model’s cross-validated score, illustrating how performance evolves with different feature counts. The analysis identifies two key points: the peak performance number of features and a threshold number of features where performance drops by 0.02 from the peak, marked by a dashed line. This threshold indicates the balance between feature inclusion and model reliability.

The analysis in Figs 6 and 7 helps identify optimal feature configurations for the dataset, systematically evaluating how different feature subsets impact the model’s performance. This observation suggests that employing a reduced set of features may not adversely impact the model performance, and a more concise feature set may still yield reliable classification results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The relationship between feature set selection and the model’s performance across literary and column contexts (hand-crafted features).

https://doi.org/10.1371/journal.pone.0310057.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The relationship between feature set selection and the model’s performance across literary and column contexts (TF-IDF features).

https://doi.org/10.1371/journal.pone.0310057.g007

5.3 Learning curves

In this subsection, we present the learning curves to show the correlation between the training score and the cross-validated test score for the XGBoost estimator using the OvR multiclass classification strategy with different training sample sizes.

Since the training score in Figs 8 and 9 is greater than the validation score, and the difference between them gradually decreases over time, it suggests that the model is slowly beginning to generalize better to the unseen data. This improvement may be attributed to an increase in the training data size or the inclusion of more diverse examples, which allows the model to learn more representative patterns of the underlying data distribution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Learning curve for the XGBoost estimator of the OvR multiclass classification strategy (hand-crafted features).

https://doi.org/10.1371/journal.pone.0310057.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Learning curve for the XGBoost estimator of the OvR multiclass classification strategy (TF-IDF features).

https://doi.org/10.1371/journal.pone.0310057.g009

5.4 Class prediction error analysis

In this subsection, we comprehensively analyze class prediction error to investigate ML-based models’ efficacy in accurately determining a text’s authorship.

This analysis is visually presented in the following figures, offering an alternative method of displaying the confusion matrix, a tabular representation that outlines the performance of a classification model by enumerating the correctly and incorrectly predicted instances for each author. Given the substantial number of classes involved, employing a standard confusion matrix would result in unwieldy matrices; hence, we opt for class prediction and continuous error plots as our mode of communication.

A comparison of error rates in Figs 10 and 11 reveals that models trained with hand-crafted features exhibit a lower average deviation compared to those trained on TF-IDF features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Error rates for the XGBoost classifier trained with hand-crafted features.

https://doi.org/10.1371/journal.pone.0310057.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Error rates for the XGBoost classifier trained with TF-IDF features.

https://doi.org/10.1371/journal.pone.0310057.g011

Figs 12 and 13 exhibit a pattern analogous to the one described here. In Fig 13, which depicts the continuous error plot for the XGBoost classifier, a noticeably greater distribution of blue cells can be seen in comparison to Fig 12, where the same model is trained on hand-crafted features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. The correctly and incorrectly predicted instances for each author—The performance of the XGBoost classifier (hand-crafted features).

https://doi.org/10.1371/journal.pone.0310057.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. The correctly and incorrectly predicted instances for each author—The performance of the XGBoost classifier (TF-IDF features).

https://doi.org/10.1371/journal.pone.0310057.g013

Based on the error rate and continuous error plots, we conclude that despite the classification models trained on TF-IDF representations displaying superior F1 scores, a more in-depth analysis of error rates reveals enhanced robustness in models trained on hand-crafted features.