1.1 Motivation: The vietnamese administrative NER gap

Digital government transformation requires automated processing of administrative documents at scale [1]. In the field of Natural Language Processing (NLP) [2], Named Entity Recognition (NER) [3] serves as a foundational component for extracting structured information from unstructured administrative text, enabling downstream tasks such as automated document routing, compliance verification, and administrative knowledge graph construction. However, Vietnamese NLP research faces a severe dataset scarcity problem for administrative domains. Existing datasets (VLSP 2016, 2018, 2021 [4–6], and PhoNER_COVID19 [7]) focus on generic entities (Person, Location, Organization) derived from news corpora or biomedical documents. These datasets exhibit three critical mismatches with administrative requirements [8]:

1. Entity Type Mismatch: VLSP focuses on generic entities (Person, Location, Organization) derived from online news corpora. The label “Organization” fails to distinguish between a private enterprise (e.g., “Công ty TNHH ABC”) and a competent state agency (e.g., “Sở Tài chính tỉnh Thái Nguyên” (Department of Finance of Thai Nguyen Province)—a distinction with legal consequences in administrative routing. A document routed to a private company instead of the Department of Finance violates procedural law.
2. Structural Incompatibility: PhoNER_COVID19 targets medical brevity with entity types like Patient, Symptom, and Medication—typically short spans (1–3 tokens). In contrast, administrative procedures contain lengthy legal citations (VBPL entities) spanning 15–20 tokens with nested hierarchical dependencies. Models trained on short-span entities demonstrate poor generalization to multi-clause legal references.
3. Linguistic Domain Gap: News text (VLSP corpus source) employs journalistic language optimized for reader engagement, with varied vocabulary and informal phrasing. Administrative procedures use legal formalism—standardized terminology, passive voice constructions, and archaic compound words, e.g., “thẩm quyền giải quyết” (settlement authority), that rarely appear in news text. This vocabulary mismatch degrades transfer learning effectiveness.

To address this gap, we introduce PAP_NER (Public Administration Procedures NER), the first large-scale, domain-specific Vietnamese NER corpus engineered for administrative document processing.

1.2 The vietnamese administrative text challenge: A linguistic problem

Vietnamese administrative procedures exhibit a unique domain paradox that exacerbates the engineering challenge. On the one hand, these documents adhere to high-normativity, rigid, template-driven syntactic structures designed to ensure legal consistency and transparency. Standard phrases like “Hồ sơ bao gồm” (The dossier includes), “Thời gian giải quyết” (Processing time), and “Cơ quan có thẩm quyền giải quyết” (Competent authority for settlement) appear with near-deterministic regularity, suggesting that rule-based systems should suffice.

However, embedded within these rigid templates are structurally complex, nested entities that defy simple pattern matching:

1. Challenge 1: Long Legal Citations with Nested Alphanumeric Patterns. Vietnamese legal documents (entity type: VBPL) exhibit hierarchical citation structures such as “Nghị định 34/2016/NĐ-CP quy định chi tiết và biện pháp thi hành Luật ban hành văn bản quy phạm pháp luật” (Decree No. 34/2016/NĐ-CP detailing and providing measures for the implementation of the Law on Promulgation of Legislative Documents). These citations span 15–30 tokens, contain multiple sub-components (decree type, number, year, issuing code, date, issuing agency, subject matter), and must be extracted as atomic entities for legal traceability. Token-level classification models frequently fragment these citations—predicting the decree number correctly but missing the issuing agency or date—producing structurally invalid outputs.
2. Challenge 2: Ambiguous Agency Names with Hierarchical Structures. Government agencies (entity type: CQ) in Vietnam follow hierarchical naming conventions that create boundary detection ambiguity. For example, “Ủy ban nhân dân tỉnh Thái Nguyên” (Thai Nguyen Provincial People’s Committee) versus “Ủy ban nhân dân xã Hồng Tiến, tỉnh Thái Nguyên” (Hong Tien Commune People’s Committee, Thai Nguyen Province). The latter spans 10 tokens with nested geographic qualifiers. Standard Transformer models [9], using token-level Softmax classification, often truncate at the first administrative level (“Ủy ban nhân dân”), missing the critical jurisdictional qualifiers that determine routing destinations.
3. Challenge 3: Contextual Disambiguation of Generic Terms. The entity type “Object” (ĐT)—referring to citizens or organizations subject to administrative procedures—frequently overlaps with common nouns. The phrase “cá nhân” (individual) might refer to a procedural subject (entity) in one context but serve as a generic noun in another (“đối với cá nhân có nhu cầu” (for individuals who need)). Purely contextual embeddings from Transformers capture semantic nuance but lack the mechanism to enforce consistency across entity mentions within a document.

This domain paradox—high normativity with complex nested entities—demands a hybrid approach. Rule-based systems excel at exploiting normativity but fail on nested structures requiring semantic understanding. Deep learning models (BERT [10], RoBERTa [11], and PhoBERT [12]) capture semantic context but struggle to enforce the structural constraints inherent to legal text. Neither paradigm alone satisfies the engineering requirements.

1.3 Research infrastructure gap in vietnamese government NLP

Despite the urgency, the current research landscape for NER in the Vietnamese administrative domain is primarily constrained by the absence of domain-specific training infrastructure, as existing Vietnamese NER datasets reflect a severe domain imbalance. According to a 2024 systematic review [13], 18.75% of studies in this domain fail to release their datasets publicly, and research gravitates toward domains with existing resources: Public Health (12 studies) due to the COVID-19 pandemic, while core administrative functions remain underexplored—e-Government (4 studies), Human Resources (2 studies), and legal document processing (1 study). Geographically, Vietnam has limited representation compared to China (23 studies) and the USA (7 studies). For Vietnamese specifically, available datasets include VLSP (2016, 2018, 2021) [4–6] or PhoNER_COVID19 [7].

There exists limited rigorous research into hybrid architectures that systematically fuse the complementary strengths of these paradigms—specifically, integrating the contextual embedding power of monolingual Transformers (optimized for low-resource languages) with the probabilistic constraint mechanisms of Conditional Random Fields for normative domains. While Transformer-CRF combinations appear in general NLP literature, their specific efficacy for highly normative, low-resource languages under deployment constraints remains underexplored.

1.4 Contributions of this work

The central contribution of this study is a new dataset and an accompanying benchmark for Vietnamese administrative NER. Concretely, we make three contributions:

1. Domain-Specific Training Infrastructure (PAP_NER): We construct and release PAP_NER (Public Administration Procedures NER), a gold-standard corpus of 162,801 sentences with 205,807 entity annotations derived from real-world administrative procedures published by Vietnam’s National Public Service Portal and provincial government databases. Unlike existing Vietnamese datasets, PAP_NER features a fine-grained labeling scheme engineered specifically for e-Government workflows.
2. Comprehensive Deployment-Aware Benchmark: Using PAP_NER, we benchmark five established architectures (BiLSTM+CRF [14], Electra [15], XLM-RoBERTa [16], PhoBERT (Softmax) [12], and PhoBERT-CRF [17]) under realistic e-Government deployment criteria—extraction accuracy (F1-score), inference latency, and model footprint—subject to operational constraints including a 4GB memory budget, sub-50ms latency per sentence, and data sovereignty requirements for government systems. The benchmark establishes strong reference baselines on the corpus and shows that the well-established PhoBERT-CRF coupling offers the best accuracy–efficiency balance: it attains the highest accuracy (97.95% F1) among architectures satisfying all constraints, while adding only 1.6 ms latency over the base PhoBERT model (Section 5.9). Rather than proposing a new architecture, we contribute a reproducible, deployment-aware evaluation protocol that practitioners can reuse to select models for low-resource government NLP in other national and administrative contexts.
3. Reproducibility Contribution: We provide complete experimental documentation and dataset release, enabling community validation and extension of this work. This supports broader open science goals in Vietnamese NLP research.

2.1 Evolution of neural architectures for named entity recognition

The evolution of Named Entity Recognition has transitioned from statistical approaches to deep neural architectures. Early neural models, such as the BiLSTM+CRF framework [14], became the de facto standard for administrative text processing. From an engineering perspective, BiLSTM+CRF remains relevant due to its low computational footprint and its ability to enforce global constraints via the Conditional Random Field (CRF) layer [18]. However, these recurrent architectures suffer from sequential processing bottlenecks and struggle to capture the long-range semantic dependencies required to resolve nested legal citations in Vietnamese administrative documents.

The advent of Transformer-based models, such as BERT [10] and RoBERTa [19], revolutionized the field by introducing self-attention mechanisms that enable parallelized processing and deep contextual embeddings. However, standard Transformer architectures typically utilize a simple Softmax classification head (token-level classification). While sufficient for general domains, this approach often fails in normative domains (e.g., legal or biomedical), where output labels must adhere to strict state-transition rules. Consequently, recent engineering research has shifted toward Hybrid Architectures [20–22] that fuse pre-trained encoders with probabilistic decoding layers to balance semantic understanding with structural validity.

Recent work has further explored NER for low-resource and multilingual settings. Ahmad et al. [23] proposed a GPT-based approach for Urdu-English NER, demonstrating the potential of large language models for cross-lingual entity extraction. Similarly, Ullah et al. [24] applied XLM-RoBERTa to Urdu educational texts, providing insights into fine-tuning multilingual Transformers for domain-specific NER in resource-constrained languages. These studies underscore the growing interest in adapting state-of-the-art architectures to underrepresented languages and specialized domains—a challenge directly addressed by our work for Vietnamese administrative text.

2.2 Vietnamese language models and NLP resources

Vietnamese NLP has benefited from monolingual pretrained models. PhoBERT, trained on 20GB of Vietnamese Wikipedia and news text, provides 64k-token vocabulary coverage compared to multilingual XLM-RoBERTa’s 2.5k-token Vietnamese coverage—a 25-fold difference. This vocabulary advantage is particularly significant for domain-specific terminology in administrative text. Existing Vietnamese NER datasets include VLSP (2016, 2018, 2021) generic entities from news, and PhoNER_COVID19 (medical domain). These datasets address different linguistic domains but lack administrative specificity and scale appropriate for government applications. This study specifically targets the optimization of monolingual discriminative Transformers coupled with structured prediction layers, hypothesizing that a domain-optimized PhoBERT-CRF architecture represents the optimal intersection of accuracy, privacy, and efficiency for Vietnamese e-Government applications [25].

2.3 Domain-specific datasets for government NER

In computational systems engineering, high-quality training data is as critical as algorithmic design. The development of Vietnamese NER has been driven by datasets such as VLSP (2016, 2018, 2021) [4–6] and PhoNER_COVID19 [7]. However, these resources exhibit a significant “Domain Gap” for administrative engineering applications:

1. Domain Mismatch for Administrative Text: VLSP focuses on generic entities (Person, Location, Organization) derived from online news. The “Organization” label conflates legally distinct entities—private enterprises (“Công ty Cổ phần Viettel”), non-governmental organizations (“Hội Chữ thập đỏ Việt Nam”), and competent state agencies (“Ủy ban nhân dân tỉnh Hà Giang”). In administrative document routing, these distinctions carry legal weight: a citizen complaint must be processed by the competent state agency with jurisdictional authority, not forwarded to private companies.
2. Structural Incompatibility: PhoNER_COVID19 targets medical brevity (Patient, Symptom). In contrast, public administration procedures involve lengthy, nested legal citations (VBPL) that require models to resolve hierarchical dependencies spanning multiple clauses.

To support the training of robust discriminative architectures, this study introduces PAP_NER, a new gold-standard material specifically engineered to capture the normativity and recursive structure of administrative workflows.

2.4 Positioning PAP_NER in the government NER landscape

According to Ramdhani’s systematic review of Government NER research [13], the field exhibits severe domain and geographic imbalances: Domain Distribution: Public Health (12 studies) overwhelmingly dominates due to COVID-19 pandemic urgency, while core administrative functions remain underexplored—e-Government (4 studies), Human Resources (2 studies), Legal/Justice (1 study); Geographic Distribution: Research is heavily concentrated in high-resource regions—China (23 studies), USA (7 studies), India (6 studies); and Dataset Availability Crisis: 18.75% of Government NER studies fail to release their datasets publicly, creating reproducibility failures and forcing redundant annotation efforts. PAP_NER addresses these gaps through three design principles:

• Domain Specificity: Five entity types engineered for administrative workflow automation rather than generic information extraction.
• Scale and Quality: 162,801 sentences (3.2× larger than VLSP, 4.6× larger than PhoNER_COVID19) with gold-standard annotation quality (=0.85 Inter-Annotator Agreement).
• Open Access Commitment: To promote reproducibility and further research in low-resource administrative NLP, the PAP_NER dataset is made available to the research community on Zenodo (DOI: 10.5281/zenodo.18044019) under Creative Commons BY 4.0 license.

3.1 Data acquisition and preprocessing pipeline

1. Source Data Selection and Rationale. Raw textual data were harvested from two authoritative government sources to ensure domain authenticity and legal validity:
   
   • National Public Service Portal (https://dichvucong.gov.vn): Vietnam’s centralized e-Government platform publishing standardized administrative procedure descriptions for all 28 provinces and 6 centrally-governed cities. This source provides procedural templates following ISO 9001:2015 quality management standards, including structured fields for: legal basis citations, competent authorities, required dossiers, processing timelines, and service fees.
   • Provincial Administrative Databases: Direct access to procedure documentation from four North-East provinces (Ha Giang, Bac Ninh, Bac Giang, and Quang Ninh) provided via official data sharing agreements with provincial Departments of Information and Communications. These sources capture provincial variations in agency naming conventions, local legal references, and administrative terminology not present in national templates.
   This dual-source strategy addresses two engineering requirements: (1) National coverage via the centralized portal ensures the dataset represents standard administrative language applicable across Vietnam’s government hierarchy, and (2) Provincial diversity via regional databases captures linguistic variation essential for model generalization to unseen provinces during deployment.
2. Data Volume and Composition. The initial raw corpus comprised:
   • Approximately 6,800 distinct administrative procedures spanning 12 government sectors (education, healthcare, construction, business registration, land administration, transportation, social welfare, taxation, environmental protection, cultural heritage, agriculture, labor).
   • Approximately 487,000 raw sentences extracted from procedure descriptions.
3. Preprocessing Pipeline: Ensuring Data Quality and Preventing Leakage. To transform raw government documents into training-ready text, we implemented a four-stage preprocessing pipeline addressing data quality and methodological rigor:
   1. (a) Noise Removal: We utilized regular expressions (Regex) to strip HTML tags, non-standard encoding characters, and navigational boilerplates.
   2. (b) Deduplication via MinHash for Template Leakage Prevention: Administrative procedures often utilize standardized templates (forms), which pose a risk of data leakage if near-identical sentences appear in both Train and Test sets. For example, the phrase “Thời gian giải quyết: Trong thời hạn 10 ngày làm việc kể từ ngày nhận đủ hồ sơ hợp lệ” (Processing time: Within 10 working days from receipt of complete valid dossier) appears with minor variations (different day counts) in 1,547 procedures. To mitigate this, we applied MinHash with a Jaccard similarity threshold of 0.8 to remove near-duplicate sentences across the corpus.
   3. (c) Source-Stratified Splitting to Prevent Template Memorization: Even after deduplication, sentences from the same administrative procedure share structural patterns (e.g., all sentences describing required documents follow the pattern). Random train/test splitting would place structurally similar sentences in both sets, allowing models to memorize templates rather than learn entity recognition. We implemented Source-Stratified Splitting (also known as group-aware splitting): Instead of random shuffling, the dataset was partitioned by Procedure ID. This ensures that all sentences belonging to a specific administrative workflow appear exclusively in one split (Train, Dev, or Test), forcing the model to generalize to unseen procedures rather than memorizing formatting templates.
   4. (d) Sentence Segmentation and Tokenization: Vietnamese text uses spaces to separate constituent syllables (not words), creating segmentation ambiguity. The phrase “ủy ban nhân dân” (people’s committee) could be tokenized as:

• 4 syllables: [“ủy,” “ban,” “nhân,” “dân”] ← syllable-level
• 2 words: [“ủy_ban,” “nhân_dân”] ← word-level

We employed RDRSegmenter [26] from VnCoreNLP [27] for word segmentation.

3.2 Annotation Methodology and Quality Assurance

Data annotation was conducted using a strict Human-in-the-Loop (HITL) protocol to establish a Gold Standard benchmark. Unlike crowdsourced datasets, our protocol ensured domain accuracy:

1. Rule-based Pre-annotation (50% Automation).
   Administrative text exhibits high normativity, enabling regex-based extraction for structured entity types:
   • Develop Regex scripts to capture all Datetime (NG), Quantity (SL), and standardized Legal Documents (VBPL).
   • Utilize a dictionary/lexicon matching approach to detect common Object (ĐT), Agencies (CQ) (e.g., “UBND tỉnh” (Provincial People’s Committee), “Sở Tài chính” (Department of Finance),...).
   • Execute a Python script to generate pre-labeled JSON files.
2. LLM-Assisted Annotation (Additional 30% Automation). For sentences with incomplete rule-based coverage (complex agency names, nested entities, and ambiguous Objects), we employ few-shot prompting via commercial LLMs while satisfying data residency requirements.
   Privacy and Compliance: To adhere to data sovereignty regulations, all data processed by LLMs underwent strict local PII masking. Names, addresses, and identifying codes were replaced with placeholders (e.g., < PERSON > , < LOC>) via a regex script before transmission to cloud APIs. Labels were remapped to the original text locally.
   Quality Assessment of LLM Annotations: To quantify the reliability of LLM-assisted pre-labeling, we conducted a systematic quality assessment on a stratified sample of 5,000 sentences from the Label Studio revision logs. Of the LLM-generated labels, human annotators corrected 14.3% during the validation phase. Error distribution by entity type revealed that LLM accuracy was highest for Datetime (NG, 97.8% accuracy) and Quantity (SL, 96.2%), which follow predictable numerical patterns, while accuracy was lowest for Agency (CQ, 84.1%) and Object (ĐT, 86.7%) due to ambiguous entity boundaries and hierarchical naming conventions. The most common error type was boundary truncation (58% of corrections), followed by entity type misclassification (27%) and missed entities (15%). Regarding potential anchoring bias, we note that human annotators were instructed to treat all pre-labels as “suggestions only,” and the final inter-annotator agreement ( = 0.85) was computed between the two independent human annotators—not between human and LLM—so pre-labeling did not inflate the reported agreement. A fully independent re-annotation study (without any access to pre-labels) on a held-out subset would further strengthen confidence; we note this as a methodological improvement for future corpus extensions.
3. Expert Human Validation and Gap Annotation (Remaining 20% workload)
   • All pre-annotations from Stages 1–2 are uploaded to https://labelstud.io/ for human verification.
   • Expert Annotation Team: The task was performed by seven Public Administration specialists familiar with Vietnamese administrative procedures. Annotators simply scan the predictions; press Enter to approve if correct, or drag the mouse to correct the boundaries if incorrect.
   • Double-Blind Labeling: Each sentence was independently annotated by at least two annotators.
   • Inter-Annotator Agreement (IAA): To quantify label consistency, we calculated Cohen’s Kappa () [28], a standard measure of inter-annotator agreement that accounts for chance agreement, for which values above 0.60 indicate substantial agreement and values above 0.80 indicate strong agreement. The final corpus achieved an average , indicating strong agreement. A senior adjudicator resolved ambiguous instances to maintain the integrity of the ground truth.

3.3 The semantic labeling scheme

We devised a fine-grained labeling scheme comprising five core entity types critical for e-Government automation in Table 1. Unlike generic NER tasks, these labels are defined by their legal function rather than just semantic category. The data is serialized in the CoNLL-2003 format using the BIO (Begin, Inside, Outside) tagging scheme. This standard format ensures direct compatibility with modern deep learning frameworks (Hugging Face, TensorFlow).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Entity Definitions and Structural Characteristics.

https://doi.org/10.1371/journal.pone.0353166.t001

3.4 Dataset statistics and availability

Table 2 provides a comprehensive statistical breakdown of the PAP_NER dataset. The corpus comprises a total of 162,801 sentences, annotated with 205,807 entity mentions. The data is partitioned into Training (87.4%), Development (6.3%), and Test (6.3%) sets, with the training set containing 142,218 sentences to ensure robust model generalization. Data Format and Column Structure. The PAP_NER corpus is released in CoNLL-style column format. Each line contains a single token with tab-separated annotation columns: (1) Word: the segmented token produced by VnCoreNLP’s RDRSegmenter, (2) POS: part-of-speech tag (e.g., N for Noun, V for Verb), (3) Chunk: syntactic chunk tag in BIO format (e.g., B-NP for beginning of a Noun Phrase), and (4) NER: the named entity tag in BIO format (e.g., B-CQ for beginning of an Agency entity, O for tokens outside any entity). Sentences are separated by blank lines. For our experiments, only columns 1 (input) and 4 (target label) are used; POS and Chunk tags are provided as supplementary annotations for researchers wishing to incorporate syntactic features.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. PAP_NER Corpus Statistics (Split Distribution).

https://doi.org/10.1371/journal.pone.0353166.t002

In terms of entity distribution as in Table 3, the dataset exhibits a natural skew reflecting the administrative domain:

• Dominant Classes: The most frequent entities are Object (ĐT: 67,001 entities, 32.6%) and Agency (CQ: 62,915 entities, 30.6%), collectively representing 63.2% of entities, which reflects the heavy presence of procedural subjects and government bodies in administrative texts.
• Minority Class: Quantity (SL: 14,722 entities, 7.2%) is underrepresented, as numerical specifications (fees, counts) appear less frequently than semantic entities.
• Temporal & Legal Context: There is a significant representation of Datetime (NG) (35,647 instances) and Legal Documents (VBPL) (25,522 instances), underscoring the importance of temporal constraints and legal citations in public administration.
• I-tag prevalence: I-VBPL is the most frequent entity tag (84,604 tokens), reflecting long legal citation spans.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Detailed Token-Level Label Distribution.

https://doi.org/10.1371/journal.pone.0353166.t003

4.1 Problem formulation and notation

Let denote an input sequence of n tokens derived from a Vietnamese administrative sentence, where each represents the i-th token. Let be the set of predefined entity tags following the BIO tagging scheme:

where distinct labels. Our objective is to predict the optimal sequence of labels , where , that maximizes the conditional probability:

subject to the structural constraint that y forms a valid BIO sequence (e.g., an I-VBPL tag must be immediately preceded by either B-VBPL or I-VBPL, never by O or tags of different entity types).

Key Engineering Challenge: Standard classification approaches model each token’s label independently as failing to enforce structural validity. For instance, a token-wise Softmax classifier might predict the sequence [O, I-VBPL, B-CQ], which violates BIO constraints (I-VBPL cannot follow O). While post-processing heuristics can correct some violations, they are ad-hoc and may discard valid entity candidates. Our solution employs structured prediction via Conditional Random Fields to model label dependencies explicitly.

4.2 Architecture overview: Dual-engine design

Fig 1 presents the PhoBERT-CRF architecture which comprises three computational stages organized as a dual-engine pipeline:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The proposed PhoBERT-CRF Hybrid Architecture for Vietnamese Administrative NER.

The framework couples a Semantic Engine (PhoBERT) to capture deep contextual embeddings with a Structural Engine CRF. This dual-engine approach allows the model to leverage transfer learning for semantic understanding while strictly enforcing the probabilistic state-transition constraints required for normative legal templates. The right panel details the internal structure of the PhoBERT Transformer block used in the encoding layer.

https://doi.org/10.1371/journal.pone.0353166.g001

1. Semantic Engine (PhoBERT): Responsible for learning deep contextual representations that capture:
   • Lexical semantics: Understanding that “ủy ban” (committee) + “nhân dân” (people’s) forms a compound administrative term.
   • Long-range dependencies: Relating “Nghị định” (decree) at position i to its issuing code “NĐ-CP” at position i + 3 within a legal citation.
   • Contextual disambiguation: Distinguishing “cá nhân” (individual) as an entity vs. a common noun based on surrounding procedural language.
2. Structural Engine (CRF): Responsible for enforcing sequence-level constraints:
   • BIO validity: Ensuring I-tags follow corresponding B-tags (e.g., learning ).
   • Entity coherence: Preventing label switching mid-entity (discouraging transitions like I-VBPL → I-CQ within a span).
   • Boundary detection: Learning domain-specific boundary patterns (e.g., legal citations often end before prepositions like “về” (about), “của” (of)).

This separation enables modular optimization: the Semantic Engine is initialized from pre-trained PhoBERT weights (transfer learning), while the Structural Engine is learned from scratch on PAP_NER’s administrative-specific transition patterns.

4.3 The semantic engine: PhoBERT contextualized embeddings

4.3.2 Input representation and embedding layer.

For an input sequence , the model first maps each token to a dense vector representation. The embedding for token is constructed as the sum of its token embedding, position embedding, and segment embedding:

4.3.3 Transformer encoding: Multi-head self-attention and feed-forward networks.

The embedding matrix is processed through N = 12 Transformer layers. Each layer ℓ applies two sub-modules with residual connections and layer normalization:

1. Multi-Head Self-Attention (MHSA):

Self-attention computes contextualized representations by attending to all positions in the sequence:

where: (Query matrix), (Key matrix), (Value matrix), (dimension per attention head), is the input from previous layer.

Multi-head attention parallelizes this computation across 12 heads:

where and projects concatenated heads back to model dimension.

1. 2. Position-wise Feed-Forward Network (FFN): After attention, each token’s representation is independently processed through a two-layer FFN:

where: (expansion layer), (projection layer), GELU (Gaussian Error Linear Unit): where is the cumulative distribution function of the standard normal distribution.

1. Complete Transformer Layer:

After 12 layers, we obtain the final contextualized representations:

where each encodes token in the context of the entire sentence.

4.4 Emission score calculation: Projecting semantic features to label space

The Transformer output H serves as input for emission score calculation. We project each hidden state into the label space using a learnable linear transformation:

where: is the emission weight matrix; is the emission bias vector; and is the emission score matrix.

The element represents the unnormalized semantic fitness score of assigning label to token based solely on its contextualized representation , without considering label sequence structure.

1. Interpretation of emission scores

• High : The model believes token is semantically likely to begin a legal document citation (e.g., “Nghị định,” “Luật,” “Thông tư”).
• High : The model believes token is semantically outside any entity (e.g., prepositions “tại,” “của”; verbs “nộp,” “giải quyết”).

Why not Softmax here? Standard token classification applies Softmax to obtain probabilities and predicts independently for each token. This approach has a critical flaw for structured prediction. CRF layer (next section) models transition dependencies to maintain entity coherence.

4.5 The structural engine: Conditional random field for sequence modeling

To enforce structural validity, we integrate a Conditional Random Field (CRF) layer that models the conditional probability of the entire label sequence y given input x.

4.5.1 CRF score function.

The CRF defines the score of a label sequence as the sum of emission scores (from PhoBERT) and transition scores (learned by CRF):

where: is the emission score for assigning label to token i (from Section 4.4); is the transition score from label to label ; is the transition matrix, a learnable parameter.

Boundary conditions: To handle the first token (which has no predecessor), we introduce a special START state: represents the score of beginning a sequence with label y₁; represents the score of ending a sequence with label .

Transition matrix interpretation: The transition matrix A encodes the “grammar” of administrative entity sequences. These transition patterns are learned automatically from data during training, not hand-coded. The CRF discovers that Vietnamese administrative text exhibits: Long entity spans (high self-transition scores for I-tags); Strict BIO validity (impossible transitions to I-tags without matching B-tags); and Domain-specific boundary conventions (legal citations often end before prepositions like “về,” “của”).

Implementation of BIO Constraints. Our implementation uses the standard pytorch-crf library, which learns transition scores entirely from data without explicit hard masking. To verify that structurally invalid BIO transitions do not occur in practice, we conducted a post-hoc analysis of the learned transition matrix on the trained PhoBERT-CRF model. All structurally illegal transitions (e.g., , where ) received learned log-transition scores below −8, corresponding to probabilities below . On the full test set (10,278 sentences, approximately 334,000 tokens), Viterbi decoding produced zero structurally invalid BIO sequences. This empirically confirms that the soft-constraint formulation, given sufficient training data (142,218 sentences), converges to de facto hard constraints—the model autonomously learns that valid BIO sequences are the only sequences with meaningful probability, without requiring explicit transition masking.

4.5.2 Conditional probability via softmax over all sequences.

The CRF models the conditional probability of label sequence y given input x by normalizing the exponential score over all possible label sequences:

where: Numerator: scores the specific sequence y; Denominator: is the partition function, summing over all possible label sequences (e.g., for a 50-token sentence).

Computational challenge: Naively enumerating all sequences is intractable. The partition function is computed efficiently via the forward algorithm using dynamic programming, with complexity .

1. Forward Algorithm for Partition Function:

Define as the sum of scores of all partial sequences ending at position i with label y:

Initialization:

Recursion: For :

where prevents numerical overflow.

Partition function:

Complexity: (linear in sequence length)

4.6 Training objective: Negative log-likelihood loss

During training, we maximize the log-likelihood of the true label sequence for each training example :

where: denotes all model parameters: PhoBERT weights, emission matrix W, emission bias b, and CRF transition matrix A; is the score of the gold sequence; and is the log-partition function (computed via forward algorithm).

Intuition: The loss encourages:

• Increasing : Make the gold sequence score higher by: Increasing emission scores for correct labels; or Increasing transition scores for correct label transitions.
• Decreasing : Reduce the partition function by decreasing scores of incorrect sequences.

4.7 Inference: Viterbi decoding for optimal sequence prediction

At test time, we seek the label sequence that maximizes the score function:

Since and the exponential is monotonic, maximizing score is equivalent to maximizing probability. The partition function is constant for a given input, so it cancels out in operation.

Viterbi Algorithm: We compute efficiently using dynamic programming:

Define as the maximum score of any partial sequence ending at position i with label y:

The Viterbi algorithm guarantees finding the globally optimal label sequence under the CRF model, whereas greedy decoding (selecting independently for each token) can produce suboptimal or invalid sequences.

5.1 Experimental setup and implementation details

All experiments were conducted using the PyTorch framework on a high-performance computing environment equipped with a single NVIDIA A100 GPU (40GB VRAM). To ensure reproducibility, we fixed the random seed to 42 for all initialization steps.

Training Hyperparameters: We employed a systematic hyperparameter search on the development set to identify optimal configurations. The final hyperparameters are in the Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Hyperparameter Configuration and Optimization Strategy.

https://doi.org/10.1371/journal.pone.0353166.t004

5.2 Evaluation metrics

To rigorously quantify the extraction performance of the proposed framework, we adhere to the standard CoNLL-2003 evaluation protocol. We report performance based on Precision (P), Recall (R), and F1-Score. Crucially, we utilize the Strict Matching criterion: a predicted entity is counted as a True Positive (TP) if and only if both the predicted boundary span and the entity category exactly match the ground truth. The metrics are calculated as follows:

where TP (True Positives) denotes entities correctly identified in both span and type; FP (False Positives) denotes entities incorrectly predicted or misclassified; and FN (False Negatives) represents ground-truth entities that the model failed to detect. The F1-Score, being the harmonic mean of Precision and Recall, serves as the primary metric for comparing architectural effectiveness.

1. Aggregation schemes:

• Micro-averaging: Aggregate TP, FP, FN across all entity types, then compute P/R/F1 (gives equal weight to each entity instance)
• Macro-averaging: Compute P/R/F1 for each entity type separately, then average (gives equal weight to each entity type)

We report Micro F1 as the primary metric (standard in NER literature) and provide per-entity-type breakdowns for fine-grained analysis.

Evaluation modes. Following the MUC/SemEval evaluation framework, we report four complementary evaluation modes in our detailed analysis tables:

• Strict: Both entity boundaries and entity type must exactly match the ground truth. This is the most demanding criterion and serves as our primary metric.
• Partial: Entity type must match, and the predicted span must overlap with the ground truth span (partial boundary overlap receives proportional credit).
• Entity Type: Only entity type correctness is evaluated, regardless of boundary accuracy. This isolates classification performance from boundary detection.
• Exact: Entity boundaries must exactly match, regardless of whether the entity type is correct. This isolates boundary detection from classification.

Statistical Significance Testing To determine whether performance differences between models are statistically significant (not due to random variation), we employ McNemar’s Test [31]—a non-parametric test for paired binary outcomes.

5.3 Baseline architectures and comparative framework

To establish a comprehensive comparative evaluation, we selected four baseline architectures spanning the design space from lightweight statistical models to heavy pre-trained Transformers. Our selection criteria prioritize:

• Architectural diversity: Cover recurrent (BiLSTM), discriminative Transformers (Electra, PhoBERT), multilingual Transformers (XLM-R).
• Deployment feasibility: Exclude models violating deployment constraints (e.g., PhoBERT-large, XLM-R-large exceed memory budget)
• Prior Vietnamese NER benchmarks: Include models with published results on VLSP/PhoNER datasets for contextualization.

5.3.4 PhoBERT-Base (Softmax).

To rigorously isolate the contribution of the CRF layer, we establish PhoBERT-Base (Softmax) as a critical ablation baseline. This model utilizes the identical https://huggingface.co/vinai/phobert-base encoder and hyperparameter configuration as our proposed approach, but replaces the structured CRF decoder with a standard token-level Softmax classification head. Specifically, a linear layer projects the hidden states to label logits, where predictions are generated independently for each token via . By eliminating the transition logic while maintaining the exact same semantic engine, this baseline allows us to directly verify the hypothesis that explicit structural modeling is necessary for handling the complex label dependencies in Vietnamese administrative texts, independent of the encoder’s contextual understanding.

To ensure fair comparison across architectures with different input representations, Table 5 summarizes the segmentation and tokenization strategy used for each baseline. For models using subword tokenization (all except BiLSTM+CRF), we adopt the standard first-subword labeling strategy: the NER label is assigned to the first subword token of each word, while subsequent subword tokens receive a special X label that is ignored during loss computation and evaluation. This ensures consistent label alignment across architectures with different tokenization granularities.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Input Representation and Tokenization Strategy per Baseline.

https://doi.org/10.1371/journal.pone.0353166.t005

5.4 Baseline architecture comparison

Table 6 presents key comparisons of baselines. Analysis of the computational overhead reveals that the integration of the CRF layer into the PhoBERT architecture incurs a negligible cost. Specifically, PhoBERT-CRF increases the model size by only 2MB (0.4%) and adds a mere 1.6ms (3.8%) to the inference latency compared to the PhoBERT-Softmax baseline. In terms of deployment constraints, all evaluated models, with the exception of the memory-intensive XLM-RoBERTa-Base, comfortably operate within the strict 2GB inference memory budget. Furthermore, regarding real-time applicability, all baselines demonstrate high efficiency, satisfying the sub-50ms latency requirement on standard CPU hardware.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Baseline Architecture Comparison.

https://doi.org/10.1371/journal.pone.0353166.t006

5.5 Overall performance analysis

Table 7 presents the comparative results. The hybrid PhoBERT+CRF model achieved the highest performance across all metrics.

• Strong Baselines Achievement: Our hybrid model reached a Micro F1-score of 97.95%, significantly outperforming the BiLSTM+CRF baseline (+2.01%) and the multilingual XLM-RoBERTa (+2.52%). This confirms that hand-crafted features in statistical models cannot compete with the deep contextual embeddings of Transformers in this domain.
• Performance of Pure Transformer Baselines: Evaluation of the pure Transformer architectures reveals a distinct performance hierarchy: PhoBERT-Softmax (97.51%) leads, followed by Electra (95.63%) and XLM-R (95.43%). This ranking validates two critical hypotheses regarding model selection for Vietnamese administrative NER. First, the results confirm a significant “monolingual advantage,” as the Vietnamese-specific PhoBERT outperforms the massive, multilingual XLM-R by 2.08% in F1-score, despite being half the size (135M vs. 270M parameters). This demonstrates that for specialized administrative domains, language-specific optimization is far more valuable than raw cross-lingual capacity. Second, regarding pre-training objectives, PhoBERT’s Masked Language Modeling (MLM) approach proves superior to Electra’s Replaced Token Detection (RTD) for this specific task, yielding a nearly 2% performance margin.
• The “Electra Anomaly”: Surprisingly, the Electra baseline underperformed (95.63%). We hypothesize that Electra’s generator-discriminator training objective is less effective than PhoBERT’s Masked Language Modeling (MLM) for capturing the specific vocabulary of Vietnamese administrative law.
• Semantic vs. Structural: The pure PhoBERT model (97.51%) is a close second, but the addition of the CRF layer in our hybrid model pushes the performance to 97.95%. This +0.44% gain is critical in the legal domain, where boundary precision is paramount.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Comparative Analysis of Model Performance (Strict F1-Score, mean ± std over 3 random seeds).

https://doi.org/10.1371/journal.pone.0353166.t007

5.6 Fine-grained entity analysis

The performance variation across specific entity tags in Table 7 highlights the “Domain Gap” challenges inherent to administrative texts. We analyze each entity class in order of difficulty, from the most challenging to the most regular.

5.7 Granular performance analysis across models and entity classes

To provide a more fine-grained understanding of model behavior, we present a systematic analysis of per-class Precision (P), Recall (R), and F1-score across all four architectures (Tables 8–11), complemented by the Macro F1 and Micro–Macro divergence analysis from Table 7.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Detailed Performance Analysis: PhoBERT.

https://doi.org/10.1371/journal.pone.0353166.t008

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Detailed Performance Analysis: PhoBERT+CRF.

https://doi.org/10.1371/journal.pone.0353166.t009

Download:

• PNG
  larger image
• TIFF
  original image

Table 10. Detailed Performance Analysis: BiLSTM+CRF.

https://doi.org/10.1371/journal.pone.0353166.t010

Download:

• PNG
  larger image
• TIFF
  original image

Table 11. Detailed Performance Metrics: NER-Vietnamese-Electra-Base.

https://doi.org/10.1371/journal.pone.0353166.t011

5.8 Ablation Study: The impact of structural constraints

To verify the hypothesis that “probabilistic constraints are essential for normative domains,” we conducted a fine-grained ablation study comparing the pure PhoBERT (Softmax) against the PhoBERT-CRF (Hybrid). As detailed in Table 12, the inclusion of the CRF layer yielded statistically significant improvements for structurally complex entities.

• Resolving Nested Citations: The most notable gain was observed in the Legal Document (VBPL) class, which improved by 0.96% (p < 0.05 via McNemar’s test). Legal citations in Vietnam often exhibit nested structures (e.g., “Decree 34” nested within “Decree 34/2016/NĐ-CP”). The pure Transformer often fragmented these entities. The CRF layer, by modeling transition potentials (e.g., ), successfully enforced boundary continuity, correcting these fragmentation errors.
• Robustness on Ambiguous Objects: The Object (ĐT) class, which refers to individuals or organizations participating in procedures, also saw a substantial gain (+0.78%). This indicates that the hybrid model is better at distinguishing specific administrative roles from generic nouns using global sequence optimization.
• Comparing PhoBERT (Table 8) with PhoBERT+CRF (Table 9) reveals further information:
  • Overall Gain: Adding the CRF layer improved the Strict F1-score from 97.55% to 97.95% (+0.40%).
  • Structural Correction: The primary value of the CRF layer is observed in the Strict vs. Entity Type metrics.
    • In the pure PhoBERT model, the gap between Strict F1 (97.55%) and Entity Type F1 (98.83%) is 1.28%. This indicates that the model often identifies the correct type of entity but fails to predict the exact boundaries.
    • In the PhoBERT+CRF model, this gap narrows to 0.77% (97.95% vs. 98.72%). The CRF layer effectively optimizes the state transitions, correcting boundary errors (e.g., ensuring a multi-token entity is not fragmented) and enforcing structural consistency.

Download:

• PNG
  larger image
• TIFF
  original image

Table 12. Ablation Analysis on Complex Entity Classes (Strict F1).

https://doi.org/10.1371/journal.pone.0353166.t012

5.9 Computational efficiency and deployment feasibility

For e-Government applications, model accuracy must be balanced against inference latency. We measured the average processing time per sentence (ms/sent) on a standard server configuration: Intel Xeon Gold 6226R CPU (2.90GHz), 64GB RAM, Single-threaded inference, and Batch size = 1.

• Latency Analysis:
  • BiLSTM+CRF: 15.20 ms
  • XLM-R-Base: 46.8 ms
  • PhoBERT (Softmax): 42.50 ms
  • PhoBERT-CRF: 44.10 ms
• Trade-off Justification: The addition of the CRF decoding layer added a negligible overhead of ms per sentence compared to the standard PhoBERT. Given the critical nature of legal accuracy—where a single missed citation can invalidate an administrative decision—the 0.96% accuracy gain on legal documents justifies this marginal increase in latency. The proposed framework remains well within the acceptable threshold for real-time document processing workflows (<50 ms).

6.1 Analysis of boundary truncation (agency entities)

In Case 1, the input text refers to a specific local authority: “Ủy ban nhân dân xã Đồng Văn” (People’s Committee of Dong Van Commune).

• The Baseline Failure: The pure PhoBERT model utilizes token-level classification. Semantically, “Ủy ban nhân dân” (People’s Committee) is a strong signal for the Agency (CQ) class. However, the suffix “xã Đồng Văn” (Dong Van Commune) contains proper nouns and common nouns that often appear in non-entity contexts. Consequently, the Softmax classifier prematurely terminates the entity at “dân” (People), resulting in a Truncation Error. In a digital government routing system, this is a critical failure, as it routes the document to a generic category rather than the specific local commune.
• The Hybrid Solution: The PhoBERT-CRF model successfully captures the full span. This is achieved because the CRF layer models the transition probability . The model learns that within the administrative domain, the transition probability remains high even when the token itself (e.g., “xã” / Commune) is semantically ambiguous. The global score optimization ensures the “Agency” tag covers the entire hierarchical string.

6.2 Analysis of false positives (noisy citations)

In Case 2, the text contains a document code “123/BC-UBND,” which resembles a legal citation but is effectively a generic reference number (noise).

• The Baseline Failure: The PhoBERT model, reacting to local features like capitalized letters and hyphens, incorrectly labels “BC-UBND” as a Legal Document (VBPL) with moderate confidence (0.77). This represents a False Positive, which introduces noise into the downstream knowledge graph.
• The Hybrid Solution: The PhoBERT-CRF model correctly assigns the O (Outside) tag. The CRF structure imposes a penalty on citation formats that do not strictly adhere to learned legal templates (e.g., Number/Year/Type-Agency). By optimizing the sequence probability over the entire sentence, the CRF determines that this token sequence does not fit the valid transition structure of a VBPL entity, effectively filtering out administrative noise.

These examples confirm that the performance gains reported in Section 5 (e.g., + 0.96% on VBPL) are driven by the CRF’s ability to repair structural defects—specifically boundary fragmentation and pattern non-conformity—that semantic embeddings alone cannot resolve.

7.1 Semantic and structural modeling in administrative NER

The superior performance of the PhoBERT-CRF framework (97.95%) validates our core engineering hypothesis: administrative texts require a dual-engine approach. While PhoBERT serves as a powerful Semantic Engine, capturing deep contextual cues to distinguish ambiguous entities (e.g., distinguishing an Object from an Agency based on sentence role), it fundamentally lacks the mechanism to enforce valid output sequences. By integrating the CRF as a Structural Engine, we mathematically enforced the “grammar” of administrative procedures. This synergy explains the statistically significant gain (+0.96%, p < 0.05) in the Legal Document (VBPL) class. In pure Transformer architectures, the complex alphanumeric patterns of legal citations often led to fragmentation errors. The CRF layer corrected these by learning high transition potentials () for intra-entity tokens, effectively “gluing” the fragmented components back together. For e-Government systems, where a single broken citation link can invalidate a workflow, this structural robustness is non-negotiable.

7.2 Language-specific models in specialized domains

PhoBERT’s outperformance of multilingual XLM-RoBERTa (97.51% vs. 95.43%) despite being half the size indicates that for specialized low-resource domains, language-specific optimization is more valuable than raw model capacity. This finding has implications for other low-resource languages requiring domain-specific NLP systems.

7.3 Dataset quality and annotation methodology

The strong Inter-Annotator Agreement (=0.85) and high performance across all models (minimum 95.43% baseline) indicate PAP_NER is a high-quality, well-defined benchmark. Source-stratified splitting ensures evaluation measures true generalization rather than template memorization. The convergence between Strict and Partial F1 scores suggests errors are primarily complete misses rather than boundary overlaps.

7.4 Broader implications and transferability

This work establishes a foundational dataset and benchmark for Vietnamese administrative NLP. The benchmarked PhoBERT-CRF configuration provides a reproducible baseline for future research, and public release under the Creative Commons BY 4.0 license promotes community-driven advancement of Vietnamese NLP.

Although PAP_NER and the trained models target Vietnamese administrative text, several outcomes of this study transfer beyond that setting. First, the corpus-construction pipeline—rule-based and LLM-assisted pre-annotation with local PII masking, MinHash near-duplicate removal, and source-stratified (group-aware) splitting—is language- and domain-agnostic; it offers a cost-effective recipe for building gold-standard NER resources in other low-resource languages and in other normative domains such as legal, clinical, or financial-compliance text, where templated phrasing and long structured spans pose similar annotation challenges. Second, our central empirical finding—that a dedicated monolingual encoder outperforms a larger multilingual one on a specialized low-resource domain—is consistent with results reported for other underrepresented languages [23,24] and offers a practical heuristic for practitioners selecting encoders under data-sovereignty and memory constraints. Third, the deployment-aware benchmarking protocol, which jointly weighs accuracy, latency, and footprint, is readily portable to other national e-Government pipelines facing comparable operational limits. Realizing this transfer in a new language or domain still requires fresh annotation and fine-tuning, since the structural patterns learned by the CRF and the lexical coverage of the encoder are specific to the source text (Section 7.5); the methodology, however, carries over directly.

7.5 Limitations and engineering trade-offs

While effective, the proposed framework entails specific trade-offs:

1. Linguistic Scope: Model is optimized for formal administrative text; performance may degrade on informal language where templates are absent.
2. Geographic Coverage: Four provinces provide regional diversity; comprehensive coverage of all 28 provinces would strengthen generalization claims.
3. Class Imbalance: Quantity (SL) class underrepresented (7.2%); transfer learning to SL-heavy domains may require additional fine-tuning.
4. BIO Tagging Limitation: The BIO (Begin, Inside, Outside) tagging scheme adopted in this work does not natively support nested or discontinuous entities. While legal citations may contain nested sub-components (e.g., a decree number nested within a full citation span), our scheme treats each entity as a single contiguous span. We selected BIO for compatibility with standard NER frameworks and prior Vietnamese NER benchmarks (VLSP 2016–2021). Future work should explore span-based or nested NER approaches [32] for extracting fine-grained sub-components within legal citation entities.