The framework of ScanNet

ScanNet is a new solution driven by both regulatory mechanisms and data characteristics for scRNA-seq data annotation (Fig 1a). ScanNet takes scRNA-seq data as input and reference TRN as basis regulatory structure to construct a heterogeneous graph that simultaneously models gene expression profiles and regulatory mechanisms. Specifically, the reference TRN is modeled as a heterogeneous graph by distinguishing TF and TG nodes and their regulatory interactions. The regulatory intensity is quantified as edge weights, with 1 assigned to activating regulation and to repressive regulation. The TRN further guides the selection of relevant genes from the scRNA-seq data, yielding TF and TG expression features for model input (S1 Text). With the prior constraint, ScanNet is able to capture cellular heterogeneity from both molecular mechanisms and transcriptional state through a dual-channel encoder with iterative graph convolution. The dual-channel encoder comprises a Regulation-level Encoder, which is responsible to extract causal regulatory dependencies at the molecular level through iHGCN (Fig 1b), and an Expression-level Encoder that learns global transcriptional states at the cellular level. By embedding regulatory dependencies, ScanNet generates biologically-informed cellular representations that not only enhance annotation accuracy but also improve generalization and applicability to diverse annotation tasks, like cross-platform annotation, novel cell type discovery, and transferability to single-cell ATAC-seq (scATAC-seq).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Framework of ScanNet.

(a) Overall architecture of ScanNet. ScanNet takes the expressions of TF and TG, and a reference TRN as input. ScanNet consists of a dual-channel encoder, a Regulation-level Encoder and an Expression-level Encoder, a Reconstruction Decoder and a Classifier. (b) Details of the iHGCN in the Regulation-level Encoder. The iHGCN captures regulatory dependencies by alternately updating TF and TG representations through heterogeneous message passing on the TRN, involving TF update and TG update steps.

https://doi.org/10.1371/journal.pcbi.1014602.g001

Dataset description and experimental design

We evaluate the performance of ScanNet against ten baseline methods across eight scRNA-seq datasets for the cell annotation task. Benchmark datasets include four human datasets, the paired multimodal PBMC dataset [28], Muraro dataset [29] derived from human pancreas, BMMC dataset [30] from bone marrow and Immune dataset from multi-tissue immune system [31], as well as four mouse datasets, BaronMouse dataset [32] from mouse pancreas, ChenMouse dataset [33] and AMB dataset [34] from mouse visual cortex, and TM dataset [35] from mouse musculus. For each benchmark dataset, cell-type labels are manually curated by domain experts and treated as the gold standard for a unified evaluation of automated annotation methods. Although datasets are derived from different species, the reference TRN [24] is unified which collects TF-TG regulations from multiple cell types and tissues, covering both human and mouse data. These datasets vary from tissues, sequencing platforms, species as well as the number of TFs, TGs, cells and cell types, as shown in Table 1. This diversity ensures a comprehensive assessment of performance across various biological contexts and technical challenges in cell classification. To mitigate the impact of extremely rare cell types on classification performance, scRNA-seq-based cell annotation experiments filter out cell types with fewer than 10 cells. The excluded rare cell types can subsequently serve as novel cell types to evaluate the model’s novel type detection capability.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Description of the eight scRNA-seq benchmark datasets.

https://doi.org/10.1371/journal.pcbi.1014602.t001

scRNA-seq data generated by different sequencing platforms exhibit systematic technical variations that pose challenges for cell annotation [36]. To evaluate the generalization of ScanNet across different sequencing platforms, another dataset (PBMCbench [37]) containing scRNA-seq data from seven sequencing platforms is introduced (S1 Table). Specifically, we design cross-validation experiments in which ScanNet is trained on one platform and tested on the rest six platforms, finally getting seven groups of experimental results. In addition, the batch-correction capability of ScanNet is evaluated by comparison with six baseline methods that support batch correction.

In a real-world scenario, the most models would assign the cells from previously unknown cell types to known types wrongly [38,39]. To evaluate ScanNet’s annotation capability in open-world scenario, we apply the pre-trained ScanNet model (trained on known cell types only) to new test datasets containing unknown cell types, also referred as novel types. Specifically, rare cell types with fewer than 10 cells, which are removed from the BaronMouse and AMB datasets during the initial training phase (Table 1), are reintroduced into the test datasets as new test datasets to simulate scenarios encountering novel types.

scATAC-seq provides insights into cellular heterogeneity by measuring chromatin accessibility, where the opening and closing of chromatin regions determine regulatory protein access and thereby influence gene expression initiation [40]. Given that scATAC-seq offers insights into deeper regulatory mechanisms through chromatin state profiling, we attempt to transfer the ScanNet framework to cell annotation on scATAC-seq data. Existing scATAC-seq classification tools [41] typically convert peak matrixes into gene score matrixes for downstream analysis, therefore we employ Signac [42] to transform chromatin accessibility signals into gene score matrixes that substitute for gene expression matrices in the ScanNet framework (S2 Text and S1 Fig). To evaluate the transferability of the ScanNet framework to scATAC-seq data, we use the scATAC-seq data from PBMC dataset [28]. Since this scATAC-seq data is paired with scRNA-seq data from the same PBMC dataset, we can compare the annotation performance of ScanNet across different modalities.

Given that benchmark datasets exhibit substantial class imbalance, we employ stratified sampling [43] to split each cell type into training and test sets at a 7:3 ratio, in which samples are drawn proportionally from each types to ensure that the overall sample accurately reflects the composition of the population. In addition, four evaluation metrics including Area Under the Precision-Recall Curve (AUPRC), F1, Precision and Recall, are employed to benchmark performance (S3 Text).

To extensively evaluate the performance of ScanNet in cell annotation, we benchmark it against ten baseline methods. These methods can be categorized into three groups: (i) reference mapping method (Seurat), (ii) deep learning-based methods (ACTINN [15], scAGN [44], scGCN [45], scGraphformer [17], scANVI [46]), and (iii) biological knowledge-informed methods (e.g., sigGCN [18], GEDFN [47], scGraph [20], and scRGCL [22]). Deep learning models are solely based on transcriptional expression profiles, while biological knowledge informed methods incorporate additional gene interaction networks as prior knowledge, ensuring that biological knowledge is explicitly embedded into cellular features. The protein-protein interaction (PPI) networks used by sigGCN, scGraph, and scRGCL are derived from the STRING database [19], whereas GEDFN uses a network obtained from the HINT database [48]. In contrast, ScanNet leverages transcriptional regulatory relationships between TFs and TGs encoded in a reference TRN as prior structural information. Among all compared methods, only scGCN, scAGN, Seurat, and scANVI adopt a semi-supervised learning paradigm, in which cell-type labels are transferred from a reference dataset to a query dataset for cell annotation. The remaining methods are fully supervised. The other benchmark methods are supervised learning models. All baseline models are implemented with their default hyperparameters.

ScanNet outperforms baselines in cell type annotation

We benchmark ScanNet against ten baseline methods on the eight benchmark datasets spanning different species, tissues and sizes. ScanNet performs better than all baselines in terms of F1 score and Precision (Fig 2a and S2 Table). ScanNet achieves an F1 score of 0.9762 and Precision of 0.9776 in BaronMouse dataset, outperforming the second-best baseline by 1.40% and 2.06% respectively. In PBMC dataset, the precision of ScanNet is 0.9742, while those for the best three baselines (ACTINN, Seurat and scGCN) are 0.8959, 0.8899 and 0.8891. For AUPRC and Recall, ScanNet outperforms all baselines on all datasets. In AMB dataset, ScanNet is superior to the top two baseline methods, scGraph and ACTINN, with 1.83% and 2.41% improvement in terms of AUPRC. Both the ChenMouse and BMMC datasets are considered challenging due to the high similarity among cell types. Although ScanNet achieves a slightly lower AUPRC than scGraphformer in the ChenMouse dataset, it outperforms scGraphformer in terms of F1 score and Precision. Moreover, ScanNet achieves the best performance across all evaluation metrics on the BMMC dataset. These results indicate that ScanNet is also well suited for complex datasets.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Performance evaluation of ScanNet on cell type annotation.

(a) Comparison with ten methods across eight benchmark datasets on four evaluation metrics: AUPRC, F1 score, Precision, and Recall. (b) Critical difference diagrams for the two metrics: AUPRC and Precision. Methods connected by red are not significantly different in performance according to the Nemenyi test.

https://doi.org/10.1371/journal.pcbi.1014602.g002

In order to quantify the comparison with statistical rigor, we employed the Nemenyi test [49] and visualized the results using Critical Difference (CD) diagrams [50] at confidence level = 0.95, following the procedure established by Zhang et al. [40]. The CD diagrams are designed to determine whether one algorithm achieves significantly better performance than others through statistical testing. In the CD diagram, the horizontal axis denotes the mean rank of each method across all datasets, and red lines connect methods whose differences are not statistically significant. ScanNet ranks first across all classification metrics (Figs 2b and S2). Although its AUPRC performance is comparable to that of scANVI, ScanNet achieves a markedly better average rank (1.28 vs. 3.85). Moreover, ScanNet significantly outperforms all competing methods, including scANVI, in term of Precision. Finally, although ScanNet does not achieve the highest AUPRC in the ChenMouse dataset, it exhibits statistically significant superiority over scGraphformer in overall AUPRC across datasets.

scRNA-seq datasets usually exhibit obvious imbalance in types (S3 Fig). The cell numbers in different cell types are illustrated by the bar heights spanning from 894 beta cells to 10 B cells in BaronMouse dataset (Fig 3). This uneven distribution of sample numbers across cell types creates a typical scenario where model tends to learn the feature distribution of types with large sample size (e.g., beta cells) while neglecting the learning of minority class types (e.g., gamma cells). The Sankey diagrams (Fig 3) visualizes the correspondence between predicted and true cell types. We illustrate the results on BaronMouse dataset, which has the most pronounced cell type imbalance, and compare ScanNet with scGraphformer, the strongest baseline method. Nodes on the left represent ground-truth annotations, while nodes on the right denote model predictions. The width of each connecting link is proportional to the number of cells in each true-predicted type pair, which highlights that, for the minority gamma cell population, ScanNet demonstrates more direct correspondences with thicker Sankey flows between true and predicted labels, yielding a recall of 66.67%, whereas scGraphformer attains only 16.67%. Moreover, ScanNet attains a precision of 94.38% on alpha cells, surpassing scGraphformer’s 77.03%. This result reveals that ScanNet achieves superior accuracy on minority cell types, demonstrating its enhanced ability to mitigate recognition bias toward majority classes while maintaining robust overall predictive performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison of cell type annotation performance between ScanNet and scGraphformer on the BaronMouse dataset.

Sankey diagrams illustrate the correspondence between ground-truth (left) and predicted (right) cell type annotations on the test set. Each node represents a cell type, and the width of each connecting link reflects the number of cells assigned to the corresponding true-predicted pair. ScanNet demonstrates higher annotation accuracy with fewer cross-type misclassifications compared to scGraphformer.

https://doi.org/10.1371/journal.pcbi.1014602.g003

In addition to benchmark datasets of varying complexity and class imbalance, we further evaluate ScanNet on a large-scale human immune atlas dataset containing 329,762 cells to assess its performance on large-scale atlases. Because large-scale single-cell datasets introduce significant computational challenges, particularly for methods that rely on constructing global cell-cell graphs, as the number of cells increases, the cost of nearest-neighbor search, graph construction, and message passing grows rapidly in both time and memory, making some methods difficult to execute reliably under our hardware constraints. Therefore, for the large-scale dataset, we report the results of representative baseline methods that can be successfully executed under our computational environment, including Seurat, scGraph, sigGCN, scANVI, scGraphformer, GEDFN, and scRGCL. ScanNet maintains strong annotation performance on this large-scale dataset, achieving competitive or superior results compared with representative baseline methods across all evaluation metrics (Fig 2), indicating that ScanNet not only achieves strong performance across datasets of varying biological complexity, but also scales effectively to large single-cell atlases while maintaining robust annotation accuracy. Importantly, the performance advantage of ScanNet remains consistent across scales, suggesting that the incorporation of transcriptional regulatory priors provides stable benefits that are not diminished by increasing data size.

All these results demonstrate ScanNet achieves consistent classification advantages over tissues, species, and sample distribution. By systematically integrating TRN with gene expression data, ScanNet captures both regulatory mechanisms and transcriptional outcomes, providing mechanistic insights into cellular heterogeneity.

ScanNet exhibits robust performance across different sequencing platforms

To evaluate ScanNet’s robustness to technical noise led by different sequencing platforms, we conduct experiments on PBMCbench dataset [37], which contains scRNA-seq data from the same donor’s peripheral blood mononuclear cells sequenced with seven different platforms.

ScanNet is evaluated by leave-six-out validation, that is trained on data from one sequencing platform and tested on the remaining six, yielding a total of seven results. Fig 4a presents AUPRC pairwise comparisons between ScanNet and six baseline models. Most scatter points lie above the diagonal, demonstrating that ScanNet’s AUPRC outperforms that of most methods. In CD diagrams, ScanNet achieves the top overall ranking across all classification metrics, and exhibiting batch robustness comparable to that of the state-of-the-art method scANVI (Fig 4b).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Evaluation of the generalization ability and robustness of ScanNet across different sequencing platforms.

(a) Scatter plot comparing AUPRC scores of ScanNet with seven baselines across seven experimental settings. Each point corresponds to one comparison, with the x-axis representing baseline AUPRC and the y-axis representing ScanNet AUPRC. The dashed diagonal line (x = y) denotes performance parity. (b) Critical difference diagrams for four evaluation metrics (AUPRC, F1 score, Precision, and Recall). (c) t-SNE visualizations on the test set, colored by cell type (upper panels) and sequencing protocol (lower panels). The left panels show PCA-based t-SNE embeddings derived from the normalized and scaled expression profiles of selected TFs and TGs, while the right panels are based on embeddings learned by ScanNet, showing improved separation of cell types and reduced batch effects across protocols.

https://doi.org/10.1371/journal.pcbi.1014602.g004

Additionally, t-SNE visualizations compare cellular representations derived from the PCA-based embeddings of normalized expressions (Fig 4c, left panel) and ScanNet-generated embeddings (Fig 4c, right panel), with upper plots colored by cell type and lower plots by batch. In the expression-based visualization (left panel), different cell types show considerable overlap and mixing, making it difficult to distinguish clear cell-type boundaries. Moreover, when colored by sequencing platform (lower left), cells from the same platform tend to cluster together, indicating strong platform-driven batch effects. In contrast, the ScanNet embedding visualization (right panel) demonstrates markedly different patterns. When colored by cell type (upper right), ScanNet forms compact and well-separated clusters for each cell type, with clear boundaries between different cellular populations. Importantly, when colored by sequencing platform (lower right), ScanNet embeddings show reduced platform-driven clustering compared with the expression space, although some residual platform-associated structure remains in specific cell populations, such as cytotoxic T cells. The residual platform-specific structure is present to varying extents across methods, and that no method completely removes platform effects while simultaneously preserving clear cell-type separation (S4 Fig). Compared with other methods, ScanNet provides a more favorable balance between cell-type discrimination and cross-platform mixing. S3 Table summarizes the cross-platform classification metrics across seven experimental combinations and S4 Table records batch correction metrics of ScanNet and baselines.

By incorporating the underlying gene regulatory mechanisms and directed relationship from TFs to TGs, ScanNet captures the intrinsic regulatory processes that drive cellular identity, thereby mitigating platform-specific biases inherent to divergent sequencing technologies and exhibiting robust classification performance.

ScanNet can identify novel cell types

ScanNet leverages prediction uncertainty quantified by entropy to distinguish novel cell types adapting to real-world scenarios. ScanNet exhibits higher uncertainty (higher entropy) when encountering previously unseen cell types compared to the confident predictions (lower entropy) for known cell types.

In the experiment, cell types with fewer than 10 cells are removed from the training set and applied as entirely unknown novel cell types during testing. Unknown cell types (first row in each matrix) are predominantly classified as “Novel” rather than being assigned to known cell types (Fig 5a), indicating ScanNet demonstrates robust performance on both AMB and BaronMouse datasets, accurately predicting known cell types while effectively identifying novel cell types not encountered during training. Quantitatively, for the AMB dataset, ScanNet achieves 71.43% recall and 71.43% precision for novel cell detection. For the BaronMouse dataset, while precision is 66.67%, recall reaches an impressive 95.24%, highlighting ScanNet’s high sensitivity in identifying novel cells. In addition, we evaluate entropy-based novel cell-type identification for more cells, further demonstrating ScanNet can identify novel types with large number of cells (S4 Text and S5 Fig).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Identification of novel cell types by ScanNet.

(a) Confusion matrices of ScanNet on the test set of AMB and BaronMouse datasets, showing that ScanNet can distinguish novel types from known cell types. (b) Entropy-based density distributions. The higher entropy values associated with novel cells demonstrate that prediction uncertainty can effectively separate unknown from known cell types.

https://doi.org/10.1371/journal.pcbi.1014602.g005

The entropy-based uncertainty quantification is effective and shows distinct distributions of entropy values in known and novel cell types (Fig 5b). In the AMB dataset, the mean and variance of known type distribution are 0.0168 and 0.0069, while these of novel type are 0.911 and 0.2155, implying that entropy of known types exhibit high prediction confidence. In contrast, novel cell types display broader distributions with higher entropy values, reflecting greater prediction uncertainty. This clear discrepancy validates entropy as an effective metric for distinguishing novel from known cell types based on model confidence.

These results demonstrate that ScanNet successfully identifies novel cell types through entropy-based uncertainty quantification, addressing a critical limitation of existing supervised models that misclassify unknown types into predefined categories. With high detection sensitivity, ScanNet provides practical functionality for real application scenarios where new cell populations continuously emerge, making it valuable for both exploratory research and clinical applications.

While ScanNet can identify cells that deviate from the reference distribution via entropy as “Novel”, such novel cells require further biological interpretation in practice. In real single-cell analysis pipelines, high-resolution annotation is typically achieved by combining automated predictions with marker-based validation. We therefore evaluate whether ScanNet can be integrated with marker-based tools to form a hybrid pipeline for improving the annotation of novel cells. Using scCATCH [51], a widely used marker-based tool, we adopt a stratified strategy in which ScanNet first partitions cells according to prediction confidence. High-confidence cells retain the labels predicted by ScanNet, whereas low-confidence cells are further annotated with scCATCH (S6 Fig). Importantly, for these low-confidence cells, scCATCH is applied to clusters derived from ScanNet embeddings rather than to clusters derived from the original expression profiles, corresponding results are provided in S7 Fig. This hybrid pipeline enhances the resolution of novel cell types. While ScanNet identifies such cells as “Novel”, scCATCH refines them into biologically meaningful candidate lineages, with reduced lineage confusion compared with standalone scCATCH based on the original expression profiles (S7 Fig). These results demonstrate that ScanNet can serve as an effective front-end module in practical annotation pipelines.

ScanNet performs well with transferred scATAC-seq data

The framework of ScanNet can be transferred to other omics data, scATAC-seq data. To evaluate annotation performance on scATAC-seq data, annotation performance on scATAC-seq data is compared with Cellcano [41], a well-known cell classification tool specifically designed for scATAC-seq, implemented here with default hyperparameters. ScanNet is applied to paired scRNA-seq data and scATAC-seq data separately. ScanNet performs superior on scATAC-seq data than Cellcano across all evaluation metrics demonstrating the flexible extension ability of ScanNet from scRNA-Seq to scATAC-Seq data (Fig 6a).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Annotation performance of ScanNet on scATAC-seq data.

(a) Bar chart comparing annotation performance on the PBMC dataset. (b) Accuracy comparison between ScanNet and Cellcano across all cell types. The upper scatter plot depicts accuracy differences by cell type, where point size indicates the magnitude of the difference and color represents the performance direction (red: ScanNet superior; blue: Cellcano superior). The lower bar chart summarizes the number of cell types in the PBMC dataset. (c) The ROC and PRC of two models on the CD16 Mono type.

https://doi.org/10.1371/journal.pcbi.1014602.g006

We also evaluate ScanNet’s performance on sample size imbalance across types for scATAC-Seq data. Fig 6b presents a detailed accuracy comparison between ScanNet and Cellcano across all cell types in scATAC-seq data. The upper scatter plot visualizes accuracy differences by cell type where point size reflects the magnitude of differences and color denotes whose performance is better. Specifically, red represents ScanNet superior, blue stands Cellcano superior. The lower bar chart displays the sample size (number of cells) for each cell type shown in the scatter plot above, with both subplots sharing a common horizontal axis for cell-type alignment. ScanNet outperforms Cellcano in 12 out of 19 cell types, with particularly substantial advantages in minority cell types such as Naive B, gdT, and Treg cells, where ScanNet’s performance exceeds Cellcano by over 40–70%. In contrast, Cellcano shows marginal advantages only in 6 cell types, where the performance differences are relatively small (typically < 15%). However, Cellcano faces challenges when classifying cell types with limited sample sizes, often misclassifying them, as seen with CD16 Mono and Treg cells. In contrast, ScanNet demonstrates exceptional robustness to severe class imbalance, maintaining high performance across both abundant cell types and rare cell types (e.g., HSPC, Plasma with < 20 cells). ScanNet is also evaluated on scATAC-seq without paired scRNA-seq data and compared with Cellcano (S5 Table). ScanNet consistently outperforms Cellcano, indicating that ScanNet can generalize well to scATAC-seq datasets even when no matched scRNA-seq data are available, demonstrating its strong transferability and generalization across modalities.

To further exhibits the efficacy of ScanNet on minority cell types, we plot the receiver operating characteristic (ROC) and precision-recall curves (PRC) for the CD16 Mono population (Fig 6c). While ScanNet and Cellcano exhibit comparable AUROC performance (0.98 vs 0.99), ScanNet achieves a notable higher AUPRC (0.89 vs 0.84), highlighting its superior precision-recall trade-off in minority class annotation.

Although ScanNet’s performance on scATAC-seq data is slightly lower than its scRNA-seq performance, it still substantially outperforms the scATAC-seq-specific method Cellcano, demonstrating ScanNet’s effective cross-modal transferability and adaptability to diverse single-cell sequencing technologies.

The dual-channel encoder

Transcriptional regulation orchestrates gene expression [26], establishing a causal link form TF to TG. With this prior knowledge, ScanNet deploys a dual-channel encoder, consists of a Regulation-level Encoder and an Expression-level Encoder, to dissect cellular heterogeneity from regulatory perspective and expression perspective respectively. The Regulation-level Encoder employs iHGCN to alternately update TF and TG representations through heterogeneous message passing on the TRN, combining self-projection, cross-projection, and attention-based fusion and learns cellular regulatory states, encoded into Regulation-level Embeddings. The Expression-level Encoder utilizes fully connected neural network to extract global cellular transcriptional states and regulatory outcome features inside cells from scRNA-seq data into Expression-level Embeddings, while neglecting inner regulation interactions within cells. The Regulation-level Encoder solely capture local features, due to learning filters’ localization property of GCN. Therefore, through the dual-channel encoder, ScanNet constructs a unified and biologically informed cell representation that captures not only the overall transcriptional state but also the underlying regulatory mechanisms. Comprehensive definitions of all variables, matrices, and operators used from data preprocessing through model training are provided in S11 Table.

Iterative heterogeneous fusing encoding regulatory process in Regulation-level Encoder

iHGCN-based Regulation-level Encoder capture the regulatory dependencies by iteratively learning embeddings of TFs and TGs, which aggregates information from TGs to update TF representations, and vice versa. This bidirectional message passing enables Regulation-level Encoder to facilitate precise localization of cell states. The Regulation-level Encoder employs two iHGCN layers to integrate TF-TG regulatory relationships encoded in the TRN into the embeddings of TF and TG nodes, followed by a pooling layer to derive the Regulation-level Embedding for each cell. Taking the “Transcriptional Factor Update Step” in Fig 1b as an example, a detailed explanation of the iHGCN layer in Regulation-level Encoder is provided as follows. The iHGCN layer consists of two steps: heterogeneous feature extraction from TF-TG regulatory relationships, and attention-driven TF-TG representation fusion for cellular embedding generation.

Heterogeneous feature extraction from TF-TG regulatory relationships

The iHGCN applies heterogeneous convolution to learn TF-TG regulatory associations from the TRN. In order to eliminate the influence of both TF and TG node degree on information transmission, row normalization is employed to the adjacency matrix, ensuring that the information from each node is appropriately represented during the aggregation.

(1)

(2)

(3)

(4)

where is the transpose of , denotes absolute values, and represents the row-wise sum. is the diagonal degree matrix of , where each diagonal entry records the number of TGs regulated by the corresponding TF. Likewise, quantifies the number of TFs that regulate each TG.

The functions and are designed to extract intrinsic features from TF nodes and regulatory information from TG nodes, respectively:

(5)

where focuses on capturing the inherent characteristics of TFs, while uses graph convolution to integrate regulatory interaction information from neighboring TG nodes to infer TFs’ potential features. Specifically, and are the input feature of TF and TG of the layer (the output of layer), is the weight matrix of aggregating information from TG, is the projection matrix to project into the same eigenspace as .

However, the embedding based on TRN does not retain the feature information of the TF itself, potentially limiting the completeness of the feature representation. To overcome this limitation, iHGCN introduces an attention mechanism to integrate the embedding vector of TF itself () and the inferred embedding vector of TF from TG (), thereby generating a more comprehensive and informative embedding for TF.

Attention-driven TF-TG representation fusion and cellular embedding generation: As depicted in Fig 1b, the attention mechanism maps a set of queries and a set of key-value pairs to an output. In practice, we pack together queries, keys and values into three matrixes , and respectively. Then attention mechanism can be formulated as: , where is the attention function such as dot-product [52] or neural network [53]. Here, embedding vectors and are both as values matrix . We define a weight matrix to map them into keys matrix , and define a weight matrix to map into the query matrix , where is the hidden layer dimension. Formally,

(6)

(7)

(8)

Note that mapping as the query is the key to achieve personalized importance estimation for each TF node. The attention function is implemented as follows:

(9)

(10)

where denotes the row-wise concatenation operation, is the parameter vector and ELU is the activation function. The element of denotes the self-contribution weight of the TF, whereas the element of represents the regulation aggregated importance vector of its neighbors (TGs) on the TF. Then, the normalized attention coefficients are computed with the softmax function to update the representation of TF nodes:

(11)

(12)

where is the iHGCN-updated TF embeddings, represents the normalized self-information attention weight for TFs, represents the normalized weight of regulatory information from TGs, and is hadamard operator. The iHGCN-updatpled TG embeddings are obtained in an analogous manner.

The Regulation-level Encoder employs two iHGCN layers to learn molecular-level embeddings for TFs and TGs , and then aggregates them via a function to produce cells’ Regulation-level Embeddings:

(13)

where refers to the max-pooling layer, and represents the Regulation-level Embedding of cells, in which is the output dimension of the Regulation-level Encoder.

Gene expression encoding in Expression-level Encoder

The Expression Encoder is designed to capture the holistic cellular state by modeling transcriptional profiles, thereby generating comprehensive global representations. This global perspective compensates for the limitation inherent to the Regulation Encoder, which focuses on gene-level regulatory relationship and consequently lacks a unified view of the entire cell system. Specifically, the Expression-level Encoder employs a simple MLP to capture the overall cellular state:

(14)

where is the concatenated expression of TFs and TGs, representing gene expression within cells, is the Expression-level Embedding vector where is the output dimension of Expression-level Encoder same as that of Regulation-level Encoder, and is the activation function.

The decoder to reconstruct gene expression and the classifier to annotate cell type

The Reconstruction Decoder is designed to reconstruct downstream cell expressions from upstream Regulation-level Embeddings, ensuring that Regulation-level Embeddings accurately capture the regulatory state of cells. Thereby, the Reconstruction Decoder is designed to rebuild the expression of TFs and TGs from :

(15)

where is the weight matrix, and is the bias vector. As a result, has the same dimension as the concatenated input , which is the number of all TF nodes and TG nodes in TRN.

It is worth noting that the reconstruction objective is applied only to the gene expression matrix, rather than the TRN. In our framework, the TRN is incorporated as a shared prior across all cells to guide information propagation between transcription factors and target genes. Since this prior is identical for all cells, introducing a reconstruction loss on the TRN would implicitly enforce the model to recover the same graph structure across samples. However, the primary objective of cell type annotation is to capture cell-specific transcriptional variations. Therefore, constraining the model to reconstruct a global and invariant network structure may reduce its ability to learn discriminative representations. For this reason, we restrict the reconstruction objective to the gene expression matrix, which directly reflects cell-level heterogeneity, while the TRN is used as structural prior knowledge rather than a reconstruction target.

Since Regulation-level Embeddings and Expression-level Embeddings characterize cell features from regulatory mechanisms and expression outcomes, the Classifier is trained on the concatenation of these two embeddings to achieve accurate cell annotation:

(16)

(17)

where is the weight matrix in which is the number of cell types, is the bias vector, and is the predicted probabilities matrix.

The optimization of loss function

The loss function comprises three key components: the reconstruction loss of expression data , the classification loss of cell typing and regularization, as given in Eq (18)–(20) respectively,

(18)

(19)

(20)

is the reconstruction loss between normalized and reconstructed expression data to minimize the loss of regulatory information that occurs when converting embeddings from gene to cell within Regulation-level Encoder; is the classification loss of ground truth labels and predicted probabilities to enhance classification performance; and is regularization applied to all parameters ensuring a more stable training process and improved robustness. By minimizing with the SGD optimizer, ScanNet preserves gene regulatory information to the greatest extent while simultaneously enhancing cell type prediction through the inferred cell potential embedding.

Recognize novel types by prediction uncertainty

In real application scenarios, the emergence of previous unknown cell types, also referred as novel types, is frequently encountered. In other words, real cell data not only contain known types trained during training process but may also include novel cell types. Therefore, prediction uncertainty is introduced as a quantitative metric of output confidence, while keeping the model architecture unchanged. Prediction uncertainty is quantified via the entropy of the predicted probability distribution derived from classifier outputs, where higher entropy indicates a more uniform distribution, reflecting greater uncertainty in predictions, and lower entropy implies greater confidence.

To emulate real application scenarios, the experimental design is as follows:

(21)

(22)

where represents cells of novel types, not encountered in training, and is integrated into the test set to create a new test set containing known and novel cell types, then is fed into the pre-trained ScanNet to obtain the probability distribution over predefined classes. We hypothesize that prediction uncertainty, quantified via entropy, will be higher for cells of novel types compared to known types. Hence, prediction uncertainty is calculated for each cell and cells with high uncertainty are identified as novel types. Mathematically, given the probability distribution from the Classifier, the prediction uncertainty is calculated as Eq (23):

(23)

where denotes the predicted probability that the cell belongs to the class in , represents the number of predefined classes during training and is the prediction uncertainty of cell.

Then, the decision is made by comparing prediction uncertainty with the threshold :

(24)

When the prediction uncertainty exceeds , the probability distribution is more dispersed, reflecting less confidence on the prediction, and such cells are assigned as novel . Conversely, if the prediction uncertainty is less than , ScanNet show high confidence on the prediction and the cell is assigned to the class with the highest probability. In this manuscript, the default value of threshold is set to 0.75. indicates that the cell belongs to novel types not encountered during prior training.

Code availability

The code used for model training and evaluation is available on GitHub (https://github.com/Ying-Lab/ScanNet).