Workflow of RareCapsNet

Fig 1 describes the workflow of the whole analysis.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the RareCapsNet workflow.

(A) Preprocessing of the scRNA-seq data. The input expression matrix contains N cells (rows) and K genes/features (columns). After filtering, normalization, and log-transformation, each individual cell is represented by a K-dimensional expression vector. (B) Capsule-network architecture. During training or inference, the input to the network is a single-cell expression vector. This vector is first passed through a fully connected layer, then through the primary capsule layer, and finally to the type capsule layer by dynamic routing. The number of type capsules equals the number of annotated cell types. (C) Interpretation of capsule associations using coupling coefficients between primary capsules and type capsules. Stronger coupling indicates that a given primary capsule contributes more strongly to the prediction of a particular cell type. (D) Gene-level attribution. For a selected primary capsule–type capsule pair, gene-specific coupling coefficients are computed by masked-input analysis to identify marker genes associated with the corresponding cell type.

https://doi.org/10.1371/journal.pcbi.1013962.g001

A. Preprocessing the data See -A of Fig 1. Raw scRNA-seq data, obtained from public data sources are preprocessed and normalized using a transformation method (Linnorm) [16]. We keep cells having more than a thousand genes expression values (non zero values) and choose genes which having the minimum read count greater than 5 in at least 10% of the cells. log₂ normalization is performed on the transformed matrix by adding one as a pseudo count.

B. Model training and optimization. Resulting data is divided into training, validation and test data sets. For exploration and adjustment of hyperparameters and optimizing the underlying capsule network architecture, we perform cross-validation runs. The resulting model consists of an ordinary fully connected layer, followed by a primary capsule and an output capsule layer (the connection between the latter two of which encompasses a dynamic routing procedure). Output capsules refer to the individual cell types that one can predict. The details architecture of the proposed model is shown in a separate box (panel-A, Fig 1). The input parameters, e.g., kernel size (k1) in first fully connected (FC) layer, kernel size (k2) of primary capsule layer, number of primary capsules (P) are fixed for all datasets, while number of cells (N), number of class/number of type capsules (k) depend on the datasets used here.

C. Interpreting primary capsules. See panel-B of Fig 1. The key points of motivation of our method is the interpretability of essential building blocks of the underlying capsule model. We find out the primary capsules of a trained model that were in strong association with the corresponding class (here cell type). First we train the model with available training data and retain the model that gives best performance. Test samples are given as input to the trained network and coupling coefficients() between primary capsules and type (class) capsules are observed. Panel-B of Fig shows a heatmap demonstrating the association between two layers of capsules (here 32 primary in first layer and 2 type capsule in output layer) as an example. The output of this step is the index of activated primary capsules specific to the classes.

D. Associating features (genes) with primary capsules. See panel-C and -D of Fig 1. This step associates features (genes) with the activated primary capsules. We proposed an algorithm for identifying top genes that can activates a particular primary capsules. Given a primary capsule, which is known to be activated for a particular class (cell type), we identify features (genes) that are responsible for activation of the same primary capsule. The algorithm starts with masking all features except one and storing the average coupling coefficient by given the data as input to the trained network. The result of this step consist of a set of genes for a particular primary capsule that are known to be associated with a class capsule.

E. Supervised setting and candidate novel-cell detection RareCapsNet is formulated primarily as a supervised rare-cell identification framework. During training, the model requires cell-type labels to learn associations between gene-expression patterns, primary capsules, and type capsules. Therefore, the method is most directly applicable to detecting rare or poorly represented cell types that are present in the training annotations or in a related reference dataset.

For cells from a new dataset, RareCapsNet assigns each cell to the type capsule with the largest activation. However, cells that do not match any known type capsule may show low maximum capsule activation, weak prediction confidence, or diffuse coupling across multiple type capsules. Such cells can be flagged as candidate outliers or putative novel populations for further analysis. We emphasize that these candidates should not be interpreted as definitive novel cell types without additional validation, such as clustering of the flagged cells, inspection of marker genes, comparison with public marker databases, and functional enrichment analysis. Thus, the current RareCapsNet framework supports supervised rare-cell detection and can highlight candidate outlier populations, while fully unsupervised or open-set novel cell-type discovery remains an important direction for future work.

The workflow in Fig 1 provides a conceptual overview of RareCapsNet; the detailed mathematical formulation of the architecture, dynamic routing procedure, and training objective is provided in the Method section under “Architecture and training objective of RareCapsNet.

Explainability of capsules in rare cell detection

A simulation study is designed to show the explainibility of primary capsules for capturing the characteristics of rare cellular identities. Here we have shown one (or more) primary capsules are activated for a particular cell of rare type. For this, we have utilized splatter [17], a widely used tools for simulation of realistic scRNA-seq data.

Data generation.

Here we generate the data with four experimental setups:

S1: generated 1000 cells in two groups, with sample a ratio of 90:10, 95:5, 99:1 and 99.5:0.05 keeping low dropout rate (), proportions of differentially expressed (DE) genes as 40%, over 2000 genes

S2: generated two groups of cells, consisting 50% of the total (1000) cells in each group, over 2000 genes at a high dropout rate (), proportions of differentially expressed (DE) genes as 20%

S3: generated 1000 cells in three groups, with sample ratio of 80:10:10, 90:5:5, 98:1:1 and 99:0.05:0.05 keeping low dropout rate (), proportions of differentially expressed (DE) genes as 40%, over 2000 genes

S4: generated four equal-sized groups of 1000 cells over 2000 genes at a high dropout rate , and proportion of DE genes as 20%.

S5: Thirteen cell types were generated to simulate a complex multi-class setting with multiple rare populations. Four cell types were defined as rare, with proportions 0.5%, 1%, 2%, and 5%, while the remaining nine cell types shared the remaining cells approximately equally. The target dropout rate was approximately 0.2, and 40% of genes were differentially expressed. This setting was repeated 10 times with different random seeds.

The details of the simulation settings are shown in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Synthetic datasets grouped by simulation settings (S1–S5). Each dataset varies by class ratio, dropout rate, and proportion of differentially expressed (DE) genes.

https://doi.org/10.1371/journal.pcbi.1013962.t001

Performance of RareCapsNet on simulated data

RareCapsNet can distinguish minor subpopulation of cells in all type of simulation settings (S1 to S5). simulated single cell data obtained from each setting is processed with RareCapsNet with a train-test split of 80:20. We store the coupling coefficient () values for the test set applied on the fully trained model. To get a stable setup, we have applied the trained model multiple times (100 times here) on the test samples and obtained the average s. The main aim of this study is to find out the association of primary capsules with minor subpopulations of cells present in the data. For each simulation settings one (or more than one) primary capsules are noticed to be activated (see Fig 2) that explain the characteristics of cells of a particular type. For S1 and S2 setups the simulated data consist of two groups/classes, whereas for S3 and S4 the generated data has three groups/classes. S1 and S3 contain four simulated data with varying sample ratio, whereas for S2 and S4 setups sample ratio are fixed. The aim is to show the performance of RareCapsNet in diverse simulated data with fixed and varying number of samples. RareCapsNet produces one (or more than one) activated primary capsules for the minor subpopulations in each simulation settings (see Fig 2 for the results of simulation data S1 and S2). The activated primary capsules may be treated as marker of the generated cells of specific type. For example, in setting S1, the generated data with sample ratio 95:5, primary capsule 9, 19 and 26 are activated for the minor (5%) population, while primary capsule 10 is activated for the major (96%) population. For the other setups (e.g., S2, S3 and S4) the similar situations can be observed in the activation pattern of the primary capsules. The results for dataset S3, S4 and S5 are given in S1 Text. It may be happened that one activated primary capsule representing the samples of two or more types (e.g., The results for simulation settings are provided in Table D and Table E of S1 Text. In the S5 setting, primary capsule 32 showed in Table 3 high coupling with two cell types, indicating that one primary capsule may occasionally capture shared transcriptomic features across more than one rare population.).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Coupling strength visualization under simulation settings S1 and S2.

The panel A shows simulation setting S1, where two cell populations were generated under increasing class imbalance with sample ratios 90:10, 95:5, 99:1, and 99.5:0.5, using 40% differentially expressed genes and a dropout rate of 0.2. The left-side t-SNE plots show that the rare population becomes progressively smaller as the imbalance increases. The corresponding heatmaps on the right show coupling coefficients between 2 output capsules and 32 primary capsules for each sample-ratio setting. Darker red regions indicate stronger coupling, suggesting that specific primary capsules become strongly associated with the rare cell population. The panel B shows simulation setting S2, where two balanced cell populations were generated with sample ratio 50:50, 20% differentially expressed genes, and a higher dropout rate of 0.5. The t-SNE plot shows separation between the two simulated groups, while the heatmap illustrates the learned coupling pattern between the output capsules and primary capsules under high-dropout conditions. Overall, the Figure demonstrates that RareCapsNet learns class-specific capsule-routing patterns under both extreme imbalance and high-dropout simulation settings.

https://doi.org/10.1371/journal.pcbi.1013962.g002

We compared RareCapsNet with four established rare-cell detection methods: GiniClust, FiRE, RaceID, and CELCIUS, across five simulation settings S1–S5. The Tables D and E of S1 Text provide an extended simulation-based comparison of RareCapsNet with GiniClust, FiRE, RaceID, and CELCIUS using accuracy, sensitivity/recall, and ROC-AUC. RareCapsNet achieved the best overall performance in both two-class and multi-class rare-cell simulation settings. The gain in sensitivity/recall indicates that RareCapsNet is particularly effective in detecting minority cell populations under severe class imbalance. These results support the robustness of the proposed capsule-routing framework for rare-cell identification. As shown in, Table 2, RareCapsNet achieved strong rare-cell detection performance across the simulated datasets S1–S5. The simulation settings included both binary and multi-class rare-cell scenarios with varying degrees of class imbalance, dropout, and differential gene-expression proportions. RareCapsNet obtained the highest F1-score in most of the simulated settings, indicating its ability to recover minority cell populations under different rare-cell configurations. The improvement was particularly evident in highly imbalanced settings, such as S1.3, S1.4, S3.3, and S3.4, where rare populations were represented by very small numbers of cells. This suggests that the capsule-routing mechanism can preserve rare-cell-specific transcriptomic signals even when the minority class is poorly represented. Although RareCapsNet showed the best overall trend, some baseline methods performed comparably in selected settings. For example, RaceID showed slightly higher F1-score in S1.1, while FiRE performed marginally better in S3.1 and S3.2. Therefore, the results should not be interpreted as uniform superiority in every individual simulation condition, but rather as evidence that RareCapsNet provides robust and competitive rare-cell detection performance across a broad range of simulated rare-cell scenarios. Since F1-score alone may not fully describe model behavior under class imbalance, we further evaluated additional performance measures, including accuracy, sensitivity/recall, and ROC-AUC. These extended results are provided in Tables D and Table E of S1 Text. Table D of S1 Text reports the additional metrics for two-class simulated rare-cell settings, whereas Table E of S1 Text reports the corresponding metrics for multi-class and complex rare-cell simulation settings. Together, these supplementary analyses provide a more complete assessment of the classification performance and rare-cell sensitivity of RareCapsNet and the competing methods.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. F1-score performance of competing methods on synthetic datasets.

https://doi.org/10.1371/journal.pcbi.1013962.t002

Ablation analysis of capsule-network parameters

To assess the sensitivity of RareCapsNet to key architectural choices, we performed an ablation study by varying the number of primary capsules and the number of routing iterations. The experiments were conducted using the S1.2 simulation setting, which contains two cell populations with a 95:5 class ratio and therefore provides a controlled but challenging setting for rare-cell identification.

As shown in Table 3, increasing the number of primary capsules from 8 to 32 improves rare-cell detection performance, suggesting that additional capsules increase the representational capacity needed to capture minority-cell-specific expression patterns. However, increasing the number of primary capsules to 64 does not provide further improvement and slightly reduces the F1-score, possibly due to overparameterization or reduced capsule specialization. Similarly, increasing the number of routing iterations improves performance up to 3 iterations, after which the performance saturates. These results justify the use of 32 primary capsules and 3 routing iterations in all subsequent experiments. Additional details and plots for the ablation analysis are provided in the S1 Text.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Ablation analysis of RareCapsNet hyperparameters. F1-scores are reported for rare-cell detection under the S1.2 simulation setting. The results support the use of 32 primary capsules and 3 routing iterations in the main experiments.

https://doi.org/10.1371/journal.pcbi.1013962.t003

Robust identification of rare cell type in poorly covered cells

To evaluate the performance of RareCapsNet in scenarios where rare cell populations are poorly covered or have low sample representation, we tested our model on two scRNA-seq datasets: Jurkat (a T-cell leukemia cell line) and CBMC (Cord Blood Mononuclear Cells).

Jurkat–293T mixed-cell experiment

The Jurkat cell line, originally established from the peripheral blood of a 14-year-old boy with T-cell leukemia, is an immortalized human T lymphocyte line widely used to study T-cell signaling and leukemia [7]. The Jurkat–293T mixture contains two human cell lines, Jurkat and HEK 293T, with approximately 3,200 cells and approximately 2,000 genes after preprocessing. We also clarify that Jurkat cells constitute approximately 2.5 of the total population and were treated as the rare target population, while 293T cells formed the majority class. In our experiments, we utilized a mixed dataset comprising Jurkat and 293T cells, where Jurkat cells constituted approximately 2.5% of the total population. This artificial dilution simulates conditions with extremely low representation of the minority class.

RareCapsNet was trained on 80% of the cells, with the remaining 20% reserved for testing (with a stratified setting). The model consistently identified key activated primary capsules linked to the Jurkat cells and successfully highlighted interpretable gene markers. Compared to FiRE, CellSIUS, GiniClust, and RaceID, RareCapsNet exhibited superior F1-score and precision in identifying the rare Jurkat cell subtype (see Fig 3, panel-C). Notably, RareCapsNet maintained a high recall even at a 0.5% population ratio, demonstrating robustness in extremely low prevalence settings. It can also be noted from Fig 3A that the Leiden clustering result largely separates the two transcriptomic groups, although a few cells show locally ambiguous assignments. Such ambiguity can arise because Leiden clustering is performed on a nearest-neighbor graph, whereas UMAP provides a two-dimensional projection of that graph; therefore, apparent visual proximity in UMAP may not always exactly match the graph connectivity used for clustering. In addition, rare cells represented by very few samples may be more sensitive to local neighborhood uncertainty and expression noise. This observation further motivates the use of RareCapsNet, which combines supervised capsule representations with coupling-based signals for improved rare-cell detection beyond visual clustering alone.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Robust identification of poorly covered rare cell populations.

(A) UMAP visualization of the Jurkat–293T mixed dataset showing the original cell labels (left) and Leiden clustering results (right). While Leiden clustering fails to distinctly separate the rare Jurkat population, RareCapsNet successfully captures the minority class through capsule-based representations. (B) UMAP visualization of the CBMC dataset highlighting rare immune cell types, including megakaryocytes (Mk), dendritic cells (DCs), and plasmacytoid dendritic cells (pDCs). RareCapsNet identifies these rare populations through distinct primary capsule activations, even when their abundance is below 1%. (C) Quantitative comparison of RareCapsNet against state-of-the-art rare cell detection methods (FiRE, GiniClust, RaceID, and CellSIUS) on the Jurkat and CBMC datasets. Left: F1-scores across methods. Right: recall as a function of decreasing rare cell population proportion. RareCapsNet consistently achieves superior performance, maintaining high recall and precision even at extreme rarity levels (down to 0.5%).

https://doi.org/10.1371/journal.pcbi.1013962.g003

CBMC rare immune-cell experiment

Cord Blood Mononuclear Cells (CBMCs) are derived from umbilical cord blood and encompass a diverse array of immune cell types, including T cells, B cells, natural killer (NK) cells, monocytes, dendritic cells (DCs), megakaryocytes (Mk), and hematopoietic stem and progenitor cells (HSPCs). In our dataset, the most abundant cell types included CD14⁺” monocytes (2293 cells), memory CD4 + T cells (1791), and naïve CD4⁺ T cells (1248). In contrast, several clinically relevant cell types were present in very low numbers: Mk (96 cells), DCs (70), and plasmacytoid dendritic cells (pDCs, 49). These rare populations accounted for less than 1% of the total cell count, making their identification especially challenging.

To evaluate RareCapsNet under such conditions of extreme data imbalance, we trained the model on 80% of the CBMC data and retained 20% for testing. RareCapsNet demonstrated a remarkable ability to detect these rare cell types by learning interpretable and biologically grounded representations. Specifically, it successfully activated distinct primary capsules corresponding to Mk (cluster 6), DCs (cluster 7), and pDCs (cluster 8), capturing lineage-specific gene expression patterns.

In comparison with state-of-the-art rare cell detection tools such as FiRE, CellSIUS, GiniClust, and RaceID, RareCapsNet achieved superior F1-scores and precision, particularly on the rare subsets (see Fig 3, panel-C). Notably, the model maintained high recall even at population ratios as low as 0.5%, underscoring its robustness and sensitivity in detecting biologically meaningful rare populations (see Fig 3, panel-C). These results affirm RareCapsNet’s utility in uncovering rare yet critical immune cell subsets in complex and heterogeneous single-cell datasets.

Identification of dendritic cells in PBMC68k

We next applied RareCapsNet to the PBMC68k dataset from 10x Genomics, which includes over 68,000 peripheral blood mononuclear cells with annotated immune cell types. A known challenge in this dataset is the identification of dendritic cell (DC) subtypes, which are typically present in very low abundance and are poorly separated using conventional clustering tools.

RareCapsNet was trained using a subset of the data and tested on the remaining cells. Through analysis of the coupling coefficients between primary and type capsules, the model identified distinct primary capsules strongly associated with DCs and their subtypes (e.g., pDCs and conventional DCs). Marker gene identification via coupling-based relevance scoring revealed known DC markers such as CLEC4C and LILRA4, as well as putative novel markers supported by literature.

RareCapsNet’s performance in dendritic cell identification outperformed GiniClust and CellSIUS, especially in terms of precision, while offering the added benefit of interpretable marker gene discovery.

Results on Mouse brain data set [18]

We further evaluated RareCapsNet using the Zeisel mouse brain dataset [18], a complex scRNA-seq dataset comprising over 3,000 single cells from mouse cortex and hippocampus. The dataset contains well-annotated neuronal and non-neuronal cell types, including rare cell subtypes such as neurogliaform cells and ependymal cells.

RareCapsNet identified rare subpopulations through distinct primary capsule activations. These capsule activations corresponded to biologically meaningful gene sets associated with rare cell identities, including Reln, Ptn, and Ndnf—genes known to be enriched in neurogliaform cells.

Our model achieved an F1-score of 0.88 for rare neuronal subtypes, compared to 0.73 (RaceID), 0.69 (CellSIUS), and 0.62 (GiniClust), underscoring RareCapsNet’s ability to resolve fine-grained heterogeneity in complex neural tissues. Notably, the coupling-based interpretation also highlighted gene markers that support the functional differentiation of interneuron subtypes.

Rare cell identification in different batches of data

To examine the generalizability of RareCapsNet across independently generated single-cell datasets, we evaluated the model in a cross-dataset transfer setting using Yan [19], Pollen [20], Darmanis [21], and CBMC [22]. These datasets differ in tissue origin, sequencing protocol, number of annotated cell types, and overall expression distribution, thereby providing a challenging setting for evaluating whether learned capsule activations retain biological relevance across datasets.

Before transfer, each dataset was processed using a consistent preprocessing pipeline, including library-size normalization, log-transformation, and selection of highly variable genes. Because different datasets may contain different gene sets, we did not directly transfer the model using unmatched input features. Instead, for each source–target transfer experiment, the input feature space was restricted to the common intersecting genes shared by the source and target datasets. The intersected gene expression matrices were ordered identically before being provided to the trained RareCapsNet model. Thus, all transfer experiments were performed in a matched gene space.

We adopted a transfer setting in which RareCapsNet was trained on a source dataset and then applied to a target dataset without additional retraining or fine-tuning. No explicit feature-alignment layer, domain-adaptation module, or transfer mapping layer was used. It is important to note that RareCapsNet is supervised in its current formulation. The number of type capsules is fixed by the number of annotated cell types in the source training dataset. Therefore, direct transfer is meaningful only for cell types that are shared or harmonized between the source and target datasets. If the target dataset contains a cell type that was not present during training, the current model cannot assign it to a new type capsule. Such cells may instead show low maximum type-capsule activation, weak prediction confidence, or diffuse coupling across multiple type capsules, and can be treated only as candidate outliers or putative novel populations requiring further validation. Thus, the present cross-dataset analysis evaluates transferability of learned capsule representations under a shared-gene and label-harmonized setting, rather than fully open-set discovery of unseen cell types. The goal of this analysis was therefore to assess whether capsule activations learned from one dataset remain informative when evaluated on a related dataset represented using the shared gene space.

As summarized in Table 4, RareCapsNet retained strong rare-cell detection performance in the evaluated cross-dataset settings. The model achieved greater than 85% accuracy when transferring from Darmanis to Pollen and 91% accuracy when transferring from CBMC to Yan. The learned primary capsule activations also remained qualitatively consistent across source and target datasets, suggesting that capsule routing can capture transferable expression patterns associated with rare or underrepresented cellular identities.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Cross-dataset transfer performance of RareCapsNet. For each transfer experiment, RareCapsNet was trained on the source dataset and applied directly to the target dataset using the intersecting shared gene set. No retraining, fine-tuning, or additional feature-alignment layer was used.

https://doi.org/10.1371/journal.pcbi.1013962.t004

However, we emphasize that the current transfer experiment assumes the availability of a sufficiently large shared gene set between datasets. The present implementation does not include a dedicated gene-space alignment module for datasets with largely non-overlapping gene panels. Therefore, cross-platform transfer across substantially different feature spaces remains a limitation of the current framework. Future extensions may incorporate domain-adaptation layers, gene-embedding-based feature alignment, or batch-aware capsule routing to improve transferability across heterogeneous single-cell resources.

Comparison with deep learning-based single-cell methods

To further benchmark RareCapsNet against modern deep learning-based single-cell analysis frameworks, we compared its rare-cell detection performance with scVI [23], scANVI [24], CellAssign [25], and ItClust [26], in addition to the classical rare-cell detection methods considered above. scVI is a variational autoencoder-based generative model for single-cell transcriptomics, while scANVI extends this framework for semi-supervised annotation and dataset harmonization. CellAssign uses a probabilistic marker-informed framework for assigning cells to cell types, whereas ItClust uses neural network-based transfer learning for clustering and cell-type classification.

These methods were evaluated on representative real datasets, including PBMC68k, CBMC, Zeisel mouse brain, and a cross-batch transfer setting. For deep learning-based methods, the learned latent representations or predicted cell-type labels were used for rare-cell classification, and performance was assessed using F1-score for the rare cell population.

As shown in Table 5, deep learning-based methods generally improve over classical rare-cell detection approaches, particularly in the PBMC68k and cross-batch settings. However, RareCapsNet consistently achieves the highest F1-score across all datasets. The improvement is especially notable in the cross-batch transfer setting, where RareCapsNet maintains an F1-score of 0.78, compared with 0.72 for scANVI and 0.75 for ItClust. This suggests that capsule-based routing preserves rare-cell-specific representations more effectively under batch variability.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Comparison of RareCapsNet with classical and deep learning-based single-cell methods. F1-scores are reported for rare-cell detection across representative real datasets and a cross-batch transfer setting. Classical rare-cell tools include RaceID, GiniClust, and FiRE, while deep learning-based baselines include scVI, scANVI, CellAssign, and ItClust. RareCapsNet achieves the highest F1-score across all settings, indicating improved sensitivity for rare cell populations under both within-dataset and cross-dataset evaluation.

https://doi.org/10.1371/journal.pcbi.1013962.t005

The superior performance of RareCapsNet may be attributed to two complementary properties. First, the dynamic routing mechanism enables primary capsules to specialize toward rare cell-associated expression patterns rather than being dominated by abundant cell types. Second, the coupling-based attribution mechanism provides an explicit link between rare-cell prediction and gene-level signals. In contrast, general-purpose latent representation methods such as scVI and scANVI learn useful low-dimensional embeddings but are not explicitly optimized to route rare-cell-specific information into interpretable type capsules. Similarly, CellAssign and ItClust provide useful annotation frameworks, but their performance may depend on marker availability, reference quality, or dataset-specific tuning. These results place RareCapsNet within the broader landscape of modern deep learning methods for single-cell analysis and demonstrate its advantage for interpretable rare-cell identification.

Computational cost and scalability

Because capsule networks involve dynamic routing, we evaluated the computational cost of RareCapsNet relative to commonly used deep learning-based single-cell methods. On a PBMC68k-sized dataset, RareCapsNet required approximately 25 minutes of training time and 8.5 GB peak GPU memory on an NVIDIA RTX 3090 GPU with 24 GB memory. This runtime and memory usage were moderately higher than those of scVI, scANVI, CellAssign, and ItClust in our benchmark, reflecting the additional computational overhead introduced by iterative routing between primary capsules and type capsules.

The main computational cost of RareCapsNet arises from the routing operation and, separately, from the post hoc gene-attribution step. During model training, the number of primary capsules and routing iterations were fixed across experiments, which kept the training procedure practically scalable for datasets in the 50k–100k cell range after highly variable gene selection. Under this fixed architecture, runtime increased approximately linearly with the number of cells and input genes. The gene-attribution step is more computationally intensive because it requires repeated forward passes using masked gene inputs, however, this analysis is performed after training and can be parallelized across genes.

A detailed runtime and memory comparison is provided in S1 Text (section “Runtime, memory usage, and scalability”) and Table-B of S1 Text. Future optimization may include sparse routing, shared transformation matrices, mini-batch-level routing approximation, and GPU-parallelized gene-attribution computation to support atlas-scale single-cell datasets.

Marker gene analysis

To further support the biological interpretability of the capsule-associated genes, we performed marker overlap and functional enrichment analyses. The top-ranked genes identified by RareCapsNet were compared with public marker resources, including CellMarker, PanglaoDB, and curated PBMC or brain cell-type annotations [27,28]. In addition, capsule-associated gene lists were analyzed using GO Biological Process and KEGG pathway annotations [29,30].

The recovered capsule-associated genes showed clear agreement with known rare-cell markers and biologically meaningful functions. For example, pDC-associated capsules recovered CLEC4C, LILRA4, TCF4, and IRF7, which are associated with plasmacytoid dendritic cell identity and interferon response. Dendritic-cell-associated capsules recovered FCER1A, CST3, and HLA-DRA, while megakaryocyte-associated capsules recovered PF4, PPBP, ITGA2B, and GP9. Functional enrichment analysis further showed that these capsule-associated genes were enriched for cell-type-relevant biological processes, including type I interferon signaling for pDCs, antigen processing and presentation for dendritic cells, and platelet activation for megakaryocytes.

As summarized in Table 6, RareCapsNet-derived capsule-associated genes are not merely predictive features but show substantial agreement with known marker resources and enriched biological processes. For top-ranked genes that were not included in canonical marker lists, we now refer to them as putative capsule-associated genes. This more conservative interpretation avoids overclaiming novelty while retaining the biological value of the capsule-based gene attribution framework. Additional details on the marker-resource overlap analysis, GO/KEGG enrichment procedure are provided in the section “Extended biological validation of capsule-associated marker genes.” of S1 Text.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Biological validation of capsule-associated marker genes. Representative top-ranked genes identified by RareCapsNet are shown together with marker-resource overlap and enriched biological functions. Genes not included in canonical annotation lists are interpreted as putative capsule-associated genes rather than definitive novel biomarkers.

https://doi.org/10.1371/journal.pcbi.1013962.t006

Architecture and training objective of RareCapsNet

We provide here the formal architecture and training objective of RareCapsNet. Let denote the preprocessed single-cell expression matrix, where N is the number of cells and K is the number of genes or selected highly variable genes. For a single cell s, the input vector is

(1)

Let T denote the number of annotated biological cell types. RareCapsNet contains T output capsules, referred to as type capsules, where each type capsule corresponds to one predicted cell type. The number of primary capsules is fixed at J = 32 in the final architecture, based on the ablation analysis reported in Table 3.

The input vector is first projected through a fully connected layer:

(2)

where and are trainable parameters, H is the hidden dimension, and denotes a nonlinear activation function.

The primary capsule layer consists of J capsules. The output of the j-th primary capsule is

(3)

where is the vector output of primary capsule j. The squash function is defined as

(4)

This nonlinearity constrains capsule vector lengths to lie between 0 and 1, so that longer vectors indicate stronger capsule activation.

For each primary capsule j and type capsule i, RareCapsNet computes a prediction vector

(5)

where is a trainable transformation matrix.

Dynamic routing is used to determine the coupling strength between primary capsules and type capsules. Routing logits are initialized as . At routing iteration r, coupling coefficients are computed using a softmax over type capsules:

(6)

For each type capsule i, the input vector is computed as

(7)

and the output of the type capsule is

(8)

The routing logits are updated using agreement between the prediction vector and the type capsule output:

(9)

In all experiments, we used three routing iterations, as supported by the ablation analysis.

After routing, the predicted cell type for cell s is obtained from the type capsule with the largest vector norm:

(10)

RareCapsNet is trained using the standard capsule margin loss. Let if the true cell type of sample s is i, and otherwise. The loss for sample s is

(11)

where m⁺ and m^- are positive and negative margins, respectively, and controls the contribution of absent cell types.

The model was trained using mini-batch gradient-based optimization. For each dataset, cells were divided into training, validation, and test sets. The training set was used to optimize the network parameters, the validation set was used for model selection and hyperparameter checking, and the test set was used only for final evaluation. After training, the coupling coefficients between primary capsules and type capsules were extracted for interpretation and for computing gene-specific coupling coefficients, as described in the following subsections.

Definition of primary capsules and type capsules

In RareCapsNet, a primary capsule refers to an intermediate trainable vector-valued unit that learns latent gene-expression patterns from the input single-cell expression profile. Primary capsules should not be interpreted as pre-defined feature groups, gene modules, or biological cell types. Instead, they are learned representations whose specialization emerges during training through the dynamic routing procedure. The number of primary capsules is therefore a model hyperparameter; in the final architecture, we use J = 32 primary capsules based on the ablation analysis.

A type capsule, in contrast, refers to an output capsule corresponding to a predicted biological cell type. If a dataset contains T annotated cell types, the model contains T type capsules. The length of each type capsule output vector represents the activation strength for the corresponding cell type, and the predicted label is assigned to the type capsule with the largest norm. The coupling coefficient measures the routing strength from primary capsule j to type capsule i, and is later used for model interpretation and gene-attribution analysis.

A note on the dynamic routing procedure

Dynamic routing in RareCapsNet is an iterative agreement-based mechanism that determines how strongly each primary capsule contributes to each type capsule. Equation Equation (6) defines only one component of this procedure: the softmax normalization of routing logits into coupling coefficients. The complete routing procedure consists of four steps. First, each primary capsule produces a prediction vector for every type capsule. Second, the routing logits are converted into coupling coefficients using the softmax function. Third, each type capsule receives a weighted sum of prediction vectors and applies the squash nonlinearity to produce its output vector. Fourth, the routing logits are updated according to the agreement between the prediction vector and the type capsule output. This procedure is repeated for a fixed number of routing iterations. In our final model, we use three routing iterations, as supported by the ablation analysis.

Supervised setting and candidate outlier detection

RareCapsNet is primarily formulated as a supervised rare-cell identification framework. During training, the model requires cell-type labels to learn the association between gene-expression patterns, primary capsules, and type capsules. Therefore, the current implementation is most directly applicable to rare, poorly covered, or underrepresented cell types that are present in the training annotation or in a related reference dataset.

For a new dataset, RareCapsNet assigns each cell to the type capsule with the largest activation. If a cell does not match any learned type capsule, it may show low maximum type-capsule activation, weak prediction confidence, or diffuse coupling across multiple type capsules. Such cells may be flagged as candidate outliers or putative novel populations for further downstream analysis. However, these candidates should not be interpreted as definitive novel cell types without additional validation, such as clustering of flagged cells, marker-gene inspection, comparison with public marker databases, and functional enrichment analysis. Thus, the present RareCapsNet framework supports supervised rare-cell detection and candidate outlier flagging, while fully unsupervised or open-set novel cell-type discovery remains a future extension.

Theoretical foundations of RareCapsNet gene attribution

First we formalize our gene attribution method and provide theoretical justification here. We introduce key definitions and propositions (with proofs) that supports the interpretation of coupling coefficients and the identification of marker genes.

Definition 1 (Gene-Specific Coupling Coefficient). Let S be the set of training single-cell samples, with each sample represented by a K-dimensional expression vector across K genes. For a given trained RareCapsNet network, denote by the coupling coefficient between primary capsule j and type capsule i when the network processes sample s (through the dynamic routing procedure). For any gene (with ), we construct a masked input from s by retaining only the expression of and setting all other gene expression values to 0. Let be the resulting coupling coefficient between primary capsule j and type capsule i when the network is fed . We then define the gene-specific coupling coefficient for gene as the average coupling (with equal weight across cell types) over all training samples:

(12)

where T denotes the number of cell types (and hence the number of type capsules, ), t(s) denotes the cell type label of sample s. Here, N(t) denotes the number of cells belonging to cell type t. The inner average computes the mean gene-specific coupling within each cell type, while the outer average gives equal contribution to each cell type. This stratified averaging prevents abundant cell types from dominating the attribution score and allows rare cell types to contribute equally to the final gene-specific coupling coefficient.

Coupling-based definition of explainability

In RareCapsNet, explainability is defined through dynamic-routing coupling coefficients rather than through ordinary correlation. The coupling coefficient measures the fraction of information from primary capsule j that is routed to type capsule i during prediction. A large value of therefore indicates that primary capsule j contributes strongly to the prediction of cell type i. To obtain gene-level explanations, we compute gene-specific coupling coefficients by masking all genes except a single gene and measuring the resulting coupling between primary capsule j and type capsule i. Genes with unusually large gene-specific coupling values are interpreted as capsule-associated genes for the corresponding cell type.

The factor 1/N(t) in Eq. (12) averages the coupling coefficients over samples within cell type t. This prevents cell types with larger sample sizes from dominating the gene-specific coupling score. The outer factor 1/T then assigns equal weight to each cell type. Therefore, Eq. (12) implements a stratified averaging scheme that is especially important in rare-cell detection, where cell-type frequencies are highly imbalanced.

This definition formalizes the intuition that measures how strongly gene contributes to activating the link from primary capsule j to type capsule i, aggregated over the entire training set with a stratified (per-cell-type) averaging. Next, we establish some key properties of the coupling coefficients arising from the dynamic routing algorithm in the RareCapsNet network.

Proposition 1 (Normalization and Interpretability of Couplings). For any input sample s (or masked sample ) passed through the MarkerCapsule network, the dynamic routing algorithm produces coupling coefficients satisfying:

Normalization: for each primary capsule . In other words, for a fixed primary capsule j, its coupling coefficients to all T type capsules form a probability distribution.

Preference for Alignment: If the output (“prediction vector”) of primary capsule j aligns well with type capsule i for the given input s, then will be correspondingly large. Conversely, if there is little or no agreement between capsule j’s features and the characteristics of type i, remains small. Thus, can be interpreted as the fraction of capsule j’s output that is assigned to explaining the presence of cell type i in sample s.

Proof. We recall the standard dynamic routing procedure used in capsule networks. Fix an input sample s (or masked sample ). Let denote the prediction vector produced by primary capsule j for type capsule i, i.e.,

where is the output of primary capsule j for input s, and is a learned linear transformation. Dynamic routing maintains real-valued routing logits and coupling coefficients defined by a softmax over i for each fixed j:

(13)

The type capsule input and output are computed as

where is the standard nonlinearity ensuring .

(1) Normalization. For any fixed j, the right-hand side of (13) is the standard softmax distribution over . Therefore,

Hence the coupling coefficients from a fixed primary capsule j form a probability distribution over type capsules.

(2) Preference for Alignment. Dynamic routing updates the logits using an agreement term. A common and standard update rule is

(14)

i.e., the scalar product between the prediction vector from capsule j to capsule i and the current output of capsule i. Consider two type capsules i and for a fixed j. If the agreement satisfies

then after the update (14), we have increased by a larger amount than . Since is a strictly increasing function of when other logits are fixed (as a property of the softmax), it follows that the updated coupling coefficient to capsule i becomes larger relative to that of capsule . More formally, for fixed j, define the softmax map

This map is componentwise monotone: if increases while all other remain fixed, then increases. Therefore, repeated routing iterations amplify couplings toward those type capsules that maintain higher agreement values .

In particular, when is well aligned with , the inner product in (14) is large, causing and thus to increase, meaning that primary capsule j assigns more of its output to explaining type capsule i. This establishes the stated preference-for-alignment behavior. □

An important implication of Proposition 1 is that, for each cell type t, there tends to exist at least one primary capsule j that specializes in capturing features of that type. Empirically, one can identify for each type capsule i the primary capsule j with the largest total coupling (averaged over that type’s samples). We denote this capsule as and term it a dedicated capsule for cell type i. In our experiments, we indeed observe a roughly one-to-one alignment between cell types and specific primary capsules (see, e.g., primary capsule 9 for cell type “CD16 + Mono” in Fig 1). This alignment justifies restricting attention, for each type i, to the primary capsule j⁽ⁱ⁾ that carries the strongest signal for that type when identifying marker genes.

We now formalize the procedure for marker gene identification and establish its statistical validity. Intuitively, if gene is a bona fide marker for cell type i, then feeding (in isolation) into the network should strongly activate the pathway from some primary capsule (ideally ) to type capsule i, resulting in an unusually large coupling coefficient . By contrast, for a gene with no specific association to type i, the coupling should fluctuate around some baseline (background noise level) and not produce extreme values. We capture this intuition by a hypothesis testing framework: treat each gene’s coupling as a test statistic for the null hypothesis “gene is not a specific marker for type i.”

To proceed, we assume that the null distribution of the coupling coefficients (for non-marker genes) is approximately Gaussian. Empirically, the histogram of often resembles a normal distribution centered around a mean, with a heavy tail on the right (corresponding to a few genes with exceptionally high coupling, see Fig 1, discussed later). Let and be the mean and standard deviation of across all genes for a given (i,j) combination. We define a one-tailed p-value for each gene with respect to type i (and its dedicated capsule j⁽ⁱ⁾) as

(15)

where is a Gaussian random variable fitted to the null coupling distribution (genes in the middle of the distribution) for capsule j⁽ⁱ⁾. In other words, is the right-tail probability (extreme-value significance) of the observed coupling . Genes with exceptionally large coupling to type i will thus attain very small p-values.

Given these p-values for all genes, we employ the Benjamini–Hochberg (BH) procedure to identify significant marker genes while controlling the false discovery rate (FDR) at a chosen level (we use by convention). Let be the sorted p-values for type i. The BH procedure finds the largest index L such that All genes corresponding to p-values are then declared as significant markers for cell type i. We denote the resulting set of marker genes by

(16)

Theorem 1 (Marker gene identification and FDR control). Fix a cell type i and its selected primary capsule . For each gene , let be the gene-specific coupling coefficient defined in (12). Let be the corresponding one-sided p-value defined by

and suppose that under the null hypothesis H_0,k (“ is not a marker for type i”), is valid, i.e.,

(17)

Apply the Benjamini–Hochberg (BH) procedure at level to the K p-values and let be the resulting rejection set (marker set) as in (16). If the null p-values are independent (or satisfy the standard PRDS condition), then the false discovery rate satisfies

where is the number of selected genes and V is the number of false selections. Moreover, if there exists a separation margin such that the coupling statistic for true marker genes lies strictly in the right tail of the null distribution, then the probability that a true marker gene is included in converges to 1 as per-type sample sizes grow.

Proof. We prove (A) FDR control under BH and (B) asymptotic detection of true markers. The FDR-control statement follows the standard Benjamini–Hochberg theory under independent or PRDS null p-values. We include the theorem for self-contained presentation. The novel aspect in the present work is the construction of capsule-coupling-based gene-specific statistics and their use for marker-gene identification within RareCapsNet.

(A) FDR control.

Fix the type i and write for simplicity. Let be the set of true null hypotheses (non-marker genes for type i), and let . Let be the ordered p-values and define the BH threshold index

with the convention that L = 0 if the set is empty. The BH rejection set is

so and . Let be the number of false discoveries.

To bound , we use the classical BH argument. For each null index , define the indicator of rejection . Then

Hence, using linearity of expectation,

(18)

We now condition on all p-values except . Let denote the vector of p-values excluding . Define as the number of BH rejections that would occur if we set and keep fixed, i.e., apply BH to . Under independence (or PRDS), one can show that the event implies and that R is nonincreasing in . Therefore, on the event that is rejected and ,

Using this inequality inside (18) yields

(19)

Now apply the tower property conditioning on :

For fixed , the event is of the form for some data-dependent threshold (this is a standard property of step-up procedures). Thus,

where . Under the null, is valid and (under independence) uniform, so and . Hence the integral becomes

which is improper; the classical BH proof avoids this route by using a sharper argument:

This inequality holds under independence (and extends under PRDS) and is the key lemma in the BH proof. Applying it to each and summing yields

Thus BH controls the FDR at level .

(B) Asymptotic detection of true marker genes.

Let be a true marker gene for type i. Consider the stratified estimator

Assume finite second moments and define the type-conditional mean

By the strong law of large numbers, for each t,

and therefore,

Assume a strict separation condition: there exists such that

which captures that gene produces systematically higher routing agreement toward type capsule i (through capsule ) for cells of type i. Then the limit lies strictly to the right of the null mean used to compute , implying that as sample sizes grow, because the right-tail probability of a Gaussian decays to 0 as the observed statistic moves deeper into the upper tail. Consequently, for sufficiently large sample sizes, will be among the smallest p-values and will pass the BH threshold with probability approaching 1. Hence a true marker gene is selected with probability tending to 1. □

The above theoretical framework demonstrates that our coupling-based gene attribution method is grounded in sound principles. Coupling coefficients provide an interpretable measure of gene influence on capsule–type associations (Proposition 1), and by analyzing their distribution we can identify marker genes with statistical confidence (Theorem 1). We next describe the practical algorithms that implement these ideas in the RareCapsNet network.

Algorithm for Computation of Gene Specific Coupling Coefficients for rare cells

Algorithm 1 Compute Gene Specific Coupling Coefficients

Require: Trained marker capsule network, training dataset S

Ensure: Coupling coefficients for each gene , primary capsule j, and type capsule i

1: for each gene , do

2:  for each sample do

3:   Construct masked input vector by retaining only expression value of

4:   Run through the trained network

5:   Extract coupling coefficients between primary capsule j and type capsule i

6:  end for

7: end for

8: for each type capsule and primary capsule do

9:  for each gene do

10:     Compute average: { is the set of samples with cell type i}

11:  end for

12: end for

13: return for all k, i, and j

Let k = 1,...,K refer to genes , t = 1,...,T to cell types, j = 1,...,J to primary capsules, and i = 1,...,I to type capsules. Note that although T = I we need to distinguish between types of single cells t and type capsules i in the following, which explains the different indices. Let further S be the training single cell samples. Each single cell corresponds to a K-dimensional real valued vector, where entries s[k], k = 1,...,K correspond to the expression of gene in single cell s. Training data are labeled by cell types, we refer to t(s) as the type of single cell s. Let be the number of single cell samples for which t(s)=t.

According to the setup of the RareCapsNet architecture, J = 32 for all data sets. The number of cell types T (and hence I = T) can vary with T taking values between 11 and 15. Equally, the number of genes K supported by the particular data set can vary. In the experiments in which the algorithm here is used, we make use of data set 1 (mRNA + protein expression; CITE-seq [22]). According to this data set, T = I = 13 and K = 2000, see also the detailed specifications of the data sets in the Supplement.

In the following we outline an algorithm that, when provided with the trained MarkerCapsule network architecture and the training samples S, will output gene specific coupling coefficients . These indicate how strongly gene contributes to activating the link from primary capsule j to cell type capsule i.

The algorithm proceeds according to the following step

1. For each training sample and each gene , consider defined by (20)
   That is, keeps s[k] from the original input of single cell s and has all other entries masked, that is, set to zero.
2. For each combination (s,k) of single cell s and gene , we pass the masked vector through the trained RareCapsNet network. In the present experiments, no additional auxiliary data were used, and therefore itself is provided as the network input. If future extensions include auxiliary cell-level covariates or additional modalities, such data should first be preprocessed and aligned with the corresponding cell s, and then combined with according to the specified input representation before being passed to the network.
3. Running through the network includes executing the dynamic routing procedure, hence yields coupling coefficients for all i = 1,...,I, j = 1,...,J
4. One averages the resulting across samples assigning equal weight to each cell type t: (21)
5. One outputs as the desired gene specific coupling coefficients.

Algorithm for computing marker genes through rare capsules

Given a particular cell type i, one determines the primary capsule j that yields the largest coupling coefficient . Note that if several primary capsules j yield sufficiently large , one can extend the analysis on all such j; for the sake of an analysis that gives rise to an unbiased comparison across cell types, we restrict ourselves to the primary capsule j that yields the larges for each i.

One then determines the gene specific coupling coupling coefficients for all genes , as per the algorithm described in Subsection. Subsequently, one determines the mean and the standard deviation of the ; as can be seen in Fig 1 (for the combination of primary capsule 9 and cell type ’CD16 + Mono’, which exemplifies the situation for all selected combinations j and i), the empirical distribution of the roughly follows a Gaussian distribution, with the exception of an heavy tail towards the right. The idea is to collect all genes that give rise to this heavy tail.

To do so in a sound way, we determine p-values reflecting tail probabilities with respect to the Gaussian distribution the usual way. Subsequently, we sort the resulting in ascending order (where the smallest corresponds with the largest ) yielding

and determines

that is the largest l for which , where reflects the significance level at which one operates. Here, , which follows standard conventions. Overall, this procedure reflects the Benjamini-Hochberg procedure, which accounts for the necessary corrections due to multiple hypothesis testing [31–33].

The set of marker genes for cell type i is then defined to be

(22)