Construction of the mouse-Geneformer

Architecture.

We constructed mouse-Geneformer, a deep learning model aimed at predicting the gene network of normal mice. We developed the mouse-Geneformer following the original human version of Geneformer, a context-aware, attention-based deep learning model, which was pretrained on large-scale transcriptome data comprising approximately 30 million human single-cell RNA-seq data, developed by Theodoris et al [10]. The architecture and transfer learning strategy employed in our development of mouse-Geneformer are the same as those of the original human Geneformer, with minor modifications. An overview diagram of the mouse-Geneformer is shown in Fig 1. Instead of human RNA-seq data, we used mouse single-cell RNA-seq data as an input corpus. The transcriptome of each single cell was presented to the model by using the Rank Value Encoding method that was also developed in the original human Geneformer [10]. Finally, a Geneformer model constructed with Transformer Encoder using the created cell texts is pre-trained with self-supervised learning to build the mouse-Geneformer.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the mouse-Geneformer.

mouse-Genecorpus-20M is a large mouse transcriptome dataset obtained from single-cell RNA-seq experiments, Rank value encoding is a method for generating “cellular sentences”, Transformer Encoder is the architecture of the mouse-Geneformer.

https://doi.org/10.1371/journal.pgen.1011420.g001

Construction of mouse-Genecoupus-20M.

We constructed a large-scale dataset of mouse single-cell RNA-seq data, termed mouse-Genecorpus-20M, derived from healthy wild-type Mus musculus strains to learn the gene network of healthy wild-type mouse. The dataset was compiled from publicly available mouse single-cell RNA-seq data sourced from multiple databases: PanglaoDB [13], Single Cell Expression Atlas [14], Single Cell Portal [15], ENCODE project [16], 10x Genomics [17], and CELLxGENE [18]. We used datasets generated by droplet-based sequencing platforms such as Chromium (10x Genomics), Drop-seq, DropNuc-seq, MIRACL-seq, and inDrops. In total, 1,089 RNA-seq datasets were compiled. The complete list of datasets, along with their metadata, is provided in S1 Table, and an overview is shown in Fig 2. These datasets collectively represent single-cell transcriptome data from 119 million cells before filtering (see below). The majority of the datasets were derived from C57BL/6 mice or closely related strains. However, the precise genetic composition of the dataset remains uncertain due to the limited availability of strain annotations in public databases. Given the substantial genetic and transcriptional differences among mouse strains, users should carefully consider these factors when applying this model, particularly in contexts where genetic diversity may influence the results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Building mouse-Genecourpus-20M from single-cell RNA-seq datasets.

(A) Overview of the process for creating mouse-Genecorpus-20M. (B) Sources of the databases used for building the mouse-Genecorpus-20M. The vertical axis shows the percentage of the number of datasets or cells derived from each database. (C) Distribution of mouse organs included in the mouse-Genecorpus-20M. The vertical axis indicates the percentage of cell numbers corresponding to each organ. (D) Age and developmental stages of mice used in the mouse-Genecorpus-20M. The vertical axis indicates the percentage of cell numbers corresponding to mouse age or embryonic stage. Labels starting with “E” represents embryonic and “P” represents postnatal stages.

https://doi.org/10.1371/journal.pgen.1011420.g002

These single-cell datasets were stored and processed in HDF5 format loom files, H5AD files, and pickle files using the tools and libraries including scanpy, loompy, pandas, numpy, pickle, and scipy. The raw gene expression data is structured in matrices where rows represent genes, columns represent cells, and the gene expression levels are stored in the cells of the matrix. The dataset uses Ensembl IDs for gene names. Since the Ensembl ID version varied across datasets, the version information was ignored.

Starting with 119 million cells from the compiled 1,089 datasets, we applied filtering based on metadata and quality. Since mouse-Genecorpus-20M is intended for learning the gene network of normal mice to build a reference genetic architecture, we excluded single-cell data with high mutation loads that could cause restructuring of gene networks, such as those from cancer cells or immortalized cell lines. Quality filtering also addressed artifacts common in droplet-based single-cell data, such as ambient RNA [19,20], doublets [21,22], and data derived from empty droplets. The specific filtering conditions applied were as follows: 1) For each dataset, cells with total gene expression levels more than three standard deviations from the mean were excluded. 2) Cells with mitochondrial RNA expression levels more than three standard deviations from the mean were excluded. 3) Cells with fewer than seven detected genes per cell were removed. 4) Cells where the sum of the expression levels of all genes per cell exceeded 20,000 were excluded, as excessively high expression levels may indicate artifacts such as doublets or multiplets. After applying these filters, the final dataset was 1,070 RNA-seq datasets and encompassed 20,630,028 cells. The mouse-Genecorpus-20M dataset is formatted in Apache Arrow and is available for download from the repository (https://huggingface.co/datasets/MPRG/Mouse-Genecorpus-20M). It can be accessed using the Python library “datasets” for downstream applications.

Pretraining of the mouse-Geneformer.

Pretraining was conducted following the procedures described previously for the original human Geneformer [10], with some modifications. Details are described in the Results section. Briefly, the mouse-Genecorpus-20M dataset was processed using Rank Value Encoding to extract genes that capture cell features, forming cell sequences with these genes as tokens. A special token called [CLS] is added for classifying cell sequences. The resulting data, consisting of cell sequences with token IDs, is used as input for pre-training the mouse-Geneformer. The pretraining employed a masked token prediction task, where 15% of the tokens in the input sequences were randomly masked, and the model was trained to predict these masked tokens using the remaining tokens. This task optimizes the network, allowing the mouse-Geneformer to learn the relationships and expression patterns of genes in mice.

We conducted the pretraining of the mouse-Geneformer using mouse-Genecorpus-20M under the conditions shown in Table 1. The training utilized 8 NVIDIA V100 GPUs, each with 32 GB of memory, and the process took approximately 2 days

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of Experimental conditions.

https://doi.org/10.1371/journal.pgen.1011420.t001

Fine-turning of the mouse-Geneformer

Fine-tuning the mouse-Geneformer enables accurate classification of cell types and disease types, even with limited mouse single-cell data, such as data containing rare cells, disease-specific data, or organ-specific data. This process leverages the knowledge gained from pretraining on mouse-Genecorpus-20M to predict gene networks accurately. Fine-tuning involves adding a classification layer to classify the [CLS] token in the final layer of the mouse-Geneformer. The model is initialized with the weights from the pre-trained mouse-Geneformer and further trained using a small amount of specific data. When the fine-tuning task is similar to the pretraining task, some layers of the Transformer Encoder Block are fixed to enhance generalization performance. To mitigate overfitting, which can occur due to the limited amount of data, the number of training iterations is reduced.

Cell type classification using the mouse-Geneformer and conventional methods

To evaluate mouse-Geneformer, we compared its performance on cell type classification tasks with that of conventional methods, Single-cell VAE (scVAE) [23] and scDeepSort [24] . The mouse data used for comparison comprised single-cell data from twelve organs of mice: urethra and prostate mixed dataset [31] (Gene Expression Omnibus (GEO): GSE145929), embryos [32] (Gene Expression Omnibus (GEO): GSE197353), and kidneys [33] (Gene Expression Omnibus (GSE): GSE190094), tongue [25], thymus [25], mammary gland [25], large intestine [25], limb muscle [25], spleen [25], heart [25], brain [25], and kidney [25] (Gene Expression Omnibus (GSE): GSE132042). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets). These data were not included in the mouse-Genecorpus-20M. To confirm the integrity of the data, we compared features using Euclidean distance, and the results indicated that no data was completely zero, confirming the absence of data leakage. Each method was employed to classify the data, and their accuracies are compared. For cell type classification with mouse-Geneformer, the model was fine-tuned for each organ data for this task. The fine-tuning conditions are detailed in Table 1. For cell type classification with scVAE, a method applying the deep generative model Gaussian-mixture VAE (GMVAE) [26] to single-cell RNA-seq analysis [23], we used the GMVAE with a three-layer neural network for the encoder, a Gaussian mixture distributions for the latent space, a three-layer neural network for the decoder, and a three-layer neural network for the class classification network. Additionally, the categorical distribution used in this GMVAE employed the Gumbel-Softmax [27,28]. For cell type classification with scDeepSort, a pretrained deep learning model utilizing weighted Graph Neural Networks (GNNs) for single-cell RNA-seq analysis [29], we employed its three-layer graph neural network architecture [24]. The data were randomly split into two groups, one for training (80%) and the other for testing, (20%). The evaluation metric for this experiment were accuracy and F1 score (a harmonic mean of precision and recall), which were calculated by using the ‘accuracy_score’ and ‘f1_score’ function from the scikit-learn library in Python, respectively, and the highest accuracy and F1 scores was recorded. Cell distribution was visualized using UMAP [30] performed using the scanpy library in Python.

Cell type classification using mouse-Geneformer with and without prior learning

To evaluate the effect of pretraining, we compared the performance of pretrained mouse-Geneformer with non-pretrained one for the task of classification of cell types. The data used for evaluation included single-cell RNA-seq data from the mouse urethra and prostate mixed dataset [31] (Gene Expression Omnibus (GEO): GSE145929), embryos [32] (Gene Expression Omnibus (GEO): GSE197353), and kidneys [33] (Gene Expression Omnibus (GSE): GSE190094). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets) and were not included in the mouse-Genecorpus-20M. This embryo dataset is whole mouse embryo at embryonic age E9.5. These datasets were not included in mouse-Genecorpus-20M. The mouse-Geneformer models with and without pretraining were employed to classify the data, and their accuracies are compared. We compared the similarity between the data used for pretraining and the data used for evaluation by calculating the Euclidean distance of their features. As a result, no data was found to be completely zero, confirming the absence of data leakage. To perform cell type classification with mouse-Geneformer, the models are fine-tuned for each organ data for the cell type classification task. The fine-tuning conditions are presented in Table 1. The data splitting method, evaluation metrics, and cell distribution visualization methods are the same as those used in cell type classification experiments (see above).

In silico perturbation experiments

The in silico gene manipulation experiment using Mouse-Geneformer applied perturbations to each gene to identify those that contribute significantly to the disease. The Mouse-Geneformer in silico perturbation experiment first normalized the gene expression data and then applied Rank Value Encoding to rank the expression of each gene, creating a ranked gene table. Then, Mouse-Geneformer was fine-tuned for disease classification tasks, ensuring that the disease classification accuracy on the test data exceeds 90%. Subsequently, the genes in the dataset were randomly perturbed multiple times, and inference was performed using the fine-tuned model. For “virtual” gene deletion in silico, rather than setting its expression value to 0, we completely removed the gene from the dataset. The rank values of genes with lower ranks than the deleted gene were each increased by one. Conversely, for “virtual” activating a gene in silico, the target gene was repositioned to have a rank value of 1, and the rank values of genes ranked higher than the activated gene were each decreased by one. Cosine similarity was used to quantify the distances between the perturbed cell state and a specific cell state. To analyze the effect of genetic perturbations, statistical significance was assessed using the Wilcoxon rank sum test (Mann-Whitney U test), following the approach used in the original human Geneformer study [10]. First, the gene expression table of each reference mouse cell is input into mouse-Geneformer, and an average feature vector representing the gene expression state of the cell was computed. Next, the gene expression table of a cell with a perturbed specific gene was input into mouse-Geneformer to generate a new set of feature vectors. The cosine similarity between these feature vectors and the reference average feature vector was then calculated to obtain the “distribution of cosine similarity under specific gene perturbation.” To establish a control, a random gene in the gene table of each perturbed mouse cell was perturbed, and the feature vectors were computed in the same manner. The cosine similarity between this set of feature vectors and the reference average feature vector was then calculated, yielding the “distribution of cosine similarity under random gene perturbation.” The Wilcoxon rank-sum test was performed on these two distributions to test the null hypothesis: “The effect of perturbing a specific gene on cell state is comparable to that of perturbing a random gene.” If the p-value was below 0.05, the null hypothesis was rejected, indicating that the perturbation of the specific gene has a statistically significant effect on the cell state. This statistical approach allowed for a quantitative assessment of the extent to which perturbation of a specific gene affects cell state compared to random perturbation, thereby enabling the prediction of genes potentially associated with disease.

To evaluate the effectiveness of in silico perturbation experiments using mouse-Geneformer, we compered the genes predicted in silico to those identified in vivo experiments. The data used for evaluation included single-cell data from diabetic nephropathy, UMOD nephropathy, and normal kidney data [33] (Gene Expression Omnibus (GSE): GSE190094), as well as data from cells with knocked out COP1 protein and normal cells [34] (Gene Expression Omnibus (GSE): GSE147559). These datasets were not included in mouse-Genecorpus-20M. To ensure the integrity of the data, we compared features using Euclidean distance, and the results indicated that no data was completely zero, confirming the absence of data leakage. To perform disease typing with mouse-Geneformer, the model was fine-tuned for each disease data for this task. Fine tuning conditions are shown in Table 1. Disease labels were assigned at the organ level: cells from diseased organs were labeled as diseased, while cells from normal organs were labeled as normal. The cells were then classified according to these labels. Only models that achieved disease subtype classification accuracies above 90.00% on the test data were used for evaluation. The data were randomly split into training (80%) and testing (20%) sets. For the in silico perturbation experiments for diabetic nephropathy, we perturbed the cell states of normal kidney cells to resemble those of diabetic nephropathy. For UMOD nephropathy experiments, we perturb the cell states of cells from UMOD nephropathy to resemble normal kidney cells. In COP1 KO experiments, we perturb the cell states of cells with COP1 knocked out to resemble normal cells.

Gene name conversion between mouse and human for cross-species application

To convert human genes to mouse genes for cross-species application, we used the databases of Mouse Genome Informatics (MGI) [35] and the HUGO Gene Nomenclature Committee (HGNC) [36]. HGNC BioMart application (https://biomart.genenames.org/martform/#!/default/HGNC?datasets=hgnc_gene_mart_2024_03_26) serves the function to provide MGI IDs for mouse homologs of queried human genes. The MGI Batch Summary (https://www.informatics.jax.org/batch/summary) provides a feature to convert Ensembl IDs to corresponding mouse MGI IDs queried. Specifically, the conversion from human genes to mouse genes involved the following steps: 1) converting human Ensembl IDs to mouse MGI IDs using HGNC; 2) converting mouse MGI IDs to mouse Ensembl IDs and Gene Symbols using MGI. When converting human Ensembl IDs to mouse Ensembl IDs, we use a conversion table that ensures a one-to-one correspondence. In cases where the human Ensembl ID to the mouse homolog MGI ID and MGI ID to the mouse Ensembl ID don’t have a one-to-one correspondence, the following steps are taken to ensure a one-to-one correspondence between human Ensembl IDs and mouse Ensembl IDs: 1) Sort the gene symbols corresponding to human Ensembl IDs in alphabetical order, and then map the last gene symbol to the mouse homolog MGI ID for the human Ensembl ID; 2) Map the mouse homolog MGI ID to the final mouse Ensembl ID in the output. After converting human Ensembl IDs to mouse Ensembl IDs, we obtain 18,269 genes. There are 910 human Ensembl IDs that have a one-to-many correspondence, and there are 687 mouse homolog MGI IDs that have a one-to-many correspondence.

Human cell type classification using mouse-Geneformer

To explore whether mouse-Geneformer could be used for the analyses of other organisms, we analyzed three single-cell RNA-seq datasets derived from three human organs: breast [37] (Gene Expression Omnibus (GEO): GSE195665), thymus [38] (Gene Expression Omnibus (GEO): GSE144870), and cerebral cortex [39] (Brain Initiative Cell Census Network (BICCN): SCR_016152). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets) and were not included in Genecorpus-30M. To confirm the integrity of the data, we compared features using Euclidean distance, and the results indicated that no data was completely zero, confirming the absence of data leakage. To perform human cell type classification using mouse-Geneformer, human genes in the original datasets were converted to their mouse homologs as described in the previous section. The mouse-Genformer and Geneformer were employed to classify each data, and their accuracies are compared. The Geneformer was a human-Geneformer model that we pre-trained using Genecorpus-30M (https://huggingface.co/datasets/ctheodoris/Genecorpus-30M). For cell type classification with mouse-Geneformer and Geneformer, models were fine-tuned for each organ data for this task. The fine-tuning conditions are detailed in Table 1. The procedures of fine-tuning and classification task were the same for both the mouse-Geneformer and human-Geneformer models. The data were randomly split into training (80%) and testing (20%) sets. Accuracy and F1 score were used as evaluation metrics. Cell distribution was visualized using UMAP [30] as described above.

In silico perturbation of human data using mouse-Geneformer

To explore whether mouse-Geneformer can be used for the analyses of other organisms, we analyzed human single-cell RNA-seq data from myocardial infarction cells [40] (European Genome-phenome Archive: EGAS00001006330) and COVID-19 human blood cells [41] (Gene Expression Omnibus (GEO), European Genome-phenome Archive: GSE150728,GSE155673,GSE150861,GSE149689,EGAS00001004571). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets). To perform in silico perturbation experiments on human cells using mouse-Geneformer, human genes in the original datasets were converted to their mouse homologs as described in the previous section. To perform disease typing with mouse-Geneformer, the model was fine-tuned for each disease data for this task. Fine tuning conditions are shown in Table 1. Only models achieving disease subtype classification accuracies above 90.00% on the test data were used for evaluation. Each dataset was randomly split into training (80%) and testing (20%) sets. The in silico perturbation experiments aimed to alter the cell expression profiles of normal cells to resemble abnormal cells and vice versa. For comparison, these human datasets were also analyzed using the original human-Geneformer model.

The number of cells, cell types, and types of diseases used in this experiment with single-cell data of mice are shown in S2 Table.

Human cell type classification using mouse-Geneformer and scGPT in zero-shot mode

We analyzed three single-cell RNA-seq datasets derived from three human organs: breast [37] (Gene Expression Omnibus (GEO): GSE195665), thymus [38] (Gene Expression Omnibus (GEO): GSE144870), and cerebral cortex [39] (Brain Initiative Cell Census Network (BICCN): SCR_016152). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets) and were not included in Genecorpus-30M. To ensure data integrity, we assessed dataset features using Euclidean distance and confirmed that no dataset contained entirely zero values, indicating the absence of data leakage. To perform human cell type classification using mouse-Geneformer, human gene names in the original datasets were converted to their mouse homologs as described in the previous section. To compare the classification performance of the mouse-Geneformer with human-native models, we also employed the original Geneformer (human-Geneformer) and scGPT, both of which were pre-trained on large-scale human scRNA-seq data. The human-Geneformer model was pre-trained using Genecorpus-30M (https://huggingface.co/datasets/ctheodoris/Genecorpus-30M). The human-scGPT model was obtained from the developer’s repository (https://github.com/bowang-lab/scGPT) [44]. Zero-shot analysis was performed for all three models, where cell types were classified using the pre-trained models without any fine-tuning. The same procedures for zero-shot classification were applied across three models. Accuracy and F1 score were used as evaluation metrics to assess classification performance.

Mouse cell type classification using human-Geneformer

To evaluate the applicability of the original human-Geneformer for mouse analysis, we analyzed twelve single-cell RNA-seq datasets derived from twelve organs of mice: urethra and prostate mixed dataset [31] (Gene Expression Omnibus (GEO): GSE145929), embryos [32] (Gene Expression Omnibus (GEO): GSE197353), and kidneys [33] (Gene Expression Omnibus (GSE): GSE190094), tongue [25], thymus [25], mammary gland [25], large intestine [25], limb muscle [25], spleen [25], heart [25], brain [25], kidney [25] (Gene Expression Omnibus (GSE): GSE132042). These datasets were downloaded from CELLxGENE (https://cellxgene.cziscience.com/datasets) and were not included in the mouse-Genecorpus-20M. To confirm the integrity of the data, we compared features using Euclidean distance, and the results indicated that no data was completely zero, confirming the absence of data leakage. To perform mouse cell type classification using Geneformer, mouse genes in the original datasets were converted to their human homologs as described in the previous section. We performed zero-shot classification on these datasets and compared their accuracy. Zero-shot classification involves classifying data without fine-tuning the pretrained model. Accuracy and F1 score were used as evaluation metrics.

Mouse cell type classification using scGPT

We analyzed twelve single-cell RNA-seq datasets derived from twelve mouse organs, using the same datasets described in the section “Mouse cell type classification using human-Geneformer”. To perform mouse cell type classification with scGPT [44], a model trained on human data, mouse gene names in the original datasets were converted to their human homologs, as described in the previous section. Zero-shot classification was then performed on these datasets following the procedures outlined in “Human cell type classification using mouse-Geneformer and scGPT in zero-shot mode”. Classification performance was evaluated using accuracy and F1 score as metrics.

Development of mouse-Geneformer

Theodoris et al. originally developed Geneformer, a context-aware, attention-based deep learning model, which was pretrained on large-scale transcriptome data comprising approximately 30 million human single-cell RNA-seq data [10]. While the original Geneformer was tailored for human transcriptome analysis, our study focuses on creating a mouse version, referred to mouse-Genefomer, from mouse transcriptome data. By leveraging the mouse-Geneformer, we aim to harness the capabilities of large-scale deep learning models to enhance research involving mice, the most extensively studied model organism. The architecture and transfer learning strategy employed in developing mouse-Geneformer are the same as those of the original human Geneformer, with minor modifications. The primary but critical difference lies in the input corpus, which was constructed from mouse single-cell RNA-seq data.

We constructed a large-scale dataset of mouse single-cell RNA-seq data, termed mouse-Genecorpus-20M (Fig 2A). The mouse-Genecorpus-20M comprises approximately 21 million raw single-cell data from healthy mice, encompassing a wide variety of organs and developmental stages. The breakdown of the dataset is shown in Fig 2B,2C and 2D and detailed information is provided in S1 Table. It is noteworthy that mouse-Genecopus include types of samples that are difficult to obtain from humans due to ethical or technical constraints (e.g., embryos). Since mouse-Genecorpus-20M is intended for learning the gene network of normal mice, we excluded single-cell data with high mutation loads, such as cancer cells or immortalized cell lines, that could restructure gene networks. We also omitted low-quality RNA-seq data derived from doublets and damaged cells. After the filtration, single-cell transcriptome data from 20,630,028 cells remained, constituting mouse-Genecorpus-20M (Fig 2A).

The transcriptome of mouse-Genecorpus-20M was modeled using the rank value encoding method and then processed through six transformer encoder units, as employed in the original Geneformer. This process yielded “cell sequences” represented by gene names as tokens. Pretraining was conducted using a masked language model under the conditions shown in Table 1. We made several modifications to the original Geneformer parameters: 1) We changed the activation function from ReLU to SiLU (Sigmoid-weighted Linear Unit, also known as Swish) because SiLU provides smoother gradients than ReLU. It also provides gradients when the input values are less than or equal to 0, and increases the gradient when the input values are greater than or equal to 1. Therefore, it is less prone to the vanishing gradient problem and can improve the performance in deep neural network models. 2) We increased the number of epochs from three to ten to allow the number of training times on the model, reduced learning loss and improved performance. 3) In comparing linear and cosine scheduling modes for the learning rate scheduler, we found that pre-training with the cosine schedule showed better performance by reducing learning loss more effectively. Therefore, we chose the cosine schedule. As a result of these procedures, the mouse-Geneformer was constructed.

Cell type classification: pretraining is effective

To evaluate the effect of pretraining, we compared the cell type classification results of the mouse-Geneformer with and without pretraining, as shown in Table 2. We conducted three experiments using samples from 1) the prostate gland and urethra, 2) embryo that is whole mouse embryo at embryonic age E9.5, and 3) kidney. Our findings indicated that the classification accuracy of the mouse-Geneformer with pretraining was higher than that without pretraining in all cases. The maximum difference in classification accuracy was 8.22% for the embryo data, while the improvements of the other cases were marginal. UMAP visualization of the cell distribution in the embryo data demonstrated that the pretrained mouse-Geneformer depicted more discrete cell clusters compared to the non-pretrained version as shown in Fig 3A and 3B. For example, primitive erythrocytes and cardiac valve cells are distributed among multiple clusters in the non-pretrained mouse-Geneformer classification, whereas in the pretrained version, they formed distinct clusters. These results suggests that pretraining is effective in enhancing mouse-Geneformer, leading to improved classification accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Cell type classification results with and without prior learning.

https://doi.org/10.1371/journal.pgen.1011420.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. UMAP visualization of mouse-Geneformer cell embeddings for embryonic cells.

(A) UMAP plot of cell embeddings generated by mouse-Geneformer without prior learning. (B) UMAP plot of cell embeddings generated by mouse-Geneformer with prior learning. Cell type annotations are derived from the original study [32].

https://doi.org/10.1371/journal.pgen.1011420.g003

Cell type classification: mouse-Geneformer is robust and outperforms conventional methods

We investigated the performance of mouse-Geneformer in cell-type classification across various organs. Twelve experiments were conducted using different organs to compare the accuracy of mouse-Geneformer with conventional methods such as scDeepSort and scVAE (Table 3). Our findings demonstrated that mouse-Geneformer greatly improves the classification accuracy of cell types in all cases, achieving an average accuracy of 96.73%. In comparison, the GNN-based method scDeepSort and Autoencoder-based method scVAE showed accuracy scores of 66.34% and 72.95%, respectively. Notably, mouse-Geneformer constantly achieved classification accuracies exceeding 93%, regardless of target organs or the number of cell types. UMAP visualization of the tongue and limb muscle samples further supported the good performance of mouse-Geneformer in cell-type annotation (Fig 4A and 4B). Each cell type was clearly separated, although the overlap of the distribution of Langerhans cells with epithelial basal cells posed a challenge in the tongue dataset. These results indicate that mouse-Geneformer is more accurate than existing methods and robust across different organs and varying cellular complexity. This suggests that mouse-Geneformer can be applicable effectively to a wide variety of mouse organs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Cell type classification results using the mouse-Geneformer and two conventional methods.

https://doi.org/10.1371/journal.pgen.1011420.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. UMAP visualization of mouse-Geneformer cell embeddings.

(A) UMAP plot of cell embeddings generated by mouse-Geneformer for tongue cells. (B) UMAP plot of cell embeddings generated by mouse-Geneformer for limbs cells. Cell type annotations are derived from the original studies [25].

https://doi.org/10.1371/journal.pgen.1011420.g004

In silico perturbation experiments

We next investigated the applicability of mouse-Geneformer to in silico perturbation experiments for mice. In silico perturbation experiments simulate genetic manipulation experiments on computers, allowing us to mimic gene deletions or activations by editing the gene list within each cell. This enables the prediction of how these mutations will impact the gene network and elucidate the functions of the mutated genes. Notably, this approach provides a powerful method for candidate gene screening, as it allows for the repeated simulation of experiments for multiple target genes of interest, or even all mouse genes, in silico. By comparing the effects of these perturbations, we can identify the most influential genes. Detailed information about in silico perturbation experiment is provided in Methods. We here analyzed three disease models using mouse-Geneformer.

In silico perturbation experiment in diabetic kidney disease.

We analyzed diabetic nephropathy using a set of single-cell transcriptome data collected from disease kidney exhibiting diabetes and from normal kidneys as a control. Detailed information about data and fine turning are provided in Methods. The UMAP visualization of the cell distribution shown in Fig 5A showed a clear separation between normal kidney cells and disease cells.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. UMAP visualization of mouse-Geneformer cell embeddings for cells of disease models.

(A) UMAP plot of cell embeddings generated by mouse-Geneformer for cells derived from three kidney disease models. (B) UMAP plot of cell embeddings generated by mouse-Geneformer for cells derived from normal microglia cells (Cop1 WT) and COP1 knocked-out cells (Cop1 KO).

https://doi.org/10.1371/journal.pgen.1011420.g005

Subsequently, we conducted in silico perturbation experiments by randomly and repeatedly choosing target genes. Detailed information about in silico perturbation experiment is provided in Methods. The analysis revealed that deleting the gene Slc12a3 from normal kidney cells brought the cells closest to those of diabetic kidney disease, with a cosine similarity of 0.018. This in silico outcome is consistent with observations from in vivo experiments, where granular cells of the glomerular apparatus suffered from diabetic kidney disease showed diminished expression of Slc12a3 [33].

Cross-species application of mouse-Geneformer through orthologous gene name conversion

We explored whether mouse-Geneformer could be used for the analyses of other organisms. If successful, this would allow non-model mammals, for which it is difficult, costly, and technically challenging to obtain sufficient single-cell transcriptome data to construct species-specific Geneformer models, to benefit from transfer learning using the mouse gene network model. Given the core metabolic and physiological features are conserved among closely related species, we hypothesize that the core genetic architectures would also be conserved. Therefore, mouse-Geneformer could potentially elucidate a significant portion of genetic architectures beyond mice. Even humans could benefit from mouse-Geneformer, despite the existence of the original Geneformer. This is because mouse-Genecopus can include types of samples that are difficult to obtain from humans due to ethical or technical constraints.

As a proof of concept for the cross-species application of mouse-Geneformer, we investigated its applicability to human transcriptome analysis. The outline of the procedure was as follows: First, to analyze human genes in mouse-Geneformer, each human gene name was converted to mouse ortholog based on the ortholog table. We then fine-tuned the mouse-Geneformer model using human transcriptome data converted to mouse otrholog. Using this fine-tuned model, we conducted in silico perturbation experiments.

Human cell type classification using mouse-Geneformer.

We conducted human cell type classification using human single-cell RNAseq data with mouse-Geneformer. Three experiments were conducted using different human organs. Detailed information about how to convert gene and fine tuning is provided in Methods. For comparison, we also analyzed the human data using original human-Geneformer model.

The cell type classification results are shown in Table 4. The results demonstrated that mouse-Geneformer (with fine-tuning) accurately classified human cell types based on the human transcriptoem data for all cases, achieving accuracy scores of 95.59%, 99.97% and 87.82% for human thymus, cerebral cortex, and breast, respectively. In addition, the classification accuracy of the ortholog-converted mouse-Geneformer for human data was nearly equivalent to that of the original human-Geneformer, with marginal differences ranging from 0.01 to 0.30%. F1 scores were also comparative each other. These data indicate that mouse-Geneformer is effective for cell type classification of human cells and suggests that the Geneformer model can work effectively across species following orthology-based data conversion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Human cell type classification using mouse-Geneformer via ortholog-based gene name conversion, compared to native human models (human-Geneformer and scGPT).

https://doi.org/10.1371/journal.pgen.1011420.t004

Mouse cell type classification using human Geneformer.

Given that the human transcriptome can be effectively analyzed using mouse-Geneformer, we sought to evaluate the reverse scenario: whether the original human Geneformer could classify mouse cell types using mouse single-cell RNA sequencing data after homology-based gene name conversion. The results of this analysis are presented in Table 5. This demonstrated that human Geneformer successfully classified mouse cell types based on mouse transcriptome data across all cases, achieving performance levels comparable to those of mouse-Geneformer. This indicates that a homology-based approach is effective for cross-species applications of the Geneformer architecture between mouse and human. Interestingly, tasks involving cell type classification with a higher number of classes showed reduced performance in the cross-species application of Geneformer. This effect was particularly evident in brain data, which included 15 cell types. This suggests that more complex tasks require native transcriptome data for high-resolution classification, while simpler tasks, such as classification involving fewer cell types, can be effectively analyzed using a cross-species approach. Notably, approximately 15% of mouse genes could not be assigned to human orthologs, likely resulting in the loss of information related to mouse-specific genetic regulation. This limitation may have contributed to the lower performance observed in more complex classification tasks, further underscoring the importance of developing Geneformer models tailored to the target species.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Mouse cell type classification using human-based models (human-Geneformer and scGPT) via ortholog-based gene name conversion.

https://doi.org/10.1371/journal.pgen.1011420.t005