1.1 Related work

To address the issue of systematic differences in multi-source gene expression data integration, current research mainly employs Meta-analysis (MA) and Data Merging (DM) approaches [5,6]. DM merges all samples in a unique dataset and subsequently performs analyses, while MA aggregates information from different samples at different stages of the analysis, for example, at the beginning by merging all samples or at the end by combining all the results of the separate analysis of specific samples [7,8]. Both methods increase the sample size by pooling all samples but fail to adjust for sample-specific heterogeneity, especially when focusing on the characteristics of specific target data. The heterogeneity among sources not only includes systematic variations caused by experimental conditions and measurement errors but also biological differences, such as variations in gene expression across different human tissue samples [9,10]. A unified model trained on all samples may exhibit bias and be unable to represent the true characteristics of the target sample of interest.

To address the challenge posed by such heterogeneity of source datasets and lack of representation of target data, transfer learning has emerged as a promising approach, aiming to transfer useful information from related but different source datasets to enhance the learning performance of the target data [11, 12]. When the target data is limited and the source data is large but biased, Bouveyron and Jacques (2010) proposed adaptive models to learn the target parameters through a linear transformation of the source model’s parameters, showing significant improvements over prediction from the target data only [13]. However, this approach gives very low adaption freedom due to the limiting of the number of transformation matrix elements learned from the target data to control the extent of transfer learning. Bastani (2021) proposed a more flexible two-step transfer learning estimator for high-dimensional linear models with a single informative auxiliary study [14]. By estimating the estimator from the source data and debiasing it using the target data, significant improvements in prediction performance for target data can be achieved. Furthermore, Li et al. (2022) further considered the estimation and prediction of a high-dimensional linear regression in the setting of two-step transfer learning with multiple source data and proposed a transferable source data detection algorithm to avoid negative transfer [15]. However, the transfer learning estimators proposed by Bastani (2021) and Li et al. (2022) both rely on quadratic optimization based on linear regression with normal distributed error, which is sensitive to heavy-tail and outliers that are common in practice [16, 17].

1.2 Our contributions

In this paper, building on the method proposed by Li et al. (2022) [15], we propose a transfer learning framework under high-dimensional linear models with t-distributed error, aimed at enhancing the robustness of transfer learning under conditions of heavy-tailed distributions and outliers, thereby improving the estimation and prediction of target data. The heavy-tailed characteristics of the t-distribution make it more robust to outliers compared to the normal distribution [18–22]. Different degrees of freedom parameters of the t-distribution can be estimated to adapt to source data with varying outliers or heavy-tailed error distributions [23]. By introducing t-distributed errors, we use the expectation-maximization (EM) algorithm to optimize the penalized likelihood function for estimating regression parameters. In the oracle case with known transferable source datasets, a three-step transfer learning algorithm is established, which combines the information from the target dataset and transferable source datasets, and adjusts their differences simultaneously. To avoid including non-informative sources, we propose to select transferable sources based on cross-validation, which ensures that the data integration will increase the prediction performance.

We not only achieve robustness improvements methodologically but demonstrate that our proposed method can transfer useful information from source datasets to target data and is robust to heavy tails and outliers through extensive simulations and application. Compared to other candidate methods, it exhibits higher accuracy in estimation, prediction, and variable selection.

2.1 Penalized linear regression with t-distributed error

Regression analysis is one of the most widely used statistical methods to understand the relationship between a response and a set of predictors [24]. For subject i = 1,…, n, let Y_i be a response and be p predictors. The linear regression model can be written as (1) where β = (β¹,…, β^p)^T is the coefficient vector of interest and ε_i is the random error. In this paper, we assume that the dimension p is high but β is sparse, that is, only a small subset of predictors is relevant to the response. In traditional linear regression, it is assumed that ε_i follows a normal distribution. However, in genomics data, this assumption may not always be met due to outliers or heavy-tailed distributions, then the ordinary least squares estimates may become biased or inefficient. In this paper, we consider the robust linear regression which assumes that ε_i follow a t-distribution, that is, ε_i ~ t(0, σ², ν) with location 0, scale parameter σ² and degrees of freedom ν [15]. The t-distribution has heavier tails than the normal distribution, making it less sensitive to outliers and more suitable for dealing with genomics data that exhibit non-normal errors [18].

The penalized log-likelihood function of θ = (β^T, σ², ν)^T for model (1) is given by , where with (2) where Γ(∙) enotes the gamma function, λ‖β‖₁ is the l₁-penalty term for variable selection [25], and λ is a regularization parameter that determines the amount of shrinkage. The maximum likelihood estimator is defined as the maximizer of . However, directly optimizing the above objective function is computationally complex. In the following, we introduce an alternative representation of [2], which is based on the fact that the t-distribution can be written as a gamma-normal hierarchical form [26] as: (3) where N(∙) denotes the normal distribution and Γ(∙,∙) denotes the gamma distribution. The τ_i is a latent variable, then the gamma-normal hierarchical form leads to expectation-maximization (EM) algorithm [27] implementations for maximum likelihood estimation of the unknown parameters.

Given the data with for i = 1,…, n, and the latent variable vector τ = (τ₁,…, τ_n)^T. According to the gamma-normal hierarchical form (3), the penalized log-likelihood function can be re-expressed as , where with (4)

In the E-step, given the parameter values θ^(u−1) at the (u − 1)-th iteration, the expectation of the complete log-likelihood is given as

In the M-step, the above objective function can be optimized by Newton-Raphson-type algorithm. The detailed procedure is presented in Algorithm 1.

Algorithm 1 EM algorithm

Input: Data D and initial value θ⁽⁰⁾.

At the u-th iteration, given θ^(u−1) from the (u − 1)-th iteration:

 E-step. Compute the expectation and .

 M-step. Update unknown parameters as θ^(u) = argmax_θ Q(θ, θ^(u−1)).

Iterate E-step and M-step, until ‖θ^(u) − θ^(u−1)‖₁ < 10⁻⁶.

Output: .

2.2 Target dataset, source dataset, and transferable dataset

Suppose there is a target dataset and S independent source datasets , where is the index set of subjects belonging to target dataset with sample size n₀, and (s = 1,…, S) is the index set of subjects belonging to the s-th source dataset with sample size n_s. The regression models corresponding to the target dataset and the source datasets are: (5) where β⁽⁰⁾ is the coefficient of the target dataset, and β^(s) (s = 1,…, S) is the coefficient of the s-th source dataset.

Our goal is to transfer useful information from the source datasets to the target dataset, to improve the estimation accuracy of β⁽⁰⁾. The index set of the transferable source datasets is denoted as , which is a subset of {1,…, S}. Intuitively, the closer β^(s) of the s-th source dataset is to β⁽⁰⁾ of the target dataset, the more transferable that source dataset may be. Deeply, if we combine the s-th source dataset with the target dataset, the estimation or prediction performance is improved relative to using the target dataset alone, then we consider that source dataset to be transferable.

2.3 Transfer learning with known transferable dataset

We first consider that index set for the transferable source dataset is known. Motivated by Li et al. (2023) [28], we propose a three-step transfer learning framework for linear regression model [5]. In the first step, we fit regression model using Algorithm 1 in each transferable source dataset , and obtain . In the second step, we use the target data to measure the difference between β^(s) () and β⁽⁰⁾, which is denoted as . In the third step, we combine the target dataset and transferable source datasets to jointly estimate β^(TL), and use to adjust the differences. The details of the algorithm are shown in Algorithm 2, in which the regularization parameters , λ_δ, and λ_β are selected by cross validation.

Algorithm 2 Transfer learning with known transferable dataset

Input: Target dataset , source datasets for , and index set for the transferable dataset.

Step 1. Fit the regression model in each transferable source dataset. For , compute

Step 2. Measure the differences using target dataset. For , compute

Step 3: Joint estimation using target dataset and transferable source datasets, and adjusting the differences. Compute

The above optimization problems can be achieved by running EM Algorithm 1.

Output: .

2.4 Detecting the transferable dataset

We now consider how to detect the index set for the transferable dataset. As previously mentioned, if we combine the s-th source dataset with the target dataset, the estimation or prediction performance is improved relative to using the target dataset alone, then we consider that source dataset to be transferable. Inspired by Eaton et al. (2008) and Zhang and Zhu (2022) [29, 30], we propose a cross-validation framework for detecting the transferable dataset. We randomly split the subjects in target dataset into two disjoint equal-size groups , where is the training set and is the validation set. We first fit regression model using Algorithm 1 in training set , and obtain . For a given s = 1,…, S, then we perform transfer learning using Algorithm 1 in with the s-th source dataset, and obtain . We evaluate the performance of and on the validation set, by computing . The indicates that the transfer of the s-th source dataset makes the prediction performance increase, so we consider . If but not exceeding for a given threshold , we also consider . The details of the algorithm are shown in Algorithm 3.

Algorithm 3 Transfer learning with unknown transferable dataset

Input: Target dataset , source dataset .

1. Detecting transferable dataset

Randomly split into two disjoint equal-size groups .

 (1) Compute .

 (2) For s = 1,…, S, compute by running the Algorithm 2 with .

 (3) For s = 1,…, S, compute .

Let .

2. Run Algorithm 2 with target dataset , and detected transferable source dataset for to obtaion .

Output:

3.1 Simulation

We consider a target dataset with n₀ = 150, and S = 10 source datasets with n₁ = … n_S = 100. For s = 0,…, S, we set p = 500 and the predictors is drawn from the multivariate normal distribution with mean zero and covariance matrix Σ(ρ), where Σ(ρ) is the first-order autoregressive correlation structure with ρ = 0.7. The response is generated as

For the coefficient β⁽⁰⁾ = (β¹⁽⁰⁾,…, β^p(0))^T corresponding to target dataset, we set k = 16, β^j(0) = 0.5 for j = 1,…, k and β^j(0) = 0 otherwise. We randomly select source datasets as the transferable datasets with the corresponding coefficients set to be where H^(s) is a random subset of {17,…, p} with |H^(s)| = 50 and h ∈ {10,20} controls the heterogeneity levels between target dataset and source datasets.

To evaluate the robustness of the proposed method, we consider four distributions of the error. For s = 0,…, S, we set (i) normal distribution as ε^(s) ~ N(0, 1); (ii) t-distribution as ε^(s) ~ t(0, 1, 5); (iii) contaminated normal distribution as ε^(s) ~ 0.9N(0, 1) + 0.1N(0, 10), which generates outliers; (iv) skew t-distribution as ε^(s) ~ St(0, 1, 5, skewness = 1).

We conduct 100 replications. In each replication, to evaluate the estimation accuracy, we calculate the estimation error ; to evaluate the prediction accuracy, we randomly split the target dataset into five folds, using four folds to train and the remaining fold to calculate the mean prediction error with ; to evaluate the accuracy of the variable selection, we calculate the proportion of true predictors among predictors selected by l₁-regularization (defined as precision) and the proportion of selected true predictors among all true predictors (defined as recall).

The threshold in Algorithm 3 is set to be 0.1. Results corresponding to or 0.15 are available in S1–S4 Fig, which shows that the estimates are insensitive to a small range of .

3.2 Application to the prediction of gene expression

Gene expression data can be used to explore the gene-gene interaction and gene regulation for a better understanding of molecular processes of pathogenesis and the purpose of prediction of disease. For instance, discovering gene expression patterns that are characteristic of a certain disease to differentiate between healthy and diseased individuals.

In this section, we consider Primary Familial Brain Calcification (PFBC) which is a hereditary neurodegenerative disease characterized by progressive bilateral cerebral calcifications, accompanied by various symptoms such as dystonia, ataxia, Parkinson’s disease, dementia, depression, headaches, and epilepsy [31]. Currently, the exact etiology of PFBC remains unclear. Recent studies indicate that variations in the JAM2 gene result in decreased JAM2 mRNA expression and loss of JAM2 protein in patient fibroblasts, consistent with the loss-of-function mechanism [32, 33]. JAM2 is a protein-coding gene located on chromosome 21, and the encoded protein is a cell adhesion molecule expressed in various tissues and cell types, influencing cell migration, adhesion, and interactions [34]. The expression pattern of JAM2 may be regulated under physiological and pathological conditions. Therefore, obtaining the expression levels of JAM2 is of great significance for the diagnosis of PFBC. Given the technical challenges and ethical concerns associated with brain tissue sequencing, constructing model to predict the expression levels of JAM2 in target brain tissues holds potential clinical diagnostic value for PFBC. The objective of this application is to train a prediction model of JAM2 expression levels in target brain tissues, transferring useful information from other tissues or cell types to improve the prediction of JAM2 expression levels in the target brain tissue.

We evaluate the predictive performance of our proposed algorithm using tissue-specific gene expression data from the Genotype-Tissue Expression (GTEx) version 8 dataset, which consists of 838 donors and 17,382 samples from 54 non-diseased tissue types (S1 Table). In the analysis of this study, we used 49 tissues or cell lines that had at least 70 individuals, including a total of 17,329 samples from 838 donors. Based on previously published literature and publicly available lists of central nervous system (CNS)-related genes, the CNS-related genes were assembled as MODULE 137, including 546 genes as well as 1,632 additional genes that are significantly enriched in the same experiments as the genes of the module. These genes may participate in similar biological pathways or regulatory networks, working together in the central nervous system to perform specific functions. Therefore, they provide an important foundation for our study of JAM2 genes in PFBC. We consider 13 brain tissues as target tissues and the remaining 36 tissues as source tissues. The average sample size of target tissues and source tissues was 203 and 408, respectively. After excluding missing values, the final predictors included 1292 genes.

To compare the performance of the proposed method, we conduct Trans-PNLR and Trans-PtLR models in each tissue based on the association of the JAM2 gene and other CNS-related genes and identify the informative tissues to transfer to improve the performance of the target model in each brain tissue. We also conduct Naive Trans-PNLR and Naive Trans-PtLR to understand the total information level of all source tissues. We also conduct PNLR and PtLR to evaluate the prediction performance using only the target tissue, without transforming information from other source tissues. The response variable is the JAM2 expression level and covariates are the expression levels of other genes included in MODULE 137. For each model, we randomly split the target dataset into five folds, using four folds to train and the remaining fold to calculate the mean prediction error with . The data is standardized before analysis.

4.1 Simulation results

Figs 1 and 2 and Table 1 show the estimation error, relative prediction error, and variable selection precision and recall of candidate methods under different heterogeneity levels (h = 10 or 20), different numbers of transferable source datasets (), and different distributions of the error (ε^(s) follows normal distribution (N), t-distribution (t), contaminated normal distribution (CN), and skew t-distribution (St)).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Estimation error of candidate methods with a threshold of 0.1.

The estimation error is evaluated under different heterogeneity levels (h = 10 or 20), different numbers of transferable source datasets (), and different distributions of the error (ε^(s) follows normal distribution (N), t-distribution (t), contaminated normal distribution (CN), and skew t-distribution (St)).

https://doi.org/10.1371/journal.pcbi.1012739.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Relative prediction error of candidate methods relative to the PNLR with a threshold of 0.1.

The relative prediction error is evaluated under different heterogeneity levels (h = 10 or 20), different numbers of transferable source datasets (), and different distributions of the error (ε^(s) follows normal distribution (N), t-distribution (t), contaminated normal distribution (CN), and skew t-distribution (St)). We randomly split the target dataset into five folds, using four folds to train and the remaining fold to calculate the mean prediction error with .

https://doi.org/10.1371/journal.pcbi.1012739.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Precision and recall (in parentheses) of candidate methods under different heterogeneity levels (h = 10 or 20), different numbers of transferable source datasets (), and different distributions of the error (ε^(s) follows normal distribution (N), t-distribution (t), contaminated normal distribution (CN), and skew t-distribution (St)).

https://doi.org/10.1371/journal.pcbi.1012739.t001

For the normal distributed (N) error, Trans-PNLR and Trans-PtLR perform better than PNLR and PtLR, which indicates that transfer learning can help us extract additional information from the source datasets to improve the accuracy of estimation and prediction on the target dataset. As the numbers of transferable datasets increases, the accuracy of estimation and prediction becomes higher. Trans-PNLR and Trans-PtLR also perform better than Naive Trans-PNLR and Naive Trans-PtLR, especially when is relatively small. The probable reason is that when the number of transferable source datasets is small, transferring all the source datasets will result in negative transfer. Therefore, it is necessary to adopt the Algorithm 3 to detect the transferable dataset (S2 Table). We also observe that when the heterogeneity levels between target dataset and source datasets are smaller (h = 10), the accuracy of estimation and prediction for Trans-PNLR and Trans-PtLR is higher.

For the comparison between Trans-PNLR and Trans-PtLR, when the error is generated from normal distribution that contains no outliers, the performances of these two methods are quite similar. When the error is generated from t-distribution with heavy tail, Trans-PtLR has a smaller estimation error and prediction error than Trans-PNLR. When the error is generated from contaminated normal distributions, Trans-PtLR is less sensitive to potential outliers than Trans-PNLR. When the error is generated from skewed t-distribution which is skewed and heavy-tailed, Trans-PtLR also achieved better estimation and prediction performance than Trans-PNLR. As for genomics data, outliers and non-normal errors are likely to be encountered, so our proposed Trans-PtLR may be more suitable for application. For single regression coefficients with non-zero true effect, the estimation error exhibits the similar trend as Fig 1, while for regression coefficients with zero true effect, the estimation errors approach zero due to being compressed to zero by the l₁-penalty (see S5 and S6 Figs). We also explored the impact of estimation performance when sample size of target and sources vary. After correctly identifying transferable datasets, even if the sample size of the sources is small, the estimation performance will still not be worse than using only the target data (S7 Fig).

We can find from Table 1 that the recall of Trans-PtLR and Trans-PNLR is close to 1, but in some scenarios, such as when ε^(s) follows a t-distribution (t) or skew t-distribution (St), for h = 10 or 20 and , the recall of Trans-PtLR is higher than that of Trans-PNLR. This indicates that our method almost never misses important predictors. We also found that in most cases, the Precision of Trans-PtLR is higher than that of Trans-PNLR. This suggests that while Trans-PNLR may select more irrelevant variables, Trans-PtLR more accurately excludes unimportant predictors, thereby enhancing precision.

4.2 Predicting the gene expression

Before the analysis, we first fit a normal linear regression model for all tissues and conducted model diagnoses, then, we took several target tissues and source tissues as examples, plotted the distribution of error and compared it to the normal distribution. As shown in Fig 3, the distribution of error is heavy-tailed or may contain outliers, indicating that fitting the tissue-specific gene expression data with a normal linear regression may not be appropriate. It is reasonable to assume the error term following t distributions to reduce the effects of the potential outliers. The transferable sources detected by all methods and the sample size of all transferable sources are available in S3 and S4 Tables.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Distribution of error by normal linear regression for six tissues compared with normal distribution.

https://doi.org/10.1371/journal.pcbi.1012739.g003

Fig 4 shows the relative prediction error of five models (PtLR, Naive Trans-PNLR, Naive Trans-PtLR, Trans-PNLR, and Trans-PtLR) relative to the PNLR using CNS gene expression levels to predict JAM2 gene expression levels in 13 brain tissues. The results show that transfer learning methods significantly reduce prediction errors in most scenarios. This highlights the practical advantages of integrating useful information from source tissues.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Relative prediction error of PtLR, Naive Trans-PNLR, Naive Trans-PtLR, Trans-PNLR, and Trans-PtLR relative to the PNLR.

We randomly split the data in target tissues into five folds, using four folds to train and the remaining fold to calculate the mean prediction error with .

https://doi.org/10.1371/journal.pcbi.1012739.g004

It is noteworthy that the model’s performance is poorer when transferable source datasets are not pre-identified (as in Naive Trans PNLR and Naive Trans PtLR). This emphasizes the importance of using the proposed transferable source detection algorithm, which effectively identifies valuable source information and improves predictive performance, demonstrating its practical applicability in complex data integration tasks.

Furthermore, the results suggest that transfer learning within the t-linear regression (Trans-PtLR) framework outperforms the normal regression (Trans-PNLR) framework. Transfer learning through Trans-PNLR reduced the average prediction error by 18.3%, while Trans-PtLR further reduced this by 33.6% (with Trans-PtLR averaging a 15.3% reduction in prediction error compared to Trans-PNLR). The Wilcoxon signed-rank test shows that the difference in prediction errors reduced by these two methods has a P-value of 0.0002. This significant improvement indicates that modeling with heavy-tailed t-distributed errors can better handle outliers and enhance the robustness of real-world datasets, particularly in gene expression data where heavy tails and noise are common.