1. Introduction

Machine Learning (ML) solutions based on Neural Networks (NN) are extensively used in multiple domains due to the remarkable results achieved in image rendering [1], business decision-making [2], pharmaceutical [3], bioinformatics [4], cancer research [5], and drug discovery, among many others. However, NN modeling demands considerable infrastructure and computing power for calculations and training with big data in a reasonable amount of time [6].

Cloud computing provides the necessary infrastructure and services to implement, train, and deploy NN models [7, 8]. However, using NNs on third-party infrastructures reveals several data privacy issues during the pre-processing, processing, and post-processing [9]. Third-party infrastructure reduces resource costs and makes it easier to solve tasks but introduces privacy issues of sensitive information processing [10, 11]. The access of the NN to the raw data can create potential privacy risks because data are located in the cloud-shared infrastructure. In general, cloud environments provide cost-savings, flexibility, scalability, and ubiquitous computing resources, but they potentially sacrifice the privacy of information [12]. Their main constraints are insufficient protection of sensitive data processing and the inherently insecure nature of shared resources.

Modern encryption techniques ensure security and are considered the best option to protect stored data and data in transit from an unauthorized third party. However, a decryption process is necessary when the data must be processed or analyzed, falling into the initial problem of data vulnerability. Efficient processing of sensitive data implies beyond an accurate prediction, analysis, or classification; it also implies adequate privacy handling.

Homomorphic Encryption (HE) schemes are solutions to address privacy problems by providing encrypted data computations to the client [13, 14]. Applications with HE provide security and confidentiality to users, even when they are executed on untrusted shared infrastructures. However, HE schemes have limitations concerning the arithmetic operations that can be executed efficiently. They are bound to solve real problems only using additions and multiplications.

Recent advances in HE focus on incorporating HE cryptosystems into NN models. However, redesigning neuron functions and developing HE-friendly non-linear operations like comparison, max/min, sign detection, etc. [15] are open problems.

Activation functions like Rectified Linear Unit (ReLU), Leaky ReLU, Parametric ReLU, Gaussian Error Linear Unit (GELU), Sigmoid, Tangent Hyperbolic (Tanh), Softmax, Swish, Binary Step, etc. are non-polynomial and use operations not supported by HE schemes. Therefore, the efficient implementation of cryptographically computable activation functions becomes imperative for developing a privacy-preserving NN with HE (NN-HE) [16].

Several polynomial approximation methods are used to support homomorphic computation in NN-HE, such as Taylor series, Least-squares, Chebyshev series, Newton, Composite polynomials, etc.

Another important aspect is to allow homomorphic computations in the domain of continuous values. The Cheon-Kim-Kim-Song (CKKS) scheme [17] is designed to perform approximate arithmetic over encrypted real numbers. While the CKKS scheme uses state-of-the-art solutions to support approximations, they are still inefficient in practice, and thus, efficient HE-compliant activation functions remain an open challenge [18].

Several limitations arise in reproducing the behavior of neurons using polynomial approximations in the homomorphic space [19]. The main goal is to control the trade-off between having a non-linear transformation, which is needed by the learning algorithm, and keeping the degree of the polynomials small to make HE feasible [20–22].

In this paper, we design an adaptive NN-HE with improved accuracy and performance. Optimization is supported by polynomial Self-Learning Activation Functions (SLAF). Trainable coefficients of polynomial approximations of NN activation functions are updated during the training together with synaptic weights at each neuron independently to learn problem-specific, dataset-specific, and NN-specific features.

Two Convolutional Neural Network (CNN) models are proposed: CNN-HE-SLAF and CNN-HE-SLAF-R. In the first model, all the activation functions are replaced with SLAFs, and CNN is trained to find weights and coefficients. In the second one, CNN is trained with the original activation function, then the found weights are fixed, activation functions are substituted by polynomials, and CNN is shortly re-trained to adapt SLAF coefficients at each polynomial.

We present two SLAF initialization approaches: SLAF₍₀₎ and SLAF_(P). The SLAF₍₀₎ considers trainable polynomial coefficients initialized to zero. In SLAF_(P), coefficients are initialized by values of known polynomial approximations.

CNN-HE-SLAF and CNN-HE-SLAF-R are non-interactive privacy-preserving models to classify encrypted data. For a detailed assessment of the capabilities of the SLAF approach, we conduct a comprehensive analysis over 1-convolutional (CNN1) and 2-convolutional (CNN2) NNs using both plaintext and ciphertext as inputs from the MNIST dataset.

These results show the remarkable capability of HE-SLAF to yield comparable accuracy to traditional activation functions, indicating that models with trainable polynomial activations have the same representative ability as their non-polynomial counterparts. HE-SLAF low-degree polynomials achieve the same accuracy as a non-polynomial ReLU activation function over non-homomorphic models in the plaintext space.

On the other hand, we show that NN-HE-SLAF models with trainable three-degree polynomial activations improve accuracy over known NN-HE solutions in the literature. The training of polynomial coefficients of each neuron independently allows the exploration of the space of all the activations to construct better homomorphically evaluated approximations.

The content of the paper is structured as follows. Section 2 introduces privacy-preserving techniques, delving into homomorphic encryption schemes. Section 3 presents the primitives of the CKKS scheme. Section 4 describes privacy-preserving neural networks with homomorphic encryption. Section 5 discusses state-of-the-art NN-HE models that polynomially approximate cryptographically non-computable activation functions. Section 6 presents related work. Section 7 introduces homomorphic self-learning activation functions. Section 8 provides a brief overview of the case study. Section 9 defines the experimental setup. The experimental results are presented in Section 10. Finally, we discuss our solutions and conclude in Section 11.

To facilitate understanding of the described ideas, we summarize the main HE, NN, and SLAF terminology in the Appendix. S1 Table shows the terms used in the paper, and S2 Table presents the main acronyms.

2. Privacy-preserving techniques

Privacy-preserving methods are emerging as a crucial consideration in cloud environments. Secure Multi-Party Computation (MPC) [23, 24], Federated Learning (FL) [25, 26], Differential Privacy (DP) [27], Functional Encryption (FE) [28], and Homomorphic Encryption (HE) are representative approaches in this domain.

3. CKKS homomorphic encryption scheme

CKKS is a lattice-based HE scheme whose security is based on the hardness of the Ring Learning with Errors (RLWE) problem. The CKKS scheme performs approximate arithmetic over encrypted real (complex) numbers. Given messages m₁ and m₂, it allows secure computing encryptions of approximate values of m₁+m₂ and m₁m₂ with a prefixed precision. The main characteristic of CKKS is that it treats the inserted noise of the RLWE problem as part of an error occurring during approximate computation.

In CKKS, after the selection of parameters such as degree N of the polynomial ring and a modulo q, real (complex) numbers are encoded into plaintext polynomials. The secret key is a randomly generated polynomial. The public key is created using the secret key, chosen parameters, and some randomness. The plaintext polynomial is encrypted using the public key and noise to increase security. After operations, rescaling is performed to reduce noise and preserve the ciphertext. The ciphertext is decrypted using the secret key to obtain an approximated version of the original plaintext polynomial that is decoded into a real or complex resulting number.

Let the polynomial ring for a power-of-two N be R = ℤ[X]/(X^N+1), and R_q = ℤ_q[X]/(X^N+1) be a modulo-q residue ring of R. The polynomial coefficients of R_q are bounded by the modulo q and the degree of polynomials by X^N+1. Let L be a level parameter that indicates the maximum multiplicative depth. Modulo q = q₀∙q₁∙…∙q_L is defined as the product of co-prime moduli q₀,…q_L, where q_ℓ = 2^ℓ∙q₀ for 1≤ℓ≤L.

The distribution χ_key = HW(h) outputs a polynomial from R_q of {±} -coefficient having h non-zero coefficients, where HW(h) denotes the set of signed binary vectors in {±}^N whose Hamming weight is h∈Z₊. χ_enc and χ_err denote discrete Gaussian distributions with some predefined standard deviation. U(R_q) refers to a uniform distribution over the ring R_q. Moreover, for , then , where ⌊∙⌉ returns the nearest integer of a real-number input, rounding upwards in case of a tie [17].

CKKS encodes real values using what is known as canonical embedding , where a plaintext vector is transformed into and then rounded to an integer-coefficient polynomial using a scaling factor Δ, i.e., . The parameter Δ affects the accuracy of the computation in CKKS.

Let us discuss the main primitives of the CKKS scheme: KeyGen, Encrypt, Decrypt, Add, Mult, Rot, and Resc.

• KeyGen (N, q, L)→sk, pk, ek. Sets secret key as sk = (1,s), where s←χ_key. Sets public key as , where b = −as+e (mod q_L), , and e←χ_err. The evaluation key is set as , where , and e′←χ_err.
• . For a plaintext vector of real (complex) numbers , ist encodes , and provides the ciphertext , where v←χ_enc and e₀, e₁←χ_err.
• : For a ciphertext , decodes the message as , and outputs a plaintext vector .
• Add(c₁, c₂)→c₊: Add two ciphertexts . It returns ciphertext .
• : For two ciphertexts , let . It returns ciphertext .
• Rot(c,r)→c′: Given an encryption c of , outputs c′ that encrypts the left-rotated vector by r positions.
• Since every is scaled, the plaintext of c←Mult(c₁, c₂) is Δ∙m₁m₂, which results in an exponential growth of plaintexts. To deal with such a problem, CKKS introduces the so-called rescaling procedure:
• Resc(c)→c′: For a ciphertext , outputs .

For more detailed information and additional considerations on the CKKS scheme, including the correctness and security analysis, refer to [17].

4. Privacy-preserving NN-HE

This section focuses on understanding the construction and evaluation of the privacy-preserving NN model with HE (NN-HE).

NN-HE is a natural extension of NN models. It consists of applying HE to the network inputs and homomorphically propagating the data across the network. NN-HE implementation has several restrictions because not all the functions for processing NN have homomorphic counterparts. So, the internal NN structure must be modified to perform secure processing. One way is to substitute non-homomorphic operations with appropriate approximations.

The main challenge in constructing NN-HE models is designing the efficient homomorphic processing of inner network functions. In the following sections, we discuss this complex problem, considering homomorphic neurons and homomorphic layers.

5. State-of-the-art NN-HE models

This section presents the state-of-the-art privacy-preserving NN-HE models, highlighting their implementation details, latency (Lat), and accuracy (Acc).

Table 1 summarizes the essential characteristics of the NN-HEs. The “Layers” column corresponds to the number of convolutional, dense, activation, and pooling layers incorporated within each NN-HE architecture. GPU (Graphic Processing Unit) and 2-arch (2-architecture) columns denote specific aspects of the model’s operation. GPU indicates that the model takes advantage of hardware acceleration methodologies through GPUs. 2-architecture indicates the use of a dual-architecture strategy by collapsing adjacent linear layers during the evaluation process.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. State-of-the-art privacy-preserving neural networks with homomorphic encryption.

https://doi.org/10.1371/journal.pone.0306420.t001

Dowlin et al. [54] propose CryptoNets to address the challenge of achieving a blind non-interactive classification. The approach uses the leveled YASHE [35] scheme for NN inputs and propagates signals across the network homomorphically.

CryptoNets contains two convolutional layers and two dense layers. It replaces the activation function with the square function, the lowest-degree non-linear polynomial function. Its performance is limited due to using the square function as an activation function, computational overhead, and the insecure YASHE scheme [61]. Several subsequent works in the literature focus on improving its constraints.

Chabanne et al. [47] present the premises of a privacy-preserving Deep Neural Network (DNN) evaluated homomorphically. To improve CryptoNets’ performance, the authors polynomially approximate the ReLU activation function and use a BN layer before each activation layer; the normalization layer limits the need for an accurate approximation to a small part of ℝ around the neighborhood of zero point. However, although the approach indicates the use of the BGV HE scheme, it does not provide any results on the encrypted data.

Chou et al. [44] introduce pruning and quantization methods that leverage sparse representations in the underlying BFV cryptosystem to accelerate CryptoNets inference. Faster-CryptoNets iteratively removes and clusters the model parameters without affecting accuracy. They derive a quantized minimax approximation for standard activation functions that achieves maximally-sparse encodings and optimizes approximation error [62]. Experiments over MNIST show that the NN-HE model maintains competitive accuracy and achieves a significant speedup over previous methods. On CIFAR-10, the proposed model takes more than 6 hours and achieves an accuracy of 76.7%. Also, the authors propose a transfer learning approach with DP to address real-world tasks like medical imaging applications.

Bourse et al. [46] implement a Discretized Neural Network (DiNN) for inference over encrypted data, whose complexity is linear in the network size. Weights and inputs are discretized into elements in [−1,1] with a threshold value equal to 128: any pixel whose value is smaller than the threshold is mapped to −1, otherwise to +1. Although this method can homomorphically evaluate NNs of any depth, each neuron output is refreshed through the bootstrapping procedure featured by the Fully Homomorphic Encryption scheme over the Torus (TFHE), resulting in high overhead and low accuracy. They construct two simple FHE-DiNN models with one hidden (dense) layer containing 30 and 100 neurons with a security level of 80 bits.

Sanyal et al. [55] propose a privacy-preserving Binary Neural Network (BNN) with a TFHE scheme built upon the FHE-DiNN approach. The BNN evaluates every arithmetic operation as a composition of binary gates; it performs homomorphic multiplication by applying the logical operator XNOR and homomorphic addition by summing the number of 1s. TAPAS implements sparsification techniques and algorithmic tools to speed up and parallelize ciphertext computation. Every operation consists of many bootstrapping procedures, is therefore inherently immune to noise, and has no architectural restrictions. Nonetheless, the model is slower than any previous work: one prediction on MNIST takes 37 hours on a single machine and 2.41 hours using a cluster of 16 machines with 16 cores each.

Van Elsloo et al. [56] propose SEALion, an extensible framework built upon the CryptoNets approach for implementing privacy-preserving NN-HE models. SEALion performs an automatic encryption parameter selection for BFV, which side-steps complex implementation details for ad-hoc homomorphic solutions. The proposed approach improves both latency and encrypted inference by sparsifying the activation functions with the L₀ norm.

Hesamifard et al. [42] develop an NN-HE based on the model architecture of [47, 54]. CryptoDL replaces standard non-linear activation functions with HE-friendly low-degree polynomial approximations within a specific error range. ReLU, Sigmoid, and Tangent Hyperbolic (Tanh) functions are approximated using Chebyshev and Taylor series. Additionally, the authors implement a scaled-up version of average pooling, which calculates the summation of values without dividing by the number of values. Building on this approach, Liao et al. [63] implement a CryptoDL-based NN-HE model over encrypted sensor data; the authors approximate Tanh, ReLU, and Swish functions to generate cryptographically computable activations.

Brutzkus et al. [57] propose Low-Latency (Lo-La) CryptoNets to improve latency and memory usage over its predecessors. While CryptoNets encodes each image’s feature as a separate message, Lo-La encrypts entire layers as a single message with the BFV scheme and uses different matrix-vector multiplication implementations throughout the inference. These technical modifications allowed the evaluation of the same model as CryptoNets in a pair of seconds and the private inference over larger datasets such as CIFAR-10. However, the scheme is still dependent on the message dimension and, therefore, impractical for applications with high-dimension data. Additionally, they present the premises of using transfer learning to solve HE limitations, such as message size and noise growth.

Boemer et al. [58] introduce nGraph-HE, an extension of the Intel nGraph compiler to deploy NN models on homomorphically encrypted data. nGraph-HE incorporates a privacy-preserving abstraction layer, enabling HE-aware optimizations at compile- and run-time. The framework supports BFV and CKKS cryptosystems without bootstrapping. For experimental analysis, they use CKKS on fixed-precision numbers as the underlying encryption scheme over an Xeon Platinum 8180 platform with 112 CPUs and 376GB of RAM.

Jiang et al. [59] propose a method to perform arithmetic operations over homomorphically encrypted matrices. They introduce the E2DM framework to demonstrate the applicability of the proposed matrix computation approach for the secure evaluation of NN-HE models. They also considered packing multiple matrices (images) into a single ciphertext and computing on them in a SIMD manner, yielding better-amortized performance; their CryptoNets implementation on MNIST takes 1.69 seconds to compute ten likelihoods of 64 input images simultaneously. E2DM reports the same accuracy for plaintext and ciphertext inference, considering CKKS as the underlying cryptosystem with a security level of 80 bits; however, it remains unclear how a square activation function in an approximate homomorphic computation achieves a zero-precision loss.

Badawi et al. [19] present an efficient NN-HE for image classification on GPUs. The authors use a GPU-based BFV implementation as an underlying engine to perform encrypted computations. The heterogeneous GPU cluster consists of three Nvidia Tesla P100 cards with 3584 CUDA cores and one Nvidia V100 with 5120 cores. The MNIST-HCNN model is five layers deep: two sequential convolutional layers with their respective square activation function followed by one dense layer. For CIFAR-10, they provide an 11-layer network: three convolutional layers, each one followed by a square activation and HE-friendly pooling layer, and finally, two dense layers. The HCNN model significantly accelerates the classification process while maintaining security and accuracy; it classifies MNIST in 2% of the time CryptoNets takes.

Falcetta and Roveri [60] develop a privacy-preserving LeNet-1 model variant by incorporating some of the best practices in the NN-HE field, such as replacing non-HE-compliant activation functions with low-degree polynomials, max pooling with average pooling, etc. They highlight the security parameter selection and the approximation of non-linear layers as the primary considerations in constructing secure and accurate NN-HEs.

The main challenge toward deploying HE schemes in real-world applications lies in overcoming the high computational costs associated with these cryptosystems. Computing over ciphertexts with state-of-the-art schemes such as CKKS represents a slowdown of 4-6 orders of magnitude compared to performing the same computations on unencrypted data [64]. In recent years, several researchers have supported the evaluation of complex DNN models over encrypted data. We remark that the highest state-of-the-art DNN accuracy on MNIST and CIFAR-10 in the unencrypted domain is 99.79% and 96.53%, respectively.

Lee et al. [50] implement a privacy-preserving ResNet-20 model using the Residue Number System CKKS (RNS-CKKS) scheme with bootstrapping. It is based on the theoretical contribution presented in [65], where authors propose a precise polynomial approximation technique for the ReLU and max-pooling functions using a composition of minimax approximate polynomials of a small degree. Lee et al. [50] homomorphically evaluate the NN-HE with 383 CIFAR-10 images and plaintext model parameters. Nonetheless, the proposed model takes about 3 hours to infer one image due to the more than a thousand bootstrapping functions implemented and the high-degree minimax composite polynomials used for approximating ReLU and Softmax functions. Although such latency is high for practical use, it represents a significant step toward a DNN with FHE using bootstrapping.

Pure MPC and hybrid MPC-HE-based solutions, such as presented in [43, 66–73], are outside the scope of the current study and, therefore, are omitted. For instance, in an MPC-HE approach like [43], the client carries out the bootstrapping process directly.

Privacy-preserving NN models have come a long way from the first introduction to actual solutions. Several companies, such as IBM [74] and Microsoft [75], offer HE services and incorporate HE into their commercial privacy-aware solutions. However, there is still a long way to go and daunting challenges. We are observing a developing research area in constant growth, with many potential applications and significant privacy benefits.

Table 2 summarizes the polynomial approximation methods used in the state-of-the-art NN-HEs to address cryptographically non-computable activation functions: Square function, Taylor series, Least-squares, Chebyshev polynomials, Newton, and Composite polynomials, implemented in CryptoNets, Chabanne-NN, Faster-CryptoNets, SEALion, Liao-CNN, CryptoDL, Lo-La, nGraph-HE, E2DM, HCNN, LeNet-HE, RNS-CKKS-NN, and our CNN-HE-SLAF model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Summary of activation function approximations used in state-of-the-art privacy-preserving neural network models with homomorphic encryption.

https://doi.org/10.1371/journal.pone.0306420.t002

6. Related work

In recent years, there has been a growing interest among the research community in exploring non-homomorphic activation functions that are trained during the learning process. In this section, we review the latest advances in this emerging field. First, we examine the taxonomy of trainable activations. Later, we highlight related works in the field of adaptative polynomial activation functions.

The idea behind a trainable activation is to search for an adequate function shape using knowledge from the training data [76]. There are three main adaptive activation families [77]: parametrized activations, ensemble-based activations, and trainable polynomial activations [78].

The first family refers to those parametrized activations such as PReLU [79], PELU [80], RePU [81], Swish [82], and Syncular [83], which are based on conventional activation functions and fine-tuned through the incorporation of one or more trainable parameters.

The second family includes activations based on ensemble methods [84–86]. These approaches comprise a collection of basic functions, which can consist of standard activations, trainable functions, or a combination of both, and a model that defines the linear combination of such a basis.

The third family refers to those polynomial activation functions with trainable coefficients, where the polynomial coefficients are learned together with the network parameters using the backpropagation learning algorithm and gradient descent optimization method.

The first and second families involve non-polynomial functions and use operations not supported by HE cryptosystems. Therefore, we focus on the third family of trainable activation functions, which consists of HE-friendly polynomials with coefficients that can be effectively learned and adjusted.

Hou et al. [78] propose a piecewise polynomial Smooth Adaptive Activation Function (SAAF) on non-homomorphic CNN models for regression tasks. The parameters of SAAF control the shape of activation functions, where these parameters are trained along with other NN model parameters. The authors argue that applying SAAFs in the regression (second-to-last) layer of a NN can significantly decrease the bias of the regression NN and avoid overfitting by simply regularizing the model. They report results over eight real-world datasets while incurring a minimal increase of less than 1% in the total number of model parameters.

Goyal et al. [87] introduce Self-Learning Activation Functions (SLAF) over non-homomorphic NN models. SLAFs are learned during training and can approximate most existing activation functions. SLAFs are a weighted sum of Taylor polynomials that can accurately approximate the optimal activation. According to the authors, non-homomorphic NN models with SLAF improve the accuracy concerning equivalent architectures with conventional activation functions on standard benchmarking datasets.

In the third family, special attention is given to trainable rational activation functions. These functions are defined as the ratio of two polynomials P(x) and Q(x), both with trainable coefficients and respective degrees m and n. Rational functions are better suited than polynomials to approximate ReLU [88]. They demonstrate competitive accuracy on non-homomorphic NN models for image classification [89–91]. Additionally, trainable Padé activations can approximate standard hand-designed activations and learn new activations with compact representations [89]. Nevertheless, their inherent inverse operation makes it infeasible for encrypted computation.

Several studies indicate that NNs with polynomial activations have the same representational power as their non-polynomial counterparts [92, 93]. Additionally, these polynomial activations enhance latency without compromising accuracy [62].

While multiple works in the NN-HE field have explored using polynomial activation functions to enable an efficient homomorphic classification (see Table 2), trainable polynomial activation functions remain unexplored. We propose the privacy-preserving SLAF-based NN-HE (NN-HE-SLAF) model, introducing a novel approach in the encrypted classification domain by considering self-learning polynomial approximations to generate cryptographically computable activation functions. We use a customized polynomial with trainable coefficients as an activation function. In contrast to the traditional use of a single fixed-interval approximation of a specific activation function for all neurons, NN-HE-SLAF models generate customized polynomials for each homomorphic neuron independently, yielding better accuracy and performance.

7. Self-learning activation functions

In this section, we present SLAF models for designing a privacy-preserving NN-HE. Given the definition of a homomorphic neuron of the NN-HE model by Eq (5), we approximate the non-linear activation function by a polynomial at each neuron independently with trainable coefficients as (6) where denote the trainable coefficients of the polynomial at neuron k.

The central concept of the SLAF training process is to find an adequate mapping from the input to the output space based on modifying NN weights and polynomial coefficients in each neuron separately. NNs can learn and generalize more information based on training examples. The process is an analogy of the human brain evolution during a person’s experience.

In addition to synaptic weights, the training process aims to find the SLAF polynomial coefficients. Therefore, NN-HE-SLAF activations are adapted to the specific problem, structure of the network, its parameters, and the dataset.

Fig 1 shows the general structure of a homomorphic neuron with SLAF.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Homomorphic neuron k with a self-learning activation function .

https://doi.org/10.1371/journal.pone.0306420.g001

We study two approaches for polynomial initialization: SLAF₍₀₎ and SLAF_(P). SLAF₍₀₎ considers n+1 trainable polynomial coefficients a₀, a₁,…,a_n initialized by zero. In the SLAF_(P) approach, coefficients are initialized by coefficients obtained by known approximation methods. While SLAF₍₀₎ starts searching from the “zero point” of the solution space, SLAF_(P) explores a search space close to the best approximation solution.

Let us consider an example of SLAF_(P) using as a polynomial the nine-degree approximation f_c of the sign(x) function performed by the composition of minimax approximate polynomials [21].

Fig 2 shows the approximation of the sign(x) function given by (7)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Nine-degree polynomial approximation of the sign function generated by composite polynomials of the minimax approximate polynomials.

https://doi.org/10.1371/journal.pone.0306420.g002

SLAF_(P) polynomial is initialized by ten coefficients γ₀ = 0, γ₁ = 3.114, γ₂ = 0, γ₃ = −6.645, γ₄ = 0, γ₅ = 8.851, γ₆ = 0, γ₇ = −5.485, γ₈ = 0, and γ₉ = 1.169. The training results in new polynomial coefficients γ₀, γ₁,…,γ_n, that is (8) where denote the trained SLAF_(P) coefficients at neuron k. All neurons initially have identical γ₀, γ₁,…,γ_n, but post-training exhibits unique coefficients for each neuron k.

Considering that an NN mimics the behavior of the biological brain, SLAF refers to the physiological capacity of the human brain called plasticity. This ability allows the brain to adapt to new learning methods by conditioning and remodeling how neurons connect. Our NN-HE-SLAF models not only enable the identification of underlying relationships in the information but also facilitate the determination of specific activation functions for a given dataset that guide the learning.

To better understand the capabilities of SLAF, we prove its feasibility in approximating a continuous activation function.

Theorem 1. An NN-HE-SLAF can approximate any continuous activation function f if its input domain is bounded to a desired error as a function of the SLAF degree.

8. Case study

This section presents details of the proposed models. To provide a fair comparison, we implement NNs with a similar architecture to the ones used in state-of-the-art models, such as CryptoNets [54], Faster-CryptoNets [44], and HCNN [19] (see Table 1).

We study two SLAF models based on CNN: CNN-HE-SLAF with a single training phase and CNN-HE-SLAF-R with training and re-training.

In the first model, we replace all activation functions with three-degree SLAF₍₀₎ or SLAF_(P) and train CNN to find weights and coefficients. We denote these models as CNN-SLAF₍₀₎ and CNN-SLAF_(P).

In the second model, we train CNN with original non-homomorphic activations function, lock weights, substitute activation functions by polynomials SLAF₍₀₎ and SLAF_(P), and (R)e-train CNN to adapt coefficients. We denote these models as CNN-SLAF₍₀₎-R and CNN-SLAF_(P)-R, respectively.

Fig 3 illustrates the training and testing procedures for the proposed privacy-preserving CNN-HE-SLAF, CNN-HE-SLAF₍₀₎-R, and CNN-HE-SLAF_(P)-R models.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Training and testing privacy-preserving CNN-HE models with self-learning activation functions.

https://doi.org/10.1371/journal.pone.0306420.g003

We study two architectures of CNNs: 1-convolutional (CNN1) and 2-convolutional (CNN2). Figs 4 and 5 illustrate their structures.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. CNN1 with a single convolutional layer.

The yellow element denotes a convolutional layer. Circular elements after Conv1 and FC1 denote activation functions. Blue elements represent dense layers.

https://doi.org/10.1371/journal.pone.0306420.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. CNN2, a CryptoNets-based network with two convolutional layers.

Red elements denote average pooling layers. Circular elements after Conv1 and FC1 denote activation functions. Blue elements represent fully connected layers.

https://doi.org/10.1371/journal.pone.0306420.g005

CNN1, with a single convolutional layer and two fully connected layers, is a variant of Lo-La small NN [57]. In contrast to the Lo-La approach, CNN1 incorporates approximated activations after the convolutional and the first fully connected layers. CNN1 details are presented in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. CNN1 architecture.

https://doi.org/10.1371/journal.pone.0306420.t003

CNN2 is a CryptoNets-based network architecture with two convolutional layers. To provide a direct comparison concerning state-of-the-art solutions, we adopt the architecture proposed in [44], which in turn is a variant of CryptoNets, CryptoDL, Lo-La, HCNN, SEALion, and others. It incorporates a BN layer before each activation to encourage the activation inputs to fit in the approximated interval.

The BN layer transforms them into normal distribution inputs with zero mean and unit variance, which reduces the overall approximation error, prevents the generalization of higher feature values and indirectly provides smaller weights for homomorphic processing. The details of such an architecture are presented in Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. CNN2 architecture.

https://doi.org/10.1371/journal.pone.0306420.t004

For CNN1 and CNN2 with SLAF₍₀₎-R and SLAF_(P)-R, we use the original ReLU function in the first training. Then, we lock trained weights, replace ReLU activation functions with SLAF₍₀₎ or SLAF_(P) approximations, and re-train CNN to learn customized polynomial approximation coefficients. For re-training, we choose considerably fewer epochs than for training (see Section 9.3: Developing tools).

Several state-of-the-art models, as presented in [40, 50, 65], have proposed high-degree approximations to obtain an accuracy similar to ReLU. They state that a precise approximation of the ReLU function is necessary to evaluate models applied to plaintext or encrypted inputs.

To construct a better ReLU approximation for each neuron independently, we collect data during the first training to find intervals of activation inputs. Then, we generate a polynomial approximation of sign(x) function on found symmetric intervals [−B, B], where B denotes the supremum norm of the activation inputs of each neuron. Due ReLU(x) is equivalent to and |x| = x∙sign(x), we approximate (see Fig 6). A large B has the advantage of enlarging the input ranges of the polynomial activation function at the cost of a higher approximation error.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Polynomial approximation of sign function on the interval [−B, B] with B = 1 and their corresponding ReLU approximation based on the equivalence ReLU(x)≈[x+x∙sign(x)]/2.

https://doi.org/10.1371/journal.pone.0306420.g006

For approximation methods such as Chebyshev polynomials, which are orthogonal in [−1, 1], the interval is extended from [−1, 1] to [−B, B] by p(x/B) for B>1. Similarly, for Newton-Schulz, which approximates the ±1 roots of by the iterative computation of , converging to sign(x) in [−1, 1].

Fig 6 shows the polynomial approximation of the sign function by a Fourier sequence (Fig 6A) and its respective ReLU approximation based on the equivalence (Fig 6B).

We can see that this simple transformation using sign(x) approximation allows better ReLU approximation.

9. Experimental setup

This section describes the evaluation method and dataset characteristics and defines the experimental setup.

9.1. Security settings

The ciphertext size, scheme performance, multiplicative depth, and security level depend on the security parameter settings. We adopt the security settings specified in the HE standard [94]. Table 5 shows such security settings for the CKKS scheme. The security level λ = 128 bits guarantees that an adversary needs to perform 2¹²⁸ elementary operations to break the scheme with a probability one.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Security settings for the CKKS scheme.

https://doi.org/10.1371/journal.pone.0306420.t005

10. Experimental analysis

To compare traditional CNNs and solutions with SLAF, we analyze their differences and similarities in the training and testing phases. We provide the performance evaluation of non-homomorphic CNN1 and CNN2 on plaintext inputs and privacy-preserving CNN1-HE and CNN2-HE on ciphertext inputs. For CNN-HE models, both inputs and weights are encrypted before testing. We perform the following experiments:

First, we train and test non-homomorphic models with the ReLU activation function. The training accuracy of CNN1 and CNN2 with ReLU over plaintext is 99.562% and 99.748%, respectively. The testing accuracy of CNN1 and CNN2 with ReLU over plaintext is 98.56% and 99.38%, respectively.

Second, we evaluate the performance of CNNs and CNN-HEs using a single training to determine both weights and SLAF coefficients.

Third, we analyze CNNs and CNN-HEs by training with ReLU activations. Then weights are fixed, ReLU is replaced by SLAF, and the model is re-trained to adapt SLAF coefficients at each polynomial independently.

Fourth, we evaluate CNNs and CNN-HEs using ReLU in training. We determine the bounded intervals of each neuron’s activation inputs and approximate sign(x) on these intervals using known polynomial approximations. Then, we apply sign(x)→ReLU(x) transformations and replace ReLU with such approximations. Finally, we perform a re-training process to optimize SLAF coefficients.

10.1 CNN1

We provide the performance evaluation of non-homomorphic CNN1 and CNN1-HE without and with SLAF.

Table 6 presents the latency (Lat) and accuracy (Acc) of the CNN1 model using SLAF₍₀₎, SLAF₍₀₎-R, different polynomials for SLAF_(P)-R, and Lo-La. Results for plaintext and ciphertext inputs are reported in the CNN1 and CNN1-HE columns, respectively. Latency corresponds to the time required to process a single classification request. ρ denotes the generalization gap, which measures the difference between the training and testing accuracies (%). Lo-La corresponds to the original implementation using specifications, parameters, and characteristics proposed in the corresponding paper [57].

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. CNN1 performance.

https://doi.org/10.1371/journal.pone.0306420.t006

We can see that CNN1-SLAF_(P)-R with three-degree Chebyshev polynomials achieves the same accuracy of 98.56% as the non-homomorphic CNN1 with the original ReLU activation function. CNN1-SLAF₍₀₎-R has the best accuracy of 98.22% in CNN-HE models. That is 1.3% better than 96.92% HE Lo-La.

CNN1 and CNN1-HE latencies illustrate the significant overhead of NN-HE models compared with their unencrypted analogous. For example, computing CNN1-SLAF₍₀₎-R over ciphertexts results in a slowdown of approximately 240 times compared to performing the same computations on unencrypted data.

To evaluate the effectiveness of SLAF_(P) approach over traditional polynomial approximation methods, we test CNN1-HE using known approximations to replace the ReLU activation: Least-squares, Chebyshev polynomials, Newton-Raphson, and Minimax composition. Polynomial approximations to an activation function usually consider fixed approximation intervals regardless of the particular dataset and problem. CNN1-SLAF_(P)-R with adaptive intervals for each neuron increases the model accuracy.

With a Chebyshev series, CNN1-HE-SLAF_(P)-R achieves an accuracy of 97.96%, showing an improvement of 11.85% compared to the 86.11% of the Chebyshev series that approximates ReLU without SLAF_(P). With composite minimax polynomials, it achieves an accuracy of 97.86%, improving by 18.55% over 79.31% of Composite minimax polynomials without SLAF_(P). With Least-squares, it provides an accuracy of 97.31%, improving by 6.01% over 91.30% of Least-squares without SLAF_(P). With Newton-Raphson, it has 97.56%, improving by 33.27% over 64.29% of Newton-Raphson without SLAF_(P).

10.2. CNN2

Since the combination of linear operations is linear and there is no intervening non-linearity between the first pooling layer and the first fully-connected layer of CNN2, the hidden layers between them can be collapsed into a single linear layer for the testing process, as depicted in Fig 7.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Collapsing CNN2 adjacent linear layers.

https://doi.org/10.1371/journal.pone.0306420.g007

This linear transformation (10) can be accurately reproduced (up to some numerical rounding error) by the dense layer FC_new. Collapsed models exhibit the same accuracy as the original models.

Tables 7 and 8 present the performance of the non-homomorphic and homomorphic CNN2, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Non-homomorphic CNN2 performance.

https://doi.org/10.1371/journal.pone.0306420.t007

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Performance of privacy-preserving CNN2 model with homomorphic encryption.

https://doi.org/10.1371/journal.pone.0306420.t008

CNN2-SLAF₍₀₎ and CNN2-SLAF_(P) models using low-degree polynomials achieve the same accuracy of 99.38% as the standard ReLU activation function over non-homomorphic models in the plaintext space.

These results show the remarkable capability of SLAF variants to yield comparable accuracy to traditional non-linear functions. It confirms that NN models with SLAF have the same representative ability as their non-polynomial counterparts.

Now, let us evaluate the performance of CNN2-HE-SLAF models in comparison to the state-of-the-art models in the ciphertext space, such as CryptoNets, Faster-CryptoNets, CryptoDL, Lo-La, nGraph-HE, E2DM, and HCNN. These models adopt the 2-convolutional CryptoNets architecture and implement diverse polynomial approximation methods of ReLU.

As shown in Table 8, the CNN2-HE-SLAF₍₀₎-R (CNN2-HE-SLAF_(P)-R) model with re-training achieves an accuracy of 99.21% (99.05%), showing improvements between 0.21% (0.05%) and 1.11% (0.95%) over CryptoNets-based NN-HE solutions, 0.21% (0.05%) over 99% of HCNN, 0.26% (0.10%) over 98.95% of CryptoNets, 0.51% (0.35%) over 98.70% of Faster-CryptoNets, 0.69% (0.53%) over 98.52% of CryptoDL, and 1.11% (0.95%) over 98.10% of E2DM.

CNN2-HE-SLAF₍₀₎-R (CNN2-HE-SLAF_(P)-R) also outperforms models with a similar 2-convolutional architecture, such as LeNet-HE and SEALion (see Table 1), showing accuracy improvements of 1.03% (0.87%) over 98.18% of LeNet-HE and 0.30% (0.14%) over 98.91% of SEALion, respectively. Furthermore, the CNN2-HE-SLAF₍₀₎-R (CNN2-HE-SLAF_(P)-R) improves accuracy by 0.61% (0.45%) compared to 98.60% of the TAPAS model, which utilizes a 19-layer architecture with six convolutional layers.

Comparing performance, we demonstrate that using trainable three-degree polynomial activations, CNN2-HE-SLAF₍₀₎-R classifies MNIST 6.26 times faster than CryptoNets with better accuracy (99.21% vs 98.95%).

11. Discussion and conclusions

Common encryption algorithms such as Advanced Encryption Standard (AES) successfully protect stored and transmitted data, preventing third parties and the public from reading them. However, data processing needs a decryption, falling into the problem of data vulnerability. Data have to be searched, computed and analyzed securely. HE is a solution to address the processing of confidential data while it remains encrypted. However, HE performs only addition and multiplication operations efficiently on ciphertexts, increasing impracticality due to the significant computational overhead. This overhead is the biggest challenge for widespread adoption. Optimization techniques are required to improve performance and usability in various domains such as cloud computing, machine learning, healthcare, finance, voting, etc.

NN-HE is a rapidly growing research area with many potential applications and significant benefits. The main direction of the current NN-HE development is to increase accuracy and efficiency. While protecting sensitive data’s privacy is essential, they must also offer acceptable accuracy and time complexity.

The CNN-HE-SLAF solutions contribute to these two goals and existing knowledge by designing an adaptive NN-HE.

This paper proposes methods of CNN-HE optimization by Self-Learning Activation Functions (SLAF). We theoretically prove that NNs with SLAF can approximate any continuous activation function to any desired error as a function of the degree of SLAF. In contrast to the traditional NN-HE approaches, where the activation functions of all neurons are replaced by a single (best) polynomial approximation, NN-HE-SLAF generates customized polynomials for each homomorphic neuron independently. It yields better accuracy and performance.

Two convolutional NN-HE models are proposed: CNN-HE-SLAF and CNN-HE-SLAF-R.

• CNN-HE-SLAF replaces all activation functions with SLAF, and it is trained to find synaptic weights and coefficients together. Then, CNN is encrypted homomorphically. We denote these models as CNN-HE-SLAF₍₀₎ if polynomial coefficients are initialized to zero and CNN-HE-SLAF_(P) if coefficients are initialized to a known polynomial approximation.
• CNN-HE-SLAF-R is trained with the original activation function, weights are locked, activation functions are substituted by SLAF₍₀₎ or SLAF_(P), and it is re-trained to adjust coefficients. We denote these models as CNN-HE-SLAF₍₀₎-R and CNN-HE-SLAF_(P)-R, respectively. For re-training, we choose considerably fewer epochs than for training.
• CNN-HE-SLAF leverages the approximation capabilities of neural networks by optimizing polynomial approximations of individual homomorphic neurons, customizing them for a given problem, dataset, network structure, parameters, and approximation intervals.
• CNN-HE-SLAF₍₀₎-R and CNN-HE-SLAF_(P)-R achieve the same accuracy of 99.38% as a non-polynomial non-homomorphic ReLU applying better approximation of sign(x) on the found intervals and sign(x)→ReLU(x) transformations.
• CNN-HE-SLAF achieves an accuracy of 99.21%, improving the state-of-the-art CNN-HE solutions on the MNIST optical character recognition benchmark dataset.
• CNN2-HE-SLAF collapses the hidden layers between the first pooling layer and the first fully connected layer into a single linear layer with batch normalization to reduce latency while keeping the same accuracy.
• CNN2-HE-SLAF₍₀₎-R increases the performance (6.26 times faster) than the state-of-the-art CNN-HE CryptoNets on the MNIST.
• CNN-HE-SLAF demonstrates the feasibility of applying trained polynomials to generate HE-compliant and problem-specific activation functions, enabling efficient, accurate, and privacy-preserving CNN-HE solutions.

These benefits are possible at the expense of increased CNN-HE model complexity. SLAF adds additional trainable parameters, such as coefficients of polynomial approximations of activation functions, making training time longer. So, it is important to find an adequate initialization point and avoid potential overfitting. In future work, it is essential to apply the state-of-the-art practical methods of NN hardware acceleration to SLAF techniques: highly parallel CPU, GPU, Tensor Processing Unit (TPU), Field Programmable Gate Array (FPGA) or Application-Specific Integrated Circuit (ASIC), etc.

Moreover, it is important to validate the applicability and efficiency of the self-learning approach to other ML models, such as logistic regression, and to other CNN-HE layers, e.g., pooling, batch, etc. Furthermore, SLAF performance can be optimized by introducing a dropout mechanism, whereby coefficients below a specific ∈-threshold are nullified or converted to zero.

Additionally, while CNN-HE-SLAF models can generate homomorphically computable activation functions, these privacy-preserving models inherit both the strengths and weaknesses of current NN models, where even advanced models trained with state-of-the-art optimization techniques do not provide a practical solution for every problem. The SLAF approach does not guarantee the best approximation for a given function and norm. In future work, it is imperative to enhance the approximation and model accuracy by considering SLAF as a weighted sum of orthogonal polynomials instead of monomials.

Another relevant line of future work involves improving the internal structure of the CNN-HE to increase the performance of homomorphic self-adaptive learning using different HE schemes and other state-of-the-art strategies, e.g., incorporating RNS HE variants to decompose the encrypted inputs into several parts and propagating them homomorphically and independently across the model. In addition to the highly parallelizable property, an RNS-based HE scheme allows for smaller encryption parameters without compromising the scheme security level, i.e., a reduced latency, keeping security and accuracy.

Finally, the list of potential applications is broad due to the strongly applied nature in real-world problems and the significant privacy benefits of CNN-HE-SLAF solutions. In future work, it is essential to explore the applicability of proposed models for sensitive domains such as medical image classification.

Supporting information