Ensemble learning method for the prediction of new bioactive molecules

Pharmacologically active molecules can provide remedies for a range of different illnesses and infections. Therefore, the search for such bioactive molecules has been an enduring mission. As such, there is a need to employ a more suitable, reliable, and robust classification method for enhancing the prediction of the existence of new bioactive molecules. In this paper, we adopt a recently developed combination of different boosting methods (Adaboost) for the prediction of new bioactive molecules. We conducted the research experiments utilizing the widely used MDL Drug Data Report (MDDR) database. The proposed boosting method generated better results than other machine learning methods. This finding suggests that the method is suitable for inclusion among the in silico tools for use in cheminformatics, computational chemistry and molecular biology.


Background
Virtual screening, which has its roots in cheminformatics, computational chemistry and structural biology [1], is the computation of the similarity between the target (reference structure) and each molecule in a database [2]. It is an established method for the discovery of new biologically active molecules [3]. It is a process whereby, through molecular modeling, each chemical agent in a database is docked into the binding region of each macro molecule target [4]. Docking is the process whereby the best fit for each agent in the binding region of the macromolecular target is calculated [4]. Schneider and Bohm [5] provided a survey of fast automated docking methods, and a detailed study on the calculation of an optimal box size for molecular docking against predicted binding pockets was carried out by Feinstein and Brylinski [6]. Wang et al. [7] extensively reviewed grapheme-based glucose sensors spanning from the period of 2008 to 2015. Huang et al. [8] worked on Drosophila, where Piwi-piRNA was the guiding epigenetic mechanism to target sites. Their work provided insight into the process involved in the recruitment of epigenetic factors to their target sites. Meanwhile, Marinov et al. [9] investigated the work of Huang et al. and discovered that their genome-wide result was not supported by their dataset. The work of Lin et al. [10] confirmed Marinov  the genomic site was not discovered and reaffirmed that the genome RNA polymerase II distribution is influenced by Piwi. Watanabe and Lin reviewed piRNA with respect to some biological processes, and their detailed work can be found in [11]. The science of processing bioactive molecules in important fields, such as lead discovery and compound optimization, has evolved in recent years [12]. The literature has extensively discussed different virtual screening techniques [13][14][15][16] and activity prediction approaches [17]. For example, Burden and Winkler [18] introduced the Quantitative Structure-Activity Relationship (QSAR) method as a solution to large datasets and then proposed back propagation (BP) after comparing this method with Multiple Linear Regression (MLR), Principal Component Regression (PCR) and Partial Least Squares (PLS) methods. They applied QSAR to massive data sets derived from combinatorial chemistry and High Throughput Screening (HTS). QSAR involves the prediction of the biological activity of a compound from a vectoral representation of molecular structure [19]. QSAR has been successfully utilized with regards to many drugs and agro-chemical design problems. In Burden and Winkler's study [18], more information concerning the challenges of QSAR was outlined, and Rogers and Hopfinger [20] solved the problem of building QSAR and Quantity Structure-Property Relationship (QSPR) models using Genetic Function Approximation (GFA). In their work, they disclosed that the secret of the GFA lies in the creation and use of multiple models, rather than the utilization of a single method. Additionally, the unclear QSAR between plant-derived flavones and their inhibiting effects on aurora B kinase (aurB) was established [21].
In the relevant literature, several similarity search methods have been proposed [22]. Sheridan and Kearsley [22] justified the need for many chemical similarity search methods in the early discovery of leads in a drug discovery project. Detailed reviews of chemical similarity searching and virtual screening can be found in Shneider and Bohm [5] and Willett, Barnard and Downs [23].
In this modern era of computational technological advancement, the adoption of machine learning algorithms for the prediction of molecules has been explored. Willet et al. [24] applied the Binary Kernel Discrimination (BKD) approach for the determination of ion channel activity. BKD was introduced and compared with merged similarity search by Harper [25]. Liu et al. [26] developed a model based on the Support Vector Machine, which can be used to automatically produce predictors. This model has a four-in-one function of extracting features, selecting parameters, training models, and cross-validation. This model improves the prediction rate.
A recent survey on the success (to date) and possible opportunities with regards to ligandbased virtual screening in machine learning was performed by Lavecchia [27]. The successes include the development of a large-scale machine learning data protocol, in the work of George et al. [28]; machine learning algorithms in multidimensional analysis of classification performance of compounds, Kurczab and Bojarski [29]; the Naive Bayesian classifier, Kurczab, Smusz and Bojarski [15], Bender et al. [30], and Glick et al. [31]; the Bayesian belief network, Abdo et al. [17], Nidhi et al. [32], and Xia et al. [33]; Support vector machines, Bruce et al. [19] and Buchwald, Ritter and Kramer [34]; Binary kernel discrimination, Willett et al. [24] and Reynolds and Sternberg [25]; the C5 (decision tree), Cao et al. [35]; and Investigational Novel Drug Discovery by Example (INDDEX TM ), Reynolds and Sterberg [16].
Krasowski and Ekins [36] addressed the challenges faced in correctly detecting and identifying a molecule intake into a class. They utilized cheminformatics to determine the cross reactivity of designer drugs to their available immunoassay (procedure for detecting or measuring specific proteins or other substances through their properties as antigens) [36].
Stumpfe and Bajorath's study [37] focuses on the practical applications, calculation, and appropriate domain of ligand-based virtual screening. Sherhod et al. [38] generated structural fragmented descriptors by applying a contrast pattern tree mining algorithm. The pattern forms hierarchical clusters of compounds that represent different classes of chemicals. This method was able to identify common toxic features and their classes. Takigawa and Mamitsuka [39] further elaborated on this idea and the procedures for mining frequent sub-graphs for compounds with molecular graphs and chemical compounds.
Smusz et al. [40] adapted virtual screening for their work on the discovery of two structurally new 5-HT 6 R ligands, and Métivier et al. [41] worked on the discovery of structural alerts. In recent research, clustering algorithms have also been used in cheminformatics to discover drugs. A detailed study [42] compares popular clustering techniques, namely, k-means, bisecting k-means and ward clustering. The applications of clustering include QSAR analysis, High Throughput Screening (HTS), and Absorption, Distribution, Metabolism, Elimination and Toxicity (ADMET) prediction [42]. Meanwhile, Pires et al. [43] proposed a novel technique, called pkCSM, to develop predictive models for toxicity properties and small-molecule pharmacokinetics using graph-based signatures [43].
Ensembles have proven to be suitable in improving the performance of a prediction model since they utilize the ability of more than one classifier. They have been used to identify DNAbinding proteins [44] and Piwi-Interacting RNAs [45].
The purpose of our research is to enhance the prediction of bioactive molecules using the boosting algorithm ensemble AdaboostM1 in conjunction with Bagging, Jrip, PART, Random Forest, REPTree and J48 as nominal classifiers. We also compared the performances of the boosting algorithm with a support vector machine classifier called LibSVM (LSVM) [17,46].

Data sets
Bioactive molecules from both natural products and synthetic compounds are precious sources that provide us with the necessary tools to create new drugs to cure diseases [17]. Molecular fingerprints are representations of chemical structures initially designed to support chemical database substructure searching. Subsequently, their use had been for analysis tasks, such as similarity searching, clustering, and classification. extended connectivity fingerprints (ECFPs) is a recently developed fingerprint methodology specifically designed to identify molecular features significant to molecular activity [47].
Three datasets from ECFP_4 standard molecular descriptors, which were used in previous studies, were used for this study. These datasets were retrieved from the MDDR database. The datasets consist of 8294, 5083, and 8568 instances for DS1, DS2, and DS3, respectively, as shown in Tables 1-3. The quality of prediction was based on these datasets and the validation of the classification of molecules was based on the structure-activity relationship.
The three datasets were pre-processed on the work bench via the following filters: unsupervised, attributes, and Numeric to Nominal. DS 1 contains eleven normal activity classes, DS2 contains ten homogenous (average) activity classes, and DS 2 contains ten heterogeneous activity classes. Tables 1-3 show activity index, activity class, active molecules and pairwise similarity (mean). The active molecules are the number of molecules or peptides belonging to the class and the diversity of classes. The diversity of the class is computed as the mean pairwise Tanimoto similarity score calculated across all pairs of molecules/peptides in the class using ECFP_4.

Ensemble learning technique
The employment of AdaboostM1 has been discussed in the literature, see for instance [48,1]. It is a boosting machine learning algorithm [49] that works with another classifier (called the nominal classifier). It works successfully when the nominal classifier in question (also referred to as weak learner) can achieve at least 50% accuracy on its own [49].
AdaboostM1 is an ensemble learning technique and the most well-known of the boosting family of algorithms. The algorithm sequentially trains models, with a new model trained at each round. At the end of each round, misclassified examples are identified and their emphasis is increased in a new training set, which is then fed into the next round and processed to train a new model [50]. The Waikato Environment for Knowledge Analysis (WEKA) software, which is cross-platform software with various machine learning algorithms written in Java, was used to carry out the study. AdaboostM1 is shown in Algorithm 1 (below).
weak learning algorithm weakLearn integer T specifying number of iterations where Z t is a normalisation constant (chosen so that D t+1 will be a distribution).

Output
The final hypothesis:

Experimental design
The need to have a known drug that is classifiable to a specific biological molecular structure is a central part of computational chemistry [51]. In this experiment, we used the extended-connectivity fingerprints (ECFP4) developed by SciTegic [32]. The ECFP4 of MDDR (MDL Drug Data Report) [52] implementation in the test cases is used in this study. Discovering the optimal parameters for a classifier was a time-consuming task. WEKA-Workbench offers the possibility of automatically finding the best possible setup for the LSVM classifier. The values of 1.0, 0.1, and 0.001 were given to the Cost, Gamma and Epsilon parameters, respectively, while the default values available in WEKA-Workbench were used for the other parameters. In this study, six AdaBoost ensemble classifiers were applied, including AdaBoostM1+Bagging (Ada_Bag), AdaBoostM1+Jrip (Ada_Jrip), AdaBoostM1+J48 (Ada_J48), AdaBoostM1+PART (Ada_PART), AdaBoostM1+RandomForest (Ada_RF), and AdaBoostM1+REPTree (Ada_RT). Subsequently, a ten-fold cross-validation was carried out, and the results were evaluated using sensitivity, specificity, and area under the curve (AUC) measurements.
All experiments were conducted using a personal computer with an Intel1 Core ™ i7-4790 CPU 3.60 GHz processor, with 16 GB RAM, and a 64-bit operating system. There are some required settings in the configuration of WEKA to increase the heap size of the memory in the "RunWeka.ini" file under the parameter named "maxheap" with the value of "4096M". This action supports the processing of the large amount of MDDR datasets being used (the original value was "1024M").
To validate the performance of each classifier, we used the confusion matrix of the classification results as a measure to compute all the evaluation parameters. The percentage of correctly classified instances from the 10-fold cross validation was used as the measure for the model. In cross validation, the parameter value of 10 was used as the default value. This result suggests that the data set is divided into 10 folds; one fold was used for testing, and the rest were used for training. This process was repeated 10 times so that all folds were used as a test fold once. The error rate is calculated by computing the average of the 10-fold errors.
The area under the receiver operating characteristic curve (AUC), specificity, sensitivity and accuracy were used as the machine learning evaluation methods. These methods are widely used as quality criteria to quantify performance. They are defined as follow:

Results and discussion
Tables 4-6 display the sensitivity measures (the true positive rates). A number of the AdaBoost ensemble classifiers exhibited the best performance and outperformed the existing best classifier in the discovery of novel drugs where 2 (Ada_Bag and Ada_RF) out of 6 AdaBoost classifiers (Table 4 -DS 1) outperformed the existing best classifier (LSVM). Table 5 (with DS2) shows that 3 (Ada_Bag, Ada_J48, and Ada_RT) out of 6 AdaBoost classifiers surpassed the LSVM classifier. However, Table 6 (with DS3) illustrates that only 1 (Ada_Bag) out of 6 AdaBoost classifiers surpassed the LSVM classifier. Tables 7-9 show the specificity measures (the true negative rates), which also demonstrate that a number of AdaBoost classifiers offered the best performance and surpassed the existing best classifier in the discovery of novel drugs, where 2 (Ada_Bag and Ada_RF) out of 6 Ada-Boost classifiers (Table 7 -DS1) outperformed the existing best classifier (LSVM).
Tables 10-12 display the AUC measures, which also shows that a number of the AdaBoost classifiers offered the best performance and surpassed the existing best classifier in the  discovery of novel drugs, where 2 (Ada_Bag and Ada_RF) out of 6 AdaBoost classifiers (Table 10 -DS1) outperformed the existing best classifier (LSVM). Furthermore, Table 11 (with DS2) illustrates that 4 (Ada_Bag, Ada_J48, Ada_RF and Ada_RT) out of 6 AdaBoost classifiers outperformed the LSVM classifier. Table 12 (with DS3) illustrates that there was 1 (Ada_Bag) out of 6 AdaBoost classifiers that surpassed the LSVM classifier for AUC measurements.
From the results illustrated in Tables 4-12, for all three measures (sensitivity, specificity and AUC), it can be seen that in most cases the AdaBoost ensemble classifiers provided better outcomes when compared with LSVM; these ensemble methods built a sequence of base models where each model was constructed based on the performance of the previous model on the training set. In other words, by suitably combining the results of a set of base classifiers, the performance obtained was better than that of any base classifier.
This study used a cut-off value of 0.05 for the significance level (p-value). The p-value was considered significant and capable of providing an overall ranking if p<0.05 and the critical value for chi-square χ 2 at p = 0.05 for 6 degrees of freedom was 12.59. The degrees of freedom are equal to the total number of algorithms minus 1. In this study, there were 7 algorithms applied (LSVM + six AdaBoost ensemble classifiers), leading to 6 degrees of freedom. The results of Kendall's W tests are presented in Tables 13-15 (below). Ensemble learning method for prediction of new bioactive molecules The analysis in Table 13 shows that Kendall's coefficients (for DS1 and DS3 using the sensitivity measure) were significant (p<0.05, χ 2 >12.59) and that the performance of Ada_Bag significantly outperformed all of the other methods. The overall rankings for DS1 were Ada_Bag>Ada_RF> LSVM >Ada_Jrip and Ada_PART>Ada_RT> Ada_J48. For DS3, they were Ada_Bag>Ada_RF> LSVM >Ada_Jrip>Ada_PART>Ada_RT> Ada_J48. Table 14 illustrates that Kendall's coefficients (also for DS1 and DS3 using the specificity measure) were significant (p <0.05, χ 2 > 12.59) and that the performance of Ada_Bag in DS1 and Ada_RF in DS3 significantly outperformed all of the other methods. The overall rankings for DS1 were Ada_Bag>Ada_RF> LSVM >Ada_Jrip>Ada_RT>Ada_PART> Ada_J48. For DS3 the rankings were Ada_RF>Ada_Bag>Ada_Jrip> LSVM >Ada_RT> Ada_J48 >Ada_PART. Table 15 illustrates that Kendall's coefficients (also for DS1 and DS3 using the AUC measure) were significant (p <0.05, χ 2 > 12.59) and that the performance of Ada_RF and Ada_Bag considerably surpassed all of the other methods. The overall rankings for DS1 were Ada_RF> Ada_Bag> LSVM >Ada_Jrip>Ada_RT>Ada_PART> Ada_J48. For DS3 they were Ada_Bag>Ada_RF> LSVM >Ada_Jrip>Ada_PART>Ada_RT> Ada_J48.
In contrast, it can be seen in Tables 13-15 that the results for DS2 using all measures (sensitivity, specificity and AUC) were not significant (p > 0.05, χ 2 < 12.59) because the performance of all classifiers in DS2, even though good, were very similar to each other. As such, the differences were not significant. Fig 1 (below) illustrates that the highest accuracy was obtained by Ada_PART 96.72% in DS1, Ada_J48 with 98.11% in DS2, and Ada_Bag with 94.54% in DS3. Thus, from the results in Fig 1, we can also conclude that AdaBoost classifiers were able to handle all the datasets.
Most importantly, the results for DS3 (Fig 1) show that using Ada_Bag as the AdaBoost classifier improved the effectiveness of the prediction of new bioactive molecules in highly Ensemble learning method for prediction of new bioactive molecules diverse data when compared to using the existing best classification method (LSVM). The results of DS3 show an accuracy of 94.54% compared to 93.73% for LSVM.
In comparison, our proposed methods outperform the method adopted by Liu et al. [44], of which the Liu et al. 2016 method supersedes four other works, as illustrated in their report.

Conclusions
In this paper, we have presented various machine learning and ensemble methods that were applied to three MDDR benchmark datasets. The results of the experiments illustrate that the incorporation of the boosting algorithm (AdaboostM1), in conjunction with Bagging (Ada_-Bag) and Random Forest (Ada_RF) as the nominal classifiers into the in silico discovery of drugs, provides a significant improvement with regard to highly diverse datasets. In future research, other ensemble methods will be examined to see if they improve the effectiveness of the prediction of new bioactive molecules.