Kernel-Based Ensemble Learning in Python

• Published in: Information (Basel) Vol. 11; no. 2; p. 63
• Main Authors: Guedj, Benjamin, Srinivasa Desikan, Bhargav
• Format: Journal Article
• Language: English
• Published: MDPI 01.02.2020; MDPI AG
• Subjects: Computation; Computer Science; ensemble learning; kernels; Machine Learning; open source software; python; Statistics; ensemble learning; kernels; python; open source software; machine learning
• ISSN: 2078-2489; 2078-2489
• DOI: 10.3390/info11020063

We propose a new supervised learning algorithm for classification and regression problems where two or more preliminary predictors are available. We introduce KernelCobra, a non-linear learning strategy for combining an arbitrary number of initial predictors. KernelCobra builds on the COBRA algorithm introduced by Biau et al. (2016), which combined estimators based on a notion of proximity of predictions on the training data. While the COBRA algorithm used a binary threshold to declare which training data were close and to be used, we generalise this idea by using a kernel to better encapsulate the proximity information. Such a smoothing kernel provides more representative weights to each of the training points which are used to build the aggregate and final predictor, and KernelCobra systematically outperforms the COBRA algorithm. While COBRA is intended for regression, KernelCobra deals with classification and regression. KernelCobra is included as part of the open source Python package Pycobra (0.2.4 and onward), introduced by Srinivasa Desikan (2018). Numerical experiments were undertaken to assess the performance (in terms of pure prediction and computational complexity) of KernelCobra on real-life and synthetic datasets.