Soft Margin Multiple Kernel Learning

• Published in: IEEE transaction on neural networks and learning systems Vol. 24; no. 5; pp. 749 - 761
• Main Authors: Xinxing Xu, Tsang, I. W., Dong Xu
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.05.2013; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial intelligence; Computer science; control theory; systems; Data processing. List processing. Character string processing; Exact sciences and technology; Fasteners; Hinges; Kernel; Kernels; Learning; Learning and adaptive systems; Linear programming; Mathematical models; Memory organisation. Data processing; Multiple kernel learning; Neural networks; Operations research; Optimization; Pattern recognition. Digital image processing. Computational geometry; Recognition; Software; Studies; Support vector machines; Training; Vectors; Multiple kernel learning; Action; Event detection; support vector machines; Motion estimation; Video signal; Modeling; Kernel method; Loss function; Behavioral analysis; Efficiency; Scene analysis; Vector support machine; Objective function; Learning algorithm; Single machine
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2012.2237183

Multiple kernel learning (MKL) has been proposed for kernel methods by learning the optimal kernel from a set of predefined base kernels. However, the traditional L 1 MKL method often achieves worse results than the simplest method using the average of base kernels (i.e., average kernel) in some practical applications. In order to improve the effectiveness of MKL, this paper presents a novel soft margin perspective for MKL. Specifically, we introduce an additional slack variable called kernel slack variable to each quadratic constraint of MKL, which corresponds to one support vector machine model using a single base kernel. We first show that L 1 MKL can be deemed as hard margin MKL, and then we propose a novel soft margin framework for MKL. Three commonly used loss functions, including the hinge loss, the square hinge loss, and the square loss, can be readily incorporated into this framework, leading to the new soft margin MKL objective functions. Many existing MKL methods can be shown as special cases under our soft margin framework. For example, the hinge loss soft margin MKL leads to a new box constraint for kernel combination coefficients. Using different hyper-parameter values for this formulation, we can inherently bridge the method using average kernel, L 1 MKL, and the hinge loss soft margin MKL. The square hinge loss soft margin MKL unifies the family of elastic net constraint/regularizer based approaches; and the square loss soft margin MKL incorporates L 2 MKL naturally. Moreover, we also develop efficient algorithms for solving both the hinge loss and square hinge loss soft margin MKL. Comprehensive experimental studies for various MKL algorithms on several benchmark data sets and two real world applications, including video action recognition and event recognition demonstrate that our proposed algorithms can efficiently achieve an effective yet sparse solution for MKL.