Bayes Optimality in Linear Discriminant Analysis

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 30; no. 4; pp. 647 - 657
• Main Authors: Hamsici, O.C., Martinez, A.M.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.04.2008; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Approximation algorithms; Artificial Intelligence; Bayes optimal; Bayes Theorem; Bayesian analysis; Classification algorithms; Computer science; control theory; systems; Computer Simulation; convex optimization; Costs; data mining; Data processing. List processing. Character string processing; data visualization; Discriminant Analysis; Error functions; Exact sciences and technology; feature extraction; Gaussian distribution; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Information systems. Data bases; Intelligence; Iterative algorithms; Kernel; Linear approximation; Linear discriminant analysis; Linear Models; Memory organisation. Data processing; Minimization; Minimization methods; Null space; Optimization; Pattern analysis; pattern recognition; Pattern Recognition, Automated - methods; Permissible error; Reproducibility of Results; Sensitivity and Specificity; Software; Studies; feature extraction; data mining; Bayes optimal; convex optimization; Linear discriminant analysis; data visualization; pattern recognition; Validation; Discriminant analysis; Data analysis; Error function; Gaussian distribution; Minimization; Information extraction; Pattern recognition; Data mining; Convex programming; Kernel function; Vector space; Linear approximation; Data visualization; Classification; Feature extraction; Convex function; Pattern analysis; Artificial intelligence; Algorithm analysis; Pattern extraction
• ISSN: 0162-8828; 1939-3539
• DOI: 10.1109/TPAMI.2007.70717

We present an algorithm that provides the one-dimensional subspace, where the Bayes error is minimized for the C class problem with homoscedastic Gaussian distributions. Our main result shows that the set of possible one-dimensional spaces v, for which the order of the projected class means is identical, defines a convex region with associated convex Bayes error function g(v). This allows for the minimization of the error function using standard convex optimization algorithms. Our algorithm is then extended to the minimization of the Bayes error in the more general case of heteroscedastic distributions. This is done by means of an appropriate kernel mapping function. This result is further extended to obtain the d dimensional solution for any given d by iteratively applying our algorithm to the null space of the (d - l)-dimensional solution. We also show how this result can be used to improve upon the outcomes provided by existing algorithms and derive a low-computational cost, linear approximation. Extensive experimental validations are provided to demonstrate the use of these algorithms in classification, data analysis and visualization.