The Quantitative Evaluation of Functional Neuroimaging Experiments: Mutual Information Learning Curves

• Published in: NeuroImage (Orlando, Fla.) Vol. 15; no. 4; pp. 772 - 786
• Main Authors: Kjems, U., Hansen, L.K., Anderson, J., Frutiger, S., Muley, S., Sidtis, J., Rottenberg, D., Strother, S.C.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.04.2002
• Subjects: Adult; Brain Mapping - methods; Cerebral Cortex - physiology; cross-validation; Data Interpretation, Statistical; Female; generalization error; Humans; Image Processing, Computer-Assisted; Imaging, Three-Dimensional; learning curve; macroscopic and microscopic models; Magnetic Resonance Imaging - statistics & numerical data; Male; Mathematical Computing; Middle Aged; Models, Statistical; multisubject PET and fMRI studies; mutual information; prediction error; Psychomotor Performance - physiology; Reference Values; sensitivity map; Tomography, Emission-Computed - statistics & numerical data; cross-validation; prediction error; mutual information; multisubject PET and fMRI studies; generalization error; macroscopic and microscopic models; sensitivity map; learning curve
• ISSN: 1053-8119; 1095-9572
• DOI: 10.1006/nimg.2001.1033

Learning curves are presented as an unbiased means for evaluating the performance of models for neuroimaging data analysis. The learning curve measures the predictive performance in terms of the generalization or prediction error as a function of the number of independent examples (e.g., subjects) used to determine the parameters in the model. Cross-validation resampling is used to obtain unbiased estimates of a generic multivariate Gaussian classifier, for training set sizes from 2 to 16 subjects. We apply the framework to four different activation experiments, in this case [15O]water data sets, although the framework is equally valid for multisubject fMRI studies. We demonstrate how the prediction error can be expressed as the mutual information between the scan and the scan label, measured in units of bits. The mutual information learning curve can be used to evaluate the impact of different methodological choices, e.g., classification label schemes, preprocessing choices. Another application for the learning curve is to examine the model performance using bias/variance considerations enabling the researcher to determine if the model performance is limited by statistical bias or variance. We furthermore present the sensitivity map as a general method for extracting activation maps from statistical models within the probabilistic framework and illustrate relationships between mutual information and pattern reproducibility as derived in the NPAIRS framework described in a companion paper.