An evaluation, comparison, and accurate benchmarking of several publicly available MS/MS search algorithms: Sensitivity and specificity analysis

• Published in: Proteomics (Weinheim) Vol. 5; no. 13; pp. 3475 - 3490
• Main Authors: Kapp, Eugene A., Schütz, Frédéric, Connolly, Lisa M., Chakel, John A., Meza, Jose E., Miller, Christine A., Fenyo, David, Eng, Jimmy K., Adkins, Joshua N., Omenn, Gilbert S., Simpson, Richard J.
• Format: Journal Article
• Language: English
• Published: Weinheim WILEY-VCH Verlag 01.08.2005; WILEY‐VCH Verlag
• Subjects: Algorithms; Benchmarking; Blood Proteins - chemistry; Computational Biology; Databases, Protein; False Positive Reactions; Humans; Internet; MASCOT; Mass Spectrometry; Mass Spectrometry - methods; PeptideProphet; Peptides - chemistry; Proteomics - methods; Reference Standards; ROC Curve; Sensitivity and Specificity; SEQUEST; Software; Sonar; Spectrum Mill; Trypsin - pharmacology; X!Tandem
• ISSN: 1615-9853; 1615-9861; 1615-9861
• DOI: 10.1002/pmic.200500126

MS/MS and associated database search algorithms are essential proteomic tools for identifying peptides. Due to their widespread use, it is now time to perform a systematic analysis of the various algorithms currently in use. Using blood specimens used in the HUPO Plasma Proteome Project, we have evaluated five search algorithms with respect to their sensitivity and specificity, and have also accurately benchmarked them based on specified false‐positive (FP) rates. Spectrum Mill and SEQUEST performed well in terms of sensitivity, but were inferior to MASCOT, X!Tandem, and Sonar in terms of specificity. Overall, MASCOT, a probabilistic search algorithm, correctly identified most peptides based on a specified FP rate. The rescoring algorithm, PeptideProphet, enhanced the overall performance of the SEQUEST algorithm, as well as provided predictable FP error rates. Ideally, score thresholds should be calculated for each peptide spectrum or minimally, derived from a reversed‐sequence search as demonstrated in this study based on a validated data set. The availability of open‐source search algorithms, such as X!Tandem, makes it feasible to further improve the validation process (manual or automatic) on the basis of “consensus scoring”, i.e., the use of multiple (at least two) search algorithms to reduce the number of FPs.∁