Identifying practical significance through statistical comparison of meta-heuristic stochastic optimization algorithms

• Published in: Applied soft computing Vol. 85; p. 105862
• Main Authors: Eftimov, Tome, Korošec, Peter
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2019
• Subjects: Benchmarking; Practical significance; Ranking scheme; Single objective problem; Statistical comparison; Stochastic optimization algorithms; Stochastic optimization algorithms; Benchmarking; Single objective problem; Practical significance; Ranking scheme; Statistical comparison
• ISSN: 1568-4946; 1872-9681; 1872-9681
• DOI: 10.1016/j.asoc.2019.105862

In this paper, we propose an extension of a recently proposed Deep Statistical Comparison (DSC) approach, called practical Deep Statistical Comparison (pDSC), which takes into account practical significance when making a statistical comparison of meta-heuristic stochastic optimization algorithms for single-objective optimization. For achieving practical significance, two variants of the standard DSC ranking scheme are proposed. The first is called sequential pDSC, and takes into account practical significance by preprocessing of the independent optimization runs in a sequential order. The second is called Monte Carlo pDSC, and avoids any dependency of practical significance with regard to the ordering of optimization runs. The analysis of identifying practical significance on benchmark tests for single-objective problems, shows that for some cases, both variants of pDSC compared to the Chess Rating System for Evolutionary Algorithms (CRS4EAs) approach give different conclusions. Preprocessing for practical significance is carried out in a similar way, but there are cases when the conclusion for practical significance differ, which comes from the different statistical concepts used to identify practical significance. •A statistical methodology for identifying practical significance.•Two ranking schemes working/dealing with practical significance.•The proposed methodology outperforming state-of-the-art approaches.