Explore First, Exploit Next: The True Shape of Regret in Bandit Problems

• Published in: Mathematics of operations research Vol. 44; no. 2; pp. 377 - 399
• Main Authors: Garivier, Aurélien, Ménard, Pierre, Stoltz, Gilles
• Format: Journal Article
• Language: English
• Published: Linthicum INFORMS 01.05.2019; Institute for Operations Research and the Management Sciences
• Subjects: Boundary value problems; Computer Science; cumulative regret; Entropy; Information theory; information-theoretic proof techniques; Lower bounds; Machine Learning; Mathematical models; Mathematical research; Mathematics; multiarmed bandits; nonasymptotic lower bounds; Operations research; Probability distribution; Randomized algorithms; Statistics; Uncertainty (Information theory); multi-armed bandits; non-asymptotic lower bounds; cumulative regret; information-theoretic proof techniques
• ISSN: 0364-765X; 1526-5471
• DOI: 10.1287/moor.2017.0928

We revisit lower bounds on the regret in the case of multiarmed bandit problems. We obtain nonasymptotic, distribution-dependent bounds and provide simple proofs based only on well-known properties of Kullback–Leibler divergences. These bounds show in particular that in the initial phase the regret grows almost linearly, and that the well-known logarithmic growth of the regret only holds in a final phase. The proof techniques come to the essence of the information-theoretic arguments used and they involve no unnecessary complications.