On evaluating stream learning algorithms

• Published in: Machine learning Vol. 90; no. 3; pp. 317 - 346
• Main Authors: Gama, João, Sebastião, Raquel, Rodrigues, Pedro Pereira
• Format: Journal Article
• Language: English
• Published: Boston Springer US 01.03.2013; Springer; Springer Nature B.V
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Change detection; Computer Science; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Control; Data analysis; Decision making models; Errors; Estimators; Exact sciences and technology; Fading; Learning; Mathematical models; Mechatronics; Natural Language Processing (NLP); Robotics; Simulation and Modeling; Software; Streams; Evaluation design; Data streams; Prequential analysis; Concept drift; Forgetting; Streaming; Event detection; Error estimation; Decision making; Standardization; Context aware; Stationary condition; Continuous process; Hypothesis test; Experimental design; Sliding window; Resource management; Dynamic model; Learning algorithm; Time analysis; Artificial intelligence
• ISSN: 0885-6125; 1573-0565; 1573-0565
• DOI: 10.1007/s10994-012-5320-9

Most streaming decision models evolve continuously over time, run in resource-aware environments, and detect and react to changes in the environment generating data. One important issue, not yet convincingly addressed, is the design of experimental work to evaluate and compare decision models that evolve over time. This paper proposes a general framework for assessing predictive stream learning algorithms. We defend the use of prequential error with forgetting mechanisms to provide reliable error estimators. We prove that, in stationary data and for consistent learning algorithms, the holdout estimator, the prequential error and the prequential error estimated over a sliding window or using fading factors, all converge to the Bayes error. The use of prequential error with forgetting mechanisms reveals to be advantageous in assessing performance and in comparing stream learning algorithms. It is also worthwhile to use the proposed methods for hypothesis testing and for change detection. In a set of experiments in drift scenarios, we evaluate the ability of a standard change detection algorithm to detect change using three prequential error estimators. These experiments point out that the use of forgetting mechanisms (sliding windows or fading factors) are required for fast and efficient change detection. In comparison to sliding windows, fading factors are faster and memoryless, both important requirements for streaming applications. Overall, this paper is a contribution to a discussion on best practice for performance assessment when learning is a continuous process, and the decision models are dynamic and evolve over time.