Experimenting with prequential variations for data stream learning evaluation

• Published in: Computational intelligence Vol. 35; no. 4; pp. 670 - 692
• Main Authors: Hidalgo, Juan I. González, Maciel, Bruno I. F., Barros, Roberto S. M.
• Format: Journal Article
• Language: English
• Published: Hoboken Blackwell Publishing Ltd 01.11.2019
• Subjects: Accuracy; concept drift; Data processing; data stream; Data transmission; Distance learning; Evaluation; online learning; Performance evaluation; prequential evaluation; Probabilistic inference; Sliding; Statistical analysis; Statistical inference
• ISSN: 0824-7935; 1467-8640
• DOI: 10.1111/coin.12208

Processing data streams requires new demands not existent on static environments. In online learning, the probability distribution of the data can often change over time (concept drift). The prequential assessment methodology is commonly used to evaluate the performance of classifiers in data streams with stationary and non‐stationary distributions. It is based on the premise that the purpose of statistical inference is to make sequential probability forecasts for future observations, rather than to express information about the past accuracy achieved. This article empirically evaluates the prequential methodology considering its three common strategies used to update the prediction model, namely, Basic Window, Sliding Window, and Fading Factors. Specifically, it aims to identify which of these variations is the most accurate for the experimental evaluation of the past results in scenarios where concept drifts occur, with greater interest in the accuracy observed within the total data flow. The prequential accuracy of the three variations and the real accuracy obtained in the learning process of each dataset are the basis for this evaluation. The results of the carried‐out experiments suggest that the use of Prequential with the Sliding Window variation is the best alternative.