A survey on data‐efficient algorithms in big data era

• Published in: Journal of big data Vol. 8; no. 1; pp. 1 - 54
• Main Author: Adadi, Amina
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 26.01.2021; Springer Nature B.V; SpringerOpen
• Subjects: Algorithms; Big Data; Communications Engineering; Computational Science and Engineering; Computer Science; Data acquisition; Data augmentation; Data hungry algorithms; Data Mining and Knowledge Discovery; Database Management; Data‐efficiency; Domains; Information Storage and Retrieval; Machine learning; Mathematical Applications in Computer Science; Networks; Small sample learning; Survey Paper; Transfer learning; Small sample learning; Data‐efficiency; Data augmentation; Transfer learning; Data hungry algorithms
• ISSN: 2196-1115; 2196-1115
• DOI: 10.1186/s40537-021-00419-9

The leading approaches in Machine Learning are notoriously data-hungry. Unfortunately, many application domains do not have access to big data because acquiring data involves a process that is expensive or time-consuming. This has triggered a serious debate in both the industrial and academic communities calling for more data-efficient models that harness the power of artificial learners while achieving good results with less training data and in particular less human supervision. In light of this debate, this work investigates the issue of algorithms’ data hungriness. First, it surveys the issue from different perspectives. Then, it presents a comprehensive review of existing data-efficient methods and systematizes them into four categories. Specifically, the survey covers solution strategies that handle data-efficiency by (i) using non-supervised algorithms that are, by nature, more data-efficient, by (ii) creating artificially more data, by (iii) transferring knowledge from rich-data domains into poor-data domains, or by (iv) altering data-hungry algorithms to reduce their dependency upon the amount of samples, in a way they can perform well in small samples regime. Each strategy is extensively reviewed and discussed. In addition, the emphasis is put on how the four strategies interplay with each other in order to motivate exploration of more robust and data-efficient algorithms. Finally, the survey delineates the limitations, discusses research challenges, and suggests future opportunities to advance the research on data-efficiency in machine learning.