Discovering dependencies with reliable mutual information

• Published in: Knowledge and information systems Vol. 62; no. 11; pp. 4223 - 4253
• Main Authors: Mandros, Panagiotis, Boley, Mario, Vreeken, Jilles
• Format: Journal Article
• Language: English
• Published: London Springer London 01.11.2020; Springer Nature B.V
• Subjects: Algorithms; Approximation; Codes; Computer Science; Data Mining and Knowledge Discovery; Database Management; Entropy; Expected values; Feature selection; Greedy algorithms; Heuristic; Heuristic methods; Informatics; Information Storage and Retrieval; Information Systems and Communication Service; Information Systems Applications (incl.Internet); Information theory; IT in Business; Knowledge discovery; Lattice theory; Machine learning; Optimization; Questions; Regular Paper; Search algorithms; Search methods; Branch-and-bound; Knowledge discovery; Algorithms; Pattern mining; Information theory; Approximate functional dependency
• ISSN: 0219-1377; 0219-3116; 0219-3116
• DOI: 10.1007/s10115-020-01494-9

We consider the task of discovering functional dependencies in data for target attributes of interest. To solve it, we have to answer two questions: How do we quantify the dependency in a model-agnostic and interpretable way as well as reliably against sample size and dimensionality biases? How can we efficiently discover the exact or α -approximate top- k dependencies? We address the first question by adopting information-theoretic notions. Specifically, we consider the mutual information score, for which we propose a reliable estimator that enables robust optimization in high-dimensional data. To address the second question, we then systematically explore the algorithmic implications of using this measure for optimization. We show the problem is NP-hard and justify worst-case exponential-time as well as heuristic search methods. We propose two bounding functions for the estimator, which we use as pruning criteria in branch-and-bound search to efficiently mine dependencies with approximation guarantees. Empirical evaluation shows that the derived estimator has desirable statistical properties, the bounding functions lead to effective exact and greedy search algorithms, and when combined, qualitative experiments show the framework indeed discovers highly informative dependencies.