A Generic Local Algorithm for Mining Data Streams in Large Distributed Systems

• Published in: IEEE transactions on knowledge and data engineering Vol. 21; no. 4; pp. 465 - 478
• Main Authors: Wolff, R., Bhaduri, K., Kargupta, H.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.04.2009; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Clustering; Clustering algorithms; Computer networks; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Costs; Data mining; Data processing. List processing. Character string processing; Distributed databases; Distributed systems; Dynamics; Exact sciences and technology; Information retrieval; Information Storage and Retrieval; Information systems. Data bases; Information Technology; Marketing; Mathematical analysis; Mathematical models; Memory organisation. Data processing; Mining methods and algorithms; Monitoring; Peer to peer computing; Peer to Peer Data Mining; Routing; Sensor systems; Software; Studies; Systems and Software; Wireless sensor networks; Systems and Software; Peer to Peer Data Mining; Distributed systems; Mining methods and algorithms; Distributed databases; Information Storage and Retrieval; Data mining; Information Technology; Complex function; Distributed data mining; Data analysis; Load distribution; Peer to peer; Global data; Measurement sensor; Information retrieval; Routing; K means algorithm; Information extraction; Distributed system; Decision tree; Distributed computing; Modeling; Date; peer-to-peer; Wireless network; Data models; local algorithms; Monitoring
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2008.169

In a large network of computers or wireless sensors, each of the components (henceforth, peers) has some data about the global state of the system. Much of the system's functionality such as message routing, information retrieval and load sharing relies on modeling the global state. We refer to the outcome of the function (e.g., the load experienced by each peer) as the \emph{model} of the system. Since the state of the system is constantly changing, it is necessary to keep the models up-to-date. Computing global data mining models e.g. decision trees, k-means clustering in large distributed systems may be very costly due to the scale of the system and due to communication cost, which may be high. The cost further increases in a dynamic scenario when the data changes rapidly. In this paper we describe a two step approach for dealing with these costs. First, we describe a highly efficient \emph{local} algorithm which can be used to monitor a wide class of data mining models. Then, we use this algorithm as a feedback loop for the monitoring of complex functions of the data such as its k-means clustering. The theoretical claims are corroborated with a thorough experimental analysis.