Resource-Aware Distributed Stream Management Using Dynamic Overlays

• Published in: 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05) pp. 783 - 792
• Main Authors: Kumar, V., Cooper, B.F., Zhongtang Cai, Eisenhauer, G., Schwan, K.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 2005
• Subjects: Computer network management; Computer networks; Distributed computing; Educational institutions; Middleware; Network servers; Resource management; Runtime; Scalability; Technology management
• ISBN: 9780769523316; 0769523315
• ISSN: 1063-6927
• DOI: 10.1109/ICDCS.2005.69

We consider distributed applications that continuously stream data across the network, where data needs to be aggregated and processed to produce a 'useful' stream of updates. Centralized approaches to performing data aggregation suffer from high communication overheads, lack of scalability, and unpredictably high processing workloads at central servers. This paper describes a scalable and efficient solution to distributed stream management based on (1) resource-awareness, which is middleware-level knowledge of underlying network and processing resources; (2) overlay-based in-network data aggregation; and (3) high-level programming constructs to describe data-flow graphs for composing useful streams. Technical contributions include a novel algorithm based on resource-aware network partitioning to support dynamic deployment of data-flow graph components across the network, where efficiency of the deployed overlay is maintained by making use of partition-level resource-awareness. Contributions also include efficient middleware-based support for component deployment, utilizing runtime code generation rather than interpretation techniques, thereby addressing both high performance and resource-constrained applications. Finally, simulation experiments and benchmarks attained with actual operational data corroborate this paper's claims