Stream Bench: Towards Benchmarking Modern Distributed Stream Computing Frameworks

• Published in: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing pp. 69 - 78
• Main Authors: Ruirui Lu, Gang Wu, Bin Xie, Jingtong Hu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2014
• Subjects: benchmark; Benchmark testing; big data; distributed stream computing; Fault tolerance; Fault tolerant systems; Measurement; Message systems; Storms; Throughput
• DOI: 10.1109/UCC.2014.15

While big data is becoming ubiquitous, interest in handling data stream at scale is also gaining popularity, which leads to the sprout of many distributed stream computing systems. However, complexity of stream computing and diversity of workloads expose great challenges to benchmark these systems. Due to lack of standard criteria, evaluations and comparisons of these systems tend to be difficult. This paper takes an early step towards benchmarking modern distributed stream computing frameworks. After identifying the challenges and requirements in the field, we raise our benchmark definition Stream Bench regarding the requirements. Stream Bench proposes a message system functioning as a mediator between stream data generation and consumption. It also covers 7 benchmark programs that intend to address typical stream computing scenarios and core operations. Not only does it care about performance of systems under different data scales, but also takes fault tolerance ability and durability into account, which drives to incorporate four workload suites targeting at these various aspects of systems. Finally, we illustrate the feasibility of Stream Bench by applying it to two popular frameworks, Apache Storm and Apache Spark Streaming. We draw comparisons from various perspectives between the two platforms with workload suites of Stream Bench. In addition, we also demonstrate performance improvement of Storm's latest version with the benchmark.