Distributed computing with load-managed active storage

• Published in: 11th International Symposium on High-Performance Distributed Computing (HPDC-11 2002) pp. 13 - 23
• Main Authors: Wickremesinghe, R., Chase, J.S., Vitter, J.S.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.01.2002
• Subjects: Computational modeling; Computer networks; Computer science; Concurrent computing; Distributed computing; Emulation; Large-scale systems; Parallel processing; Power system modeling; Sorting
• ISBN: 0769516866; 9780769516868
• ISSN: 1082-8907
• DOI: 10.1109/HPDC.2002.1029899

One approach to high-performance processing of massive data sets is to incorporate computation into storage systems. Previous work has shown that this active storage model is effective for a variety of problems. This paper explores opportunities to use active storage as a basis for exploiting asymmetric parallelism in applications using a streaming computation model on collections of fixed-size records. This model is the basis for much of the research in I/O-efficient algorithms, which deals with an important class of massive data problems not studied in previous work on active storage. We present an extension of a streaming computation model for an external memory toolkit to support a flexible mapping of computations to storage-based processors. Our approach enables load-managed active storage: it exposes parallelism, ordering constraints, and primitive computation units to the system, which can configure the application to balance load and make the best use of available processing power Emulation results from a sorting application demonstrate the potential of dynamic adaptation in load-managed active storage.