Scalable massively parallel I/O to task-local files

• Published in: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis pp. 1 - 11
• Main Authors: Frings, Wolfgang, Wolf, Felix, Petkov, Ventsislav
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 14.11.2009
• Series: ACM Conferences
• Subjects: Bandwidth; Codes; File systems; General and reference > Cross-computing tools and techniques > Performance; Government; Libraries; Metadata; Multicore processing; Programming; Scalability; Servers; Software and its engineering > Software creation and management > Software verification and validation > Operational analysis; Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > Communications management > Input > output; Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > Memory management > Distributed memory; Software and its engineering > Software organization and properties > Software system structures > Distributed systems organizing principles
• ISBN: 1605587443; 9781605587448
• ISSN: 2167-4329
• DOI: 10.1145/1654059.1654077

Parallel applications often store data in multiple task-local files, for example, to remember checkpoints, to circumvent memory limitations, or to record performance data. When operating at very large processor configurations, such applications often experience scalability limitations when the simultaneous creation of thousands of files causes metadataserver contention or simply when large file counts complicate file management or operations on those files even destabilize the file system. SIONlib is a parallel I/O library that addresses this problem by transparently mapping a large number of task-local files onto a small number of physical files via internal metadata handling and block alignment to ensure high performance. While requiring only minimal source code changes, SIONlib significantly reduces file creation overhead and simplifies file handling without penalizing read and write performance. We evaluate SIONlib's efficiency with up to 288 K tasks and report significant performance improvements in two application scenarios.