Navigating Exascale Operational Data Analytics: From Inundation to Insight

• Published in: SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis pp. 1795 - 1804
• Main Authors: Shin, Woong, Osborne, Tim, Karimi, Ahmad Maroof, Palumbo, Rachel, May, Alex, Lester, Corwin, Hines, Jesse, Sattar, Naw Safrin, Huk, Leah, Simmerman, Scott, Brewer, Wesley, Miller, Jeffrey, Adamson, Ryan, Kuchar, Olga, Prout, Ryan, Wang, Feiyi, Atchley, Scott, Oral, Sarp
• Format: Conference Proceeding
• Language: English
• Published: IEEE 17.11.2024
• Subjects: Complexity theory; Data analysis; Data Analytics; Data Governance; HPC Post-Exascale Challenges; Machine Learning Applications; Monitoring; Operational Data Analytics; Organizations; Pipelines; Software; Standards organizations; Streams; Stress; Subject matter experts; Sustainable development; Telemetry; Visual Analytics
• DOI: 10.1109/SCW63240.2024.00226

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.