A General Framework for Progressive Data Compression and Retrieval

• Published in: IEEE transactions on visualization and computer graphics Vol. 30; no. 1; pp. 1358 - 1368
• Main Authors: Magri, Victor A. P., Lindstrom, Peter
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Algorithms; Client server systems; Costs; Data analysis; Data compression; Data processing; Encoding; Error reduction; floating-point data; Image coding; Image reconstruction; Image resolution; Lossy to lossless compression; MATHEMATICS AND COMPUTING; multi-component expansion; progressive precision; Representations; Retrieval; Task analysis; Tolerances
• ISSN: 1077-2626; 1941-0506; 2160-9306; 1941-0506
• DOI: 10.1109/TVCG.2023.3327186

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.