Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing s...

Full description

Saved in:
Bibliographic Details
Published inConcurrency and computation Vol. 33; no. 16
Main Authors Müller, Juliane, Faybishenko, Boris, Agarwal, Deborah, Bailey, Stephen, Jiang, Chongya, Ryu, Youngryel, Tull, Craig, Ramakrishnan, Lavanya
Format Journal Article
LanguageEnglish
Published Hoboken Wiley Subscription Services, Inc 25.08.2021
Wiley Blackwell (John Wiley & Sons)
Subjects
Online AccessGet full text
ISSN1532-0626
1532-0634
DOI10.1002/cpe.6245

Cover

More Information
Summary:Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.
Bibliography:Funding information
Department of Energy, Office of Science and Office of Advanced Scientific Computing Research, DE‐AC02‐05CH11231
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
DE‐AC02‐05CH11231; AC02-05CH11231
ISSN:1532-0626
1532-0634
DOI:10.1002/cpe.6245