A scientific data processing framework for time series NetCDF data

• Published in: Environmental modelling & software : with environment data news Vol. 60; pp. 241 - 249
• Main Authors: Gaustad, Krista, Shippert, Tim, Ermold, Brian, Beus, Sherman, Daily, Jeff, Borsholm, Atle, Fox, Kevin
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.10.2014; Elsevier
• Subjects: Algorithms; Animal, plant and microbial ecology; Atmospheric science; Automation; Biological and medical sciences; Computer programs; computer software; Data management; environmental models; Fundamental and applied biological sciences. Psychology; General aspects. Techniques; meteorological data; Methods and techniques (sampling, tagging, trapping, modelling...); Modular; NetCDF; Nodular iron; Observation data; packaging; Scientific data analysis; Scientific workflow; scientists; Software; Source code; time series analysis; Time-series; Workflow; Atmospheric science; Data management; Scientific data analysis; NetCDF; Time-series; Observation data; Scientific workflow; Data analysis; Sciences; Time series; Data processing
• ISSN: 1364-8152
• DOI: 10.1016/j.envsoft.2014.06.005

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) is a framework designed to streamline the development of scientific algorithms that analyze, and models that use time-series NetCDF data. ADI automates the process of retrieving and preparing data for analysis, provides a modular, flexible framework that simplifies software development, and supports a data integration workflow. Algorithm and model input data, preprocessing, and output data specifications are defined through a graphical interface. ADI includes a library of software modules to support the workflow, and a source code generator that produces C, IDL®, and Python™ templates to jump start development. While developed for processing climate data, ADI can be applied to any time-series data. This paper discusses the ADI framework, and how ADI's capabilities can decrease the time and cost of implementing scientific algorithms allowing modelers and scientists to focus their efforts on their research rather than preparing and packaging data. •ARM Data Integrator (ADI) is a framework to streamline the development of algorithms that analyze time-series data.•ADI can improve the consistency of output data products by checking for adherence to data standards as products are defined.•ADI automates the process of retrieving and preparing data for analysis.•Provides a modular, flexible software development architecture.•The input data, preprocessing, and output data specifications are defined through a graphical interface.