A scientific data processing framework for time series NetCDF data - Gaustad, Krista; Shippert, Tim; Ermold, Brian; Beus, Sherman; Daily, Jeff; Borsholm, Atle; Fox, Kevin (Environmental modelling & software : with environment data news, 2014)
The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) is a framework designed to streamline the development of scientific algorithms that analyze, and models that use time-series NetCDF data. ADI automates the process of retrieving and preparing data for analysis, provides a modular, flexible framework that simplifies software development, and supports a data integration workflow. Algorithm and model input data, preprocessing, and output data specifications are defined through a graphical interface. ADI includes a library of software modules to support the workflow, and a source code generator that produces C, IDL®, and Python™ templates to jump start development. While developed for processing climate data, ADI can be applied to any time-series data. This paper discusses the ADI framework, and how ADI's capabilities can decrease the time and cost of implementing scientific algorithms allowing modelers and scientists to focus their efforts on their research rather than preparing and packaging data. •ARM Data Integrator (ADI) is a framework to streamline the development of algorithms that analyze time-series data.•ADI can improve the consistency of output data products by checking for adherence to data standards as products are defined.•ADI automates the process of retrieving and preparing data for analysis.•Provides a modular, flexible software development architecture.•The input data, preprocessing, and output data specifications are defined through a graphical interface.