Python and pipelines: Python interfaces for ESO’s CPL and HDRL data reduction libraries

• Main Authors: Horton, Anthony, Heng, Anthony, Farrell, Aidan, Deshpande, Mrunmayi, Miszalski, Brent, Fernando, Nuwanthika, Lorente, Nuria, Palsa, Ralf, Lundin, Lars, Gabasch, Armin
• Format: Conference Proceeding
• Language: English
• Published: SPIE 25.07.2024
• ISBN: 9781510675254; 1510675256
• ISSN: 0277-786X
• DOI: 10.1117/12.3019678

The ESO Common Pipeline Library (CPL) and High-Level Data Reduction Library (HDRL) together form a comprehensive, efficient and robust software toolkit for data reduction pipelines. They were developed in C for reasons of efficiency and speed, however, with the community’s preference towards Python for algorithm prototyping and data reduction, there is a need for access from Python. PyCPL and PyHDRL provide this, making it possible to run existing CPL data reduction recipes from Python as well as developing new recipes in Python. These new recipes are built using the PyCPL and PyHDRL libraries, which provide idiomatic Python interfaces to CPL and HDRL while allowing users to take advantage of the scientific Python ecosystem. PyCPL and PyHDRL are already being used to prototype recipes for the MAVIS instrument pipeline and have been used to develop an extensible pipeline development framework. Here we describe their design, implementation and usage.