Towards flexible Open Data management solutions

• Published in: 2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) pp. 233 - 237
• Main Authors: von St Vieth, Benedikt, Rybicki, Jedrzej, Brzezniak, Maciej
• Format: Conference Proceeding
• Language: English
• Published: Croatian Society MIPRO 01.05.2017
• Subjects: Containers; Data Management; Data Replication; Metadata; Middleware; Namespaces; Object Stores; Pipelines; Publishing; Uniform resource locators
• DOI: 10.23919/MIPRO.2017.7973425

Many data sharing initiatives emerged in the recent time. There are various driving factors behind this phenomena subsumed under the terms Open Data and Open Research. The most prominent one is the fact that more and more funding agencies request publishing of all project results including data. Publishing and sharing data enables the verification of obtained results by facilitating repeatability and reproducibility. Also economical aspects are important: Data reuse can reduce the research costs, and redoing of the same experimental work can be avoided (provided sufficient visibility of earlier results). Publishing or Open Data is also contributing to researcher's visibility and reputation. The implementation of data sharing is, however, quite challenging. It demands both storage technologies and services that can deal with the increasing amounts of data in an efficient and cost-effective way while remaining user-friendly and future-proof. We describe the architecture of a data management solution that is able to provide a scalable, extensible, yet cost-effective storage engine with several replication mechanisms. For the sake of user-friendliness and high data visibility, the solution has a mechanism to create flexible namespaces for a efficient organization, searching, and retrieving of the data. Such namespaces can be tailored to the researchers' needs, potential facilitating data reuse across community borders. The presented architecture embodies our experiences gathered in the EU-funded project EUDAT2020.