The Venus score for the assessment of the quality and trustworthiness of biomedical datasets

• Published in: BioData mining Vol. 18; no. 1; pp. 1 - 31
• Main Authors: Chicco, Davide, Fabris, Alessandro, Jurman, Giuseppe
• Format: Journal Article
• Language: English
• Published: London BioMed Central 09.01.2025; BioMed Central Ltd; Springer Nature B.V; BMC
• Subjects: Algorithms; Archives & records; Artificial intelligence; Bioinformatics; Biomedical and Life Sciences; Biomedical data; Biomedical data quality; Biomedical engineering; Computational Biology/Bioinformatics; Computational linguistics; Computer Appl. in Life Sciences; Data documentation; Data Mining and Knowledge Discovery; Data science; Data sheets; Data trustworthiness; Datasets; Digital Object Identifier; DNA microarrays; Electronic health records; Electronic medical records; Gene expression; Genomics; Gloss; Guidelines; Health informatics; Informatics; Information management; Language processing; Laws, regulations and rules; Life Sciences; Machine learning; Medical care; Medical data; Medical imaging; Medical imaging equipment; Medical informatics; Medical law; Medical records; Methodology; Natural language interfaces; Quality management; Questionnaires; Search engines; Trustworthiness; Usability; France; Medical data; Trustworthy data; Cheminformatics; Trustworthiness; Kaggle; Data trustworthiness; Datasheets for Datasets; Biomedical data quality; Data documentation; EU AI Act; Medical text; Bioinformatics; Health informatics; Computational biology
• ISSN: 1756-0381; 1756-0381
• DOI: 10.1186/s13040-024-00412-x

Biomedical datasets are the mainstays of computational biology and health informatics projects, and can be found on multiple data platforms online or obtained from wet-lab biologists and physicians. The quality and the trustworthiness of these datasets, however, can sometimes be poor, producing bad results in turn, which can harm patients and data subjects. To address this problem, policy-makers, researchers, and consortia have proposed diverse regulations, guidelines, and scores to assess the quality and increase the reliability of datasets. Although generally useful, however, they are often incomplete and impractical. The guidelines of Datasheets for Datasets , in particular, are too numerous; the requirements of the Kaggle Dataset Usability Score focus on non-scientific requisites (for example, including a cover image); and the European Union Artificial Intelligence Act (EU AI Act) sets forth sparse and general data governance requirements, which we tailored to datasets for biomedical AI. Against this backdrop, we introduce our new Venus score to assess the data quality and trustworthiness of biomedical datasets. Our score ranges from 0 to 10 and consists of ten questions that anyone developing a bioinformatics, medical informatics, or cheminformatics dataset should answer before the release. In this study, we first describe the EU AI Act , Datasheets for Datasets , and the Kaggle Dataset Usability Score , presenting their requirements and their drawbacks. To do so, we reverse-engineer the weights of the influential Kaggle Score for the first time and report them in this study. We distill the most important data governance requirements into ten questions tailored to the biomedical domain, comprising the Venus score. We apply the Venus score to twelve datasets from multiple subdomains, including electronic health records, medical imaging, microarray and bulk RNA-seq gene expression, cheminformatics, physiologic electrogram signals, and medical text. Analyzing the results, we surface fine-grained strengths and weaknesses of popular datasets, as well as aggregate trends. Most notably, we find a widespread tendency to gloss over sources of data inaccuracy and noise, which may hinder the reliable exploitation of data and, consequently, research results. Overall, our results confirm the applicability and utility of the Venus score to assess the trustworthiness of biomedical data.