Compression of genomic sequencing reads via hash-based reordering: algorithm and analysis

• Published in: Bioinformatics Vol. 34; no. 4; pp. 558 - 567
• Main Authors: Chandak, Shubham, Tatwawadi, Kedar, Weissman, Tsachy
• Format: Journal Article
• Language: English
• Published: England Oxford University Press 15.02.2018
• Subjects: Algorithms; Bacteria - genetics; Data Compression - methods; Eukaryota - genetics; Genome; Genomics - methods; High-Throughput Nucleotide Sequencing - methods; Humans; Original Papers; Sequence Analysis, DNA - methods; Software
• ISSN: 1367-4803; 1367-4811; 1460-2059; 1367-4811
• DOI: 10.1093/bioinformatics/btx639

Abstract Motivation New Generation Sequencing (NGS) technologies for genome sequencing produce large amounts of short genomic reads per experiment, which are highly redundant and compressible. However, general-purpose compressors are unable to exploit this redundancy due to the special structure present in the data. Results We present a new algorithm for compressing reads both with and without preserving the read order. In both cases, it achieves 1.4×-2× compression gain over state-of-the-art read compression tools for datasets containing as many as 3 billion Illumina reads. Our tool is based on the idea of approximately reordering the reads according to their position in the genome using hashed substring indices. We also present a systematic analysis of the read compression problem and compute bounds on fundamental limits of read compression. This analysis sheds light on the dynamics of the proposed algorithm (and read compression algorithms in general) and helps understand its performance in practice. The algorithm compresses only the read sequence, works with unaligned FASTQ files, and does not require a reference. Supplementary information Supplementary material are available at Bioinformatics online. The proposed algorithm is available for download at https://github.com/shubhamchandak94/HARC.