TrieDedup: a fast trie-based deduplication algorithm to handle ambiguous bases in high-throughput sequencing

• Published in: BMC bioinformatics Vol. 25; no. 1; pp. 154 - 13
• Main Authors: Hu, Jianqiao, Luo, Sai, Tian, Ming, Ye, Adam Yongxin
• Format: Journal Article
• Language: English
• Published: London BioMed Central 18.04.2024; BioMed Central Ltd; Springer Nature B.V; BMC
• Subjects: Algorithms; Ambiguous bases; Analysis; Antibodies; B cells; Bioinformatics; Biomedical and Life Sciences; Computational Biology/Bioinformatics; Computer Appl. in Life Sciences; Data structures; Deduplication; DNA sequencing; Evaluation; Gene Library; Genomics; High-Throughput Nucleotide Sequencing; Immune recognition; Immunoglobulins; Life Sciences; Microarrays; Next-generation sequencing; Nucleotide sequencing; Polymerase chain reaction; Prefix tree; Quality control; Sequence Analysis, DNA; Software; Trie; China; Ambiguous bases; Deduplication; Next-generation sequencing; Trie; Prefix tree
• ISSN: 1471-2105; 1471-2105
• DOI: 10.1186/s12859-024-05775-w

Background High-throughput sequencing is a powerful tool that is extensively applied in biological studies. However, sequencers may produce low-quality bases, leading to ambiguous bases, ‘N’s. PCR duplicates introduced in library preparation are conventionally removed in genomics studies, and several deduplication tools have been developed for this purpose. Two identical reads may appear different due to ambiguous bases and the existing tools cannot address ‘N’s correctly or efficiently. Results Here we proposed and implemented TrieDedup, which uses the trie (prefix tree) data structure to compare and store sequences. TrieDedup can handle ambiguous base ‘N’s, and efficiently deduplicate at the level of raw sequences. We also reduced its memory usage by approximately 20% by implementing restrictedDict in Python. We benchmarked the performance of the algorithm and showed that TrieDedup can deduplicate reads up to 270-fold faster than pairwise comparison at a cost of 32-fold higher memory usage. Conclusions The TrieDedup algorithm may facilitate PCR deduplication, barcode or UMI assignment, and repertoire diversity analysis of large-scale high-throughput sequencing datasets with its ultra-fast algorithm that can account for ambiguous bases due to sequencing errors.