streammd: fast low-memory duplicate marking using a Bloom filter - Leonard, Conrad R (bioRxiv, 2022)
The identification of duplicate reads is an essential pre processing step in short-read sequencing analysis. For large sequencing libraries this step is typically time-consuming and resource-intensive. Here we present streammd: a fast, memory-efficient, single-pass duplicate marking tool operating on the principle of a Bloom filter. We show that streammd closely reproduces the outputs of Picard MarkDuplicates, a widely-used duplicate marking program, while being substantially faster and suitable for pipelined applications, and that it requires much less memory than SAMBLASTER, another single-pass duplicate marking tool. Competing Interest Statement The authors have declared no competing interest. Footnotes * https://github.com/delocalizer/streammd