Validating UTF‐8 in less than one instruction per byte

The majority of text is stored in UTF‐8, which must be validated on ingestion. We present the lookupalgorithm, which outperforms UTF‐8 validation routines used in many libraries and languages by more than 10 times using commonly available single‐instruction‐multiple‐data instructions. To ensure repr...

Full description

Saved in:
Bibliographic Details
Published inSoftware, practice & experience Vol. 51; no. 5; pp. 950 - 964
Main Authors Keiser, John, Lemire, Daniel
Format Journal Article
LanguageEnglish
Published Bognor Regis Wiley Subscription Services, Inc 01.05.2021
Subjects
character encoding
Ingestion
text processing
Unicode
vectorization
Online AccessGet full text
ISSN0038-0644
1097-024X
DOI10.1002/spe.2920

Cover

More Information
Summary:The majority of text is stored in UTF‐8, which must be validated on ingestion. We present the lookupalgorithm, which outperforms UTF‐8 validation routines used in many libraries and languages by more than 10 times using commonly available single‐instruction‐multiple‐data instructions. To ensure reproducibility, our work is freely available as open source software.
Bibliography:Funding information
National Research Council Canada, RGPIN‐2017‐03910
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0038-0644
1097-024X
DOI:10.1002/spe.2920