Transcoding unicode characters with AVX‐512 instructions

Intel includes in its recent processors a powerful set of instructions capable of processing 512‐bit registers with a single instruction (AVX‐512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the m...

Full description

Saved in:
Bibliographic Details
Published inSoftware, practice & experience Vol. 53; no. 12; pp. 2430 - 2462
Main Authors Clausecker, Robert, Lemire, Daniel
Format Journal Article
LanguageEnglish
Published Bognor Regis Wiley Subscription Services, Inc 01.12.2023
Subjects
Algorithms
Libraries
Online AccessGet full text
ISSN0038-0644
1097-024X
1097-024X
DOI10.1002/spe.3261

Cover

More Information
Summary:Intel includes in its recent processors a powerful set of instructions capable of processing 512‐bit registers with a single instruction (AVX‐512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the most common formats: UTF‐8 and UTF‐16. With our novel algorithms, we are often twice as fast as the previous best solutions. For example, we transcode Chinese text from UTF‐8 to UTF‐16 at more than 5 GiB using fewer than 2 CPU instructions per character. To ensure reproducibility, we make our software freely available as an open‐source library. Our library is part of the popular Node.js JavaScript runtime.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0038-0644
1097-024X
1097-024X
DOI:10.1002/spe.3261