Implicit indexing of natural language text by reorganizing bytecodes

• Published in: Information retrieval (Boston) Vol. 15; no. 6; pp. 527 - 557
• Main Authors: Brisaboa, Nieves R., Fariña, Antonio, Ladra, Susana, Navarro, Gonzalo
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01.12.2012; Springer; Springer Nature B.V
• Subjects: Compressed; Compressing; Compression ratio; Computer Science; Data compression; Data Mining and Knowledge Discovery; Data Structures and Information Theory; Exact sciences and technology; Indexing; Information and communication sciences; Information processing and retrieval; Information retrieval; Information retrieval systems. Information and document management system; Information retrieval. Man machine relationship; Information science. Documentation; Information Storage and Retrieval; Language; Natural language; Natural language processing; Natural Language Processing (NLP); Pattern Recognition; Proposals; Queries; Random access; Ratios; Representations; Scanning; Sciences and techniques of general use; Searches; Searching; Studies; Texts; Words (language); Word-based compression; Compressed indexing; Searching compressed text; Space; Compression; Word; Regular expression; Use; Digitizing; Representation; Text; Information; Index (documentation); Compression ratio; Database; Natural language; List; Structure; Indexing
• ISSN: 1386-4564; 1573-7659; 1573-7659
• DOI: 10.1007/s10791-012-9184-1

Word-based byte-oriented compression has succeeded on large natural language text databases, by providing competitive compression ratios, fast random access, and direct sequential searching. We show that by just rearranging the target symbols of the compressed text into a tree-shaped structure, and using negligible additional space, we obtain a new implicitly indexed representation of the compressed text, where search times are drastically improved. The occurrences of a word can be listed directly, without any text scanning, and in general any inverted-index-like capability, such as efficient phrase searches, can be emulated without storing any inverted list information. We experimentally show that our proposal performs not only much more efficiently than sequential searches over compressed text, but also than explicit inverted indexes and other types of indexes, when using little extra space. Our representation is especially successful when searching for single words and short phrases.