Data compression using word encoding with Huffman code

• Published in: Journal of the American Society for Information Science Vol. 42; no. 9; pp. 685 - 698
• Main Authors: Liu, Chengwen, Yu, Clement
• Format: Journal Article
• Language: English
• Published: Washington, D.C Wiley Subscription Services, Inc., A Wiley Company 01.10.1991; John Wiley & Sons; American Documentation Institute; Wiley Periodicals Inc
• Subjects: Algorithms; Codes; Compression; Compression ratio; Computerised information retrieval; Computerized information services; Computerized information storage and retrieval; Data compression; Data structures; Decoding; Dictionaries; Exact sciences and technology; Formats. Markup languages. Codification. Conversion; Fragments; Huffman codes; Information and communication sciences; Information and document structure and analysis; Information processing and retrieval; Information science; Information science. Documentation; Information services; Information storage; Information storage and retrieval; Information work; Libraries; Magnetic tape; Methods; Online information retrieval; Ratios; Redundancy; Sciences and techniques of general use; Searching; Subject indexing; Database; Information storage; Decoding; Coding; Huffman code; Data compression
• ISSN: 0002-8231; 1097-4571
• DOI: 10.1002/(SICI)1097-4571(199110)42:9<685::AID-ASI7>3.0.CO;2-1

A technique for compressing large databases is presented. The method replaces frequent variable‐length byte strings (words or word fragments) in the database by minimum‐redundancy codes—Huffman codes. An essential part of the technique is the construction of the dictionary to yield high compression ratios. A heuristic is used to count frequencies of word fragments. A detailed analysis is provided of our implementaton in support of high compression ratios and efficient encoding and decoding under the constraint of a fixed amount of main memory. In each phase of our implementation, we explain why certain data structures or techniques are employed. Experimental results show that our compression scheme is very effective for compressing large databases of library records. © 1991 John Wiley & Sons, Inc.