Two-phase Web site classification based on Hidden Markov Tree models

• Published in: Web intelligence and agent systems Vol. 2; no. 4; pp. 249 - 264
• Main Authors: Tian, Yong-Hong, Huang, Tie-Jun, Gao, Wen
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.11.2004
• Subjects: Web site classification; Hidden Markov Tree model; entropy-based pruning; two-layered dependence tree; two-stage denoising
• ISSN: 1570-1263; 1875-9289
• DOI: 10.3233/WEB-2004-wia00044

The extensive amount of diversified Web-based information necessitates the development of automated subject-specific Web site classification techniques. Given that Web sites are in essence heterogeneous, multi-structured and often accompanied with much noise, it is important to design Web site classification algorithms that can scale well in the context of noise and heterogeneity. In this paper, we propose a novel approach for Web site classification based on the content, structure and context information of Web sites. In our approach, the site structure is represented as a two-layered tree, i.e., each page is modeled as a DOM (Document Object Model) tree, and a page tree is used to hierarchically link all pages within the site. Two context models are formulated to characterize the topical dependences between nodes in the two-layered tree. Using the Hidden Markov Tree (HMT) as the statistical model of page trees and DOM trees, a two-phase Web site classification algorithm is presented. Moreover, for further improving accuracy while reducing the classification overheads, a two-stage denoising procedure is adopted to remove the noise information within sites, and an entropy-based strategy is introduced to dynamically prune the page trees. The experiments demonstrate that the proposed approach is able to offer high accuracy and efficient processing performance.