Content Extraction from News Pages Using Particle Swarm Optimization on Linguistic and Structural Features

• Published in: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence pp. 242 - 249
• Main Authors: Ziegler, Cai-Nicolas, Skubacz, Michal
• Format: Conference Proceeding
• Language: English
• Published: Washington, DC, USA IEEE Computer Society 02.11.2007
• Series: ACM Conferences
• Subjects: Computing methodologies > Artificial intelligence > Natural language processing; Computing methodologies > Artificial intelligence > Search methodologies > Heuristic function construction; Computing methodologies > Machine learning > Learning paradigms > Supervised learning > Supervised learning by classification; Computing methodologies > Machine learning > Machine learning approaches > Classification and regression trees; Information systems > Information retrieval; Information systems > Information retrieval > Document representation; Information systems > Information retrieval > Document representation > Content analysis and feature selection; Information systems > Information retrieval > Evaluation of retrieval results; Information systems > Information systems applications > Data mining; Mathematics of computing > Mathematical analysis > Mathematical optimization > Continuous optimization > Stochastic control and optimization; Theory of computation > Design and analysis of algorithms > Mathematical optimization > Continuous optimization > Stochastic control and optimization
• ISBN: 0769530265; 9780769530260
• DOI: 10.1109/WI.2007.38

Today's Web pages are commonly made up of more than merely one cohesive block of information. For instance, news pages from popular media channels such as Financial Times or Washington Post consist of no more than 30%-50% of textual news, next to advertisements, link lists to related articles, disclaimer information, and so forth. However, for many search-oriented applications such as the detection of relevant pages for an in-focus topic, dissecting the actual textual content from surrounding page clutter is an essential task, so as to maintain appropriate levels of document retrieval accuracy. We present a novel approach that extracts real content from news Web pages in an unsupervised fashion. Our method is based on distilling linguistic and structural features from text blocks in HTML pages, having a Particle Swarm Optimizer (PSO) learn feature thresholds for optimal classification performance. Empirical evaluations and benchmarks show that our approach works very well when applied to several hundreds of news pages from popular media in 5 languages.