Extracting the Main Content of Web Pages Using the First Impression Area

• Published in: IEEE access Vol. 10; p. 1
• Main Authors: Jung, Geunseong, Han, Sungjae, Kim, Hansung, Kim, Kwanguk, Cha, Jaehyuk
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Block detection; Boilerplate removal; Feature extraction; Main content extraction; Web content extraction; Web mining; Web segmentation; Websites
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2022.3229080

Extracting the main content from a web page is essential in various applications such as web crawlers and browser reader modes. Existing extraction methods using text-based algorithms and features for English text can be ineffective for non-English web pages. This study proposes a main content extraction method that obtains visual and structural features from the rendered web page. Our method uses the first impression area (FIA), a part of a web page that users initially view. In this area, websites have applied many techniques that enable users to find the main content easily. Using the non-textual properties in the FIA, our method selects three points with high content area density and expands the area from each point until it meets several structural and visual-based conditions. We evaluated our method, browsers' (Mozilla Firefox and Google Chrome) reader modes, and existing main content extraction methods on multilingual datasets using two measures: Longest Common Subsequences and matched text blocks. The results showed that our method performed better than other methods in both English (up to 46%, matched text blocks F 0.5 ) and non-English (up to 42%, matched text blocks F 0.5 ) web pages.