基于主题相关性的网页正文提取方法优化研究
关键词：网页正文；HTML；相似度；信息提取.
Webpage Text Extraction Optimization based on the Correlation between Text and Title of a Webpage
Abstract: With the rapid development of the Internet, a large quantity of information has been generated on the Internet, which provides abundant information resources for us. However, a webpage is generally composed of some "noise information" besides useful information, such as pictures, hyperlinks, advertising and so on, which will disturb human to get useful information from the webpage. To solve this problem and extract useful text from a webpage efficiently, this paper first uses the HTML tag window technology to extract text from the webpage preliminarily, then uses Regular Expression to remove the “noise information” of the webpage, finally remove the HTML Tags. To further increase the precision of text extraction, the paper takes advantage of Cosine algorithm to calculate the similarity between the extracted and the title of the webpage. Test results show that the method can work well in text extracting from a webpage.
Keywords: Webpage Text; HTML; Similarity; Information Extraction.
目 录
1 绪论 1
1.1 选题背景 1
1.2 研究意义 2
1.3 研究现状及存在的问题 2
1.4 论文研究的内容及结构 3
2 相关知识及技术 4
2.1 信息抽取 4
2.2 HTML 4
2.3 正则表达式 6
2.4 JAVA 8
3 网页正文提取方法 9
3.1 网页正文的定义 9
3.1.1 主题型网页 9
3.1.2 导航型网页 9
3.2 各类网页正文提取方法 10
3.2.1 基于HTML的网页正文提取方法 10
3.2.2 基于DOM树的网页正文提取方法 10
3.2.3 基于内容的网页正文提取方法 11
3.2.4 基于模版的网页正文提取方法 12
3.2.5 基于视觉的网页正文提取方法 12
3.2.6 基于数据挖掘/机器学习的网页正文提取方法 13
3.3 本章小结 13
4 网页正文提取优化方法 14
4.1 正文 14
4.2 精确标题 14
4.3 正文主题相似度的计算 14
4.3.1 文本相似度 14
4.3.2 正文主题相似度的计算方法 15