無論是通用搜索還是垂直搜索，其關鍵的核心技術之一就是網(wǎng)絡爬蟲的設計。本文結合
HTMLParser 信息提取方法，對生活類垂直搜索引擎中網(wǎng)絡爬蟲進行了詳細研究。通過深入分析生活類網(wǎng)站網(wǎng)址的樹形結構的構架，開發(fā)了收集種子頁面URL 的模擬搜索器，并基于HTMLParser 的信息提取方法，從種子頁面中提取出與生活類主題相關的目標URL。經實驗測試證明該爬蟲的爬準率達93.552% ，爬全率達96.720% ，表明該網(wǎng)絡爬蟲是有效的，達到中等規(guī)模的垂直搜索企業(yè)級應用的要求。
關鍵詞：網(wǎng)絡爬蟲；垂直搜索； HTMLParser

Abstract：Whether general search engine or vertical search engine, the design of web crawler is the core technology. In this article, a novel system of life-theme web crawler based on HTMLParser information extraction is thoroughly studied. In this system, a simulation searcher is designed for collecting the seed URL by analyzing tree structure of life-theme website, then, based on the discussion of HTMLParser information extraction, the target URL that relate to life-theme is extracted from the seed pages. Empirical studies show that the Pr ecision = 93.552% and the Re call = 96.720%, proving its effectiveness and achieving requirements for general enterprise-level application of vertical search engine.
Key words：web crawler; vertical search engine;HTMLParser