Focused Crawler

A search engine is an information retrieval system designed to minimize the time required to find information over the Web of hyperlinked documents. It provides a user interface that enables the users to specify criteria about an item of interest and searches the same from locally maintained databases. The criteria are referred to as a search query. The search engine is a cascade model comprising of crawling, indexing, and searching modules. Crawling is the first stage that downloads Web documents, which are indexed by the indexer for later use by searching module, with a feedback from other stages. This module could also provide on-demand crawling services for search engines, if required. This paper discusses the issues and challenges involved in the design of the various types of crawlers.

Abstract: The exponential growth of documents available in the World Wide Web makes it ever more difficult to discover relevant information on a specific topic. In this context, growing interest is emerging in focused crawling, a technique that dynamically browses the Internet by choosing directions that maximize the probability of discovering relevant pages, given a specific topic. Predicting the relevance of a document before seeing its contents (i.e., relying on the parent pages only) is one of the central problems in focused crawling because it can save significant bandwidth resources. This paper gives an overview of the various evaluating methods and technique that can be used for focused crawlers. The ultimate aim of these techniques is to predict the values of a discrete class attribute.
Keywords: DP, Focused Crawler, Classifier, Deep Web, WEB 2.0
Title: Evaluation Methods for Focused Crawler: An Over view
Author: Dr. Vivek Chandra, Ms. Nidhi Saxena
International Journal of Computer Science and Information Technology Research
ISSN 2348-120X (online), ISSN 2348-1196 (print)
Research Publish Journals

Focused crawler is an agent to index information according to specific topic. To traverse WWW, focused crawler makes a prediction of hyperlink's visiting priority in order to download relevant documents as maximum as possible and to minimize downloaded irrelevant documents. Many researchers have proposed methods to improve focused crawling precision by minimizing irrelevant documents. However there is a precision and recall trade-off. More precision the results make less recall. This research has studied on conventional focused crawling search strategy (forward crawling) and Web documents structure. The result shows the low recall of conventional focused crawling is caused by some structural characteristics of WWW. Therefore, this research proposes a new strategy of focused crawler. The new strategy is a combination of bidirectional (forward and backward) crawling and bibliometric concepts (co-citation & co-reference). Bidirectional crawling is to improve the exploration and co-citation & co-reference concepts are to control the focusing. With this new strategy, focused crawler can obtain relevant documents that are connected through co-citations or relevant communities that act connected through co-references. Based on experiments that have been carried out, the results show that focused crawler with this new strategy, named CT-FC (more Comprehensive Traversal Focused Crawler) has better exploration capability so that recall increases and precision can remain high.

In this study, a novel product search engine system which supports “find the best products for a given category” type queries is proposed. The product search engine system consists of a focused crawler, a record linkage system, a sentiment analyzer, and a query engine system. The focused crawler is used to crawl product information from various e-commerce sites; the record linkage system determines the identical products that are crawled from different e-commerce sites; the sentiment analyzer classifies users’ reviews about the products as positive or negative so that our product search engine can decide which product is the best for a given category; and the query engine takes the user queries and displays the result. All implementations are done by using C# programming language in .NET 4.5 framework, and MS-SQL Server 2014 database management system is employed for data storage. The core of our system is the record linkage part which is based on a modified incremental Hierarchical Agglomerative Clustering algorithm. To improve the success of record linkage process we also develop a product code matching system such that if the two products from different e-commerce sites have the same product code they are considered as the same. In our experimental analysis we observe 96.25% F-measure in record linkage of E-commerce products and 100% precision in most related products search. Our system can successfully offer best products for a given category. The proposed system achieves to provide better user experience than the existing systems.

Keywords: Sentiment analysis, Focused crawler, Record linkage, Comparison shopping agent, Product ranking

Despite the growth of the web in recent years, some portion of the web remains largely underdeveloped, as shown in lack of high quality contents. An example is the botany specific web directory, in which lack of well-structured web directories have limited user's ability to brows the necessary information. In this research we propose an improved framework for constructing a specific web directory. In pervious study web directory has been constructed by automatic component and filtered by experts. Our interest is to improve automatic component by using dynamic threshold for crawling. It also has been shown that manifold ranking is not useful in some domain like botany.