SpiderServer: the MetaSearch engine of WebNaut

In this paper we review studies of the growth of the Internet and technologies that are useful for information search and retrieval on the Web. Search engines are retrieve the efficient information. We collected data on the Internet from several different sources, e.g., current as well as projected number of users, hosts, and Web sites. The trends cited by the sources are consistent and point to exponential growth in the past and in the coming decade. Hence it is not surprising that about 85% of Internet users surveyed claim using search engines and search services to find specific information and users are not satisfied with the performance of the current generation of search engines; the slow retrieval speed, communication delays, and poor quality of retrieved results. Web agents, programs acting autonomously on some task, are already present in the form of spiders, crawler, and robots. Agents offer substantial benefits and hazards, and because of this, their development must involve attention to technical details. This paper illustrates the different types of agents ,crawlers, robots ,etc for mining the contents of web in a methodical, automated manner, also discusses the use of crawler to gather specific types of information from Web pages, such as harvesting e-mail addresses.

The Web is a context in which traditional Information Retrieval methods are challenged. Given the volume of the Web and its speed of change, the coverage of modern web search engines is relatively small. Search engines attempt to crawl the web exhaustively with crawler for new pages, and to keep track of changes made to pages visited earlier. The centralized design of crawlers introduces limitations in the design of search engines. It has been recognized that as the size of the web grows, it is imperative to parallelize the crawling process. Contents other then standard documents (Multimedia content and Databases etc) also makes searching harder since these contents are not visible to the traditional crawlers. Most of the sites stores and retrieves data from backend databases which are not accessible to the crawlers. This results in the problem of hidden web. This paper proposes and implements DCrawler, a scalable, fully distributed web crawler. The main features of this crawler are platform independence, decentralization of tasks, a very effective assignment function for partitioning the domain to crawl, and the ability to cooperate with web servers. By improving the cooperation between web server and crawler, the most recent and updates results can be obtained from the search engine. A new model and architecture for a Web crawler that tightly integrates the crawler with the rest of the search engine is designed first. The development and implementation are discussed in detail. Simple tests with distributed web crawlers successfully show that the Dcrawler performs better then traditional centralized crawlers. The mutual performance gain increases as more crawlers are added.

IFIP Advances in Information and Communication Technology, 1998

The steady increase in the amount of information in digital format public on computer networks around the world, has caused the difficulty of users to find what they really need at any given time. To locate the required information, the Information Retrieval Systems were designed; whose functionalities, have a large number of configuration options and difficult to administer. Apache Nutch is a free spiders with big advantages for collection and finding information on the web; however lacks a system that enables visually configuration without using console commands and conducive working with multiple instances simultaneously. At the University of Informatics Sciences of The Havana, Cuba, Orion search engine was developed, but it has many disadvantages that prevent optimal performance of the process of setting up its tracking mechanism based on Nutch. In this paper are shown the essential elements taken into account in the implementation of a system that improves the usability and makes easy the work of administrators in the configuration tasks. The system implemented, has a set of features and functionalities that contribute, through the availability of web interfaces, increased control of configuration changes and streamlining the process; also providing information on the settings, that previously impossible or difficult to obtain.

1994

While the World−Wide Web (WWW, or Web for short) offers an incredibly rich base of information, organized as a hypertext, it does not provide a uniform and efficient way to retrieve specific information, based on user−defined search−criteria. Two types of search tools have been developed for the WWW: Gateways, offering (limited) search operations on small or large parts of the WWW, using a pre−compiled database. These databases are often built by an automated Web scanner (a "robot"). A client−based search tool that does automated navigation, thereby working more or less like a browsing user, but much faster and following an optimized strategy. This paper highlights the properties, implementation and possible future developments of a client−based search tool called the fish−search, and compares it to other approaches. The fish−search, implemented on top of Mosaic for X, offers an open−ended selection of search criteria. It allows the search to start from the "current" document or from the documents in the user's hotlist. The future of client−based searching will depend heavily on assistance from WWW servers. The fish−search algorithm is currently being used to search by navigating through individual WWW documents. Substantial speedup and reduction of network resource consumption will become possible when the same search algorithm is converted to navigate through a Web of servers instead of a Web of documents. In order to implement such a search algorithm, WWW servers need to offer a service similar to that provided by GlimpseHTTP.

2001

It is estimated that there are hundreds of thousands of information sources on the Web, including both the Surface Web and the Deep Web. Most of these sources have their own search capabilities. In order to alleviate ordinary users from the formidable task of identifying useful sources and search them individually, it is important to provide a unified access to these sources. Metasearch engine is such a system that can provide unified access to multiple existing search systems. In this paper, we provide an architectural design of a large-scale metasearch system, WebScales. We also discuss what we have already done in developing an operational system based on the proposed architecture.

2014

I. II. RELATED WORK Matthew Gray [5] wrote the first Crawler, the World Wide Web Wanderer, which was used from 1993 to 1996. In 1998, Google introduced its first distributed crawler, which had distinct centralized processes for each task and each central node was a bottleneck. After some time, AltaVista search engine introduced a crawling module named as Mercator [16], which was scalable, for searching the entire Web and extensible. UbiCrawler [14] a distributed crawler by P. Boldi , with multiple crawling agents, each of which run on a different computer. IPMicra [13] by Odysseus a location-aware distributed crawling method, which utilized an IP address hierarchy, crawl links in a near optimal location aware manner. Hammer and Fiddler [7] ,[8] has

2001

Web search engines have become the necessary tools for Web users to search useful information. However, users often face problems such as how to find the right information promptly with the least effort. This paper examines the limitations of current Web search engines and proposes a new Web meta-search engine architecture. The proposed architecture uses a multi-agent approach to process the user queries with greater personalization functionality and higher result quality than regular meta-search engines. Thus, the propose method improves the usability and efficiency of current search engines.

2005

Research into metasearch engines has traditionally been focused on source engine selection and the re-ranking and integration of multiple search engines' results. In this paper we describe existing and new techniques that metasearch engines can apply to broaden the services provided to the searcher, by capturing and analyzing information already flowing through the meteasearch engine-the queries, their results, and the clicks made by users. In addition, we demonstrate the potential for the engine to generate related queries, related pages, and to recommend queries for web site builders to target through optimization or advertising.

2007

With the popularity of computer and network technologies, the number of web sites have rapidly increased, which has significantly stimulated the development of search engines. In recent years, a considerable number of search engines have been developed. The problem of how to find information on the Internet has been replaced with the problems of where to find the search engines, what they are designed to retrieve, and how to make the best use of them. Metasearch engine is a technique to cope with these problems by selecting and interfacing with multiple remote search engines at one time. One important issue on metasearch is how accurately it can select and rank feasible remote search engines for user queries. In this paper, a ratio-based metaindex approach is proposed for selecting remote search engines. Based on the novel approach, an experimental metasearch engine has been implemented. Some experimental results have shown that the accuracy and efficiency of the novel approach are more desirable than the famous metasearch engine SavvySearch.