Overview of Search Engine and Crawler

2014

I. II. RELATED WORK Matthew Gray [5] wrote the first Crawler, the World Wide Web Wanderer, which was used from 1993 to 1996. In 1998, Google introduced its first distributed crawler, which had distinct centralized processes for each task and each central node was a bottleneck. After some time, AltaVista search engine introduced a crawling module named as Mercator [16], which was scalable, for searching the entire Web and extensible. UbiCrawler [14] a distributed crawler by P. Boldi , with multiple crawling agents, each of which run on a different computer. IPMicra [13] by Odysseus a location-aware distributed crawling method, which utilized an IP address hierarchy, crawl links in a near optimal location aware manner. Hammer and Fiddler [7] ,[8] has

This Paper presents a study of web crawlers used in search engines. Nowadays finding meaningful information among the billions of information resources on the World Wide Web is a difficult task due to growing popularity of the Internet. This paper basically focuses on study of the various kinds of web crawler for finding the relevant information from World Wide Web. A web crawler is defined as an automated program that methodically scans through Internet pages and downloads any page that can be reached via links. A performance analysis of performance of intelligent crawler is presented and data mining algorithms are compared on the basis of crawlers usability.

International Journal of Advanced Trends in Computer Science and Engineering, 2019

With the increase in number of pages being published every day, there is a need to design an efficient crawler mechanism which can result in appropriate and efficient search results for every query. Everyday people face the problem of inappropriate or incorrect answer among search results. So, there is strong need of enhance methods to provide precise search results for the user in acceptable time frame. So this paper proposes an effective approach of building a crawler considering factors of URL ranking, load on the network and number of pages retrieved. The main focus of the paper is on designing of a crawler to improve the effective ranking of URLs using a focused crawler.

International Journal of Advanced Computer Science and Applications, 2013

World Wide Web consists of more than 50 billion pages online. It is highly dynamic [6] i.e. the web continuously introduces new capabilities and attracts many people. Due to this explosion in size, the effective information retrieval system or search engine can be used to access the information. In this paper we have proposed the EPOW (Effective Performance of WebCrawler) architecture. It is a software agent whose main objective is to minimize the overload of a user locating needed information. We have designed the web crawler by considering the parallelization policy. Since our EPOW crawler has a highly optimized system it can download a large number of pages per second while being robust against crashes. We have also proposed to use the data structure concepts for implementation of scheduler & circular Queue to improve the performance of our web crawler. (Abstract)

2019

Making use of search engines is most popular Internet task apart from email. Currently, all major search engines employ web crawlers because effective web crawling is a key to the success of modern search engines. Web crawlers can give vast amounts of web information possible to explore the web entirely by humans. Therefore, crawling algorithms are crucial in selecting the pages that satisfy the users’ needs. Crawling cultural and/or linguistic specific resources from the borderless Web raises many challenging issues. This paper will review various web crawlers used for searching the web while also exploring the use of various algorithms to retrieve web pages. Keyword: Web Search Engine, Web Crawlers, Web Crawling Algorithms.

A search engine is an information retrieval system designed to minimize the time required to find information over the Web of hyperlinked documents. It provides a user interface that enables the users to specify criteria about an item of interest and searches the same from locally maintained databases. The criteria are referred to as a search query. The search engine is a cascade model comprising of crawling, indexing, and searching modules. Crawling is the first stage that downloads Web documents, which are indexed by the indexer for later use by searching module, with a feedback from other stages. This module could also provide on-demand crawling services for search engines, if required. This paper discusses the issues and challenges involved in the design of the various types of crawlers.

Web Crawler also well - known as “Web Robot”, “Web Spider” or merely “Bot” is software for downloading pages from the Web by design. Contrasting what the name may propose, a Web crawler does not in reality stir around computers connected to the Internet – as viruses or intelligent agents do – but only sends requests for documents on Web servers. The input to this software is starting or seed page. As the volume of th e World Wide Web (WWW) grows, it became essential to parallelize a web crawling process, with the intention of finish downloading pages in a rational amount of time. Web crawler which employs multi - processing to permit multiple crawler processes running in concurrent manner. There are a lot of programs out there for web crawling but it required a WebCrawler that allowed trouble - free customization. In this paper we have discussed on crawling technique and how Page Rank can increase the efficiency of web craw ling.

2014

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Web crawling is an important method for collecting data on, and keeping up with, the rapidly expanding Internet. A vast number of web pages are continually being added every day, and information is constantly changing. This Paper is an overview of various types of Web Crawlers and the policies like selection, re-visit, politeness, parallelization involved in it. The behavioral pattern of the Web crawler based on these policies is also taken for the study. The evolution of these web crawler from Basic general purpose web crawler to the latest Adaptive web crawler is studied.

2011

The World Wide Web (WWW) has grown from a few thousand pages in 1993 to more than eight billion pages at present. Due to this explosion in size, web search engines are becoming increasingly important as the primary means of locating relevant information. This research aims to build a crawler that crawls the most important web pages, a crawling system has been built which consists of three main techniques. The first is Best-First Technique which is used to select the most important page. The second is Distributed Crawling Technique which based on UbiCrawler. It is used to distribute the URLs of the selected web pages to several machines. And the third is Duplicated Pages Detecting Technique by using a proposed document fingerprint algorithm.

Extracting information from the web is becoming gradually important and popular. To find Web pages one typically uses search engines that are based on the web crawling framework. A web crawler is a software module that fetches data from various servers. The quality of a crawler directly affects the searching quality. So the time to time performance evaluation of the web crawler is needed. This paper proposes a new URL ordering algorithm .It covers major factors that a good ranking algorithm should have. It also overcomes limitation of PAGERANK. It uses all three web mining technique to obtain a score with its parameters relevance .It is expected to get better result than PAGERANK, as implementation of it in a web crawler is still under progress.