An Ontology-Based Focused Crawler

Proceedings of The International Workshop on Semantic Big Data, 2017

Information from the web is a key resource exploited in the domain of competitive intelligence. These sources represent important volumes of information to process everyday. As the amount of information available grows rapidly, this process becomes overwhelming for experts. To leverage this challenge, this paper presents a novel approach to process such sources and extract only the most valuable pieces of information. The approach is based on an unsupervised and adaptive ontology-learning process. The resulting ontology is used to enhance the performance of a focused crawler. The combination of Big Data and Semantic Web technologies allows to classify information precisely according to domain knowledge, while maintaining optimal performances. The approach and its implementation are described, and an presents the feasibility and performance of the approach.

Computer Networks, 1999

The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext resource discovery system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. The topics are specified not using keywords, but using exemplary documents. Rather than collecting and indexing all accessible Web documents to be able to answer all possible ad-hoc queries, a focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl, and avoids irrelevant regions of the Web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date.

Proceedings of the 2003 ACM symposium on …, 2003

The Web, the largest unstructured database of the world, has greatly improved access to documents. However, documents on the Web are largely disorganized. Due to the distributed nature of the World Wide Web it is difficult to use it as a tool for information and knowledge management. Therefore, users doing the difficult task of exploring the Web have to be supported by intelligent means. This paper proposes an approach for document discovery building on a comprehensive framework for ontology-focused crawling of Web documents. Our framework includes means for using a complex ontology and associated instance elements. It defines several relevance computation strategies and provides an empirical evaluation which has shown promising results.

IARJSET, 2015

To retrieve information from web we use Google, Yahoo and MSN which are more famous search engines. Search engine is one tool to locate into on the www. It search for and identifies items in database with reference to keywords entered by user (where we get relevant data also which is not exact what we which is time consuming).For web crawling we use focused crawler which based on ontology architecture. Focused crawler search for web pages having more page rank for user requirement. Where ontology is a specification about domain. It is a part of artificial intelligence. Also web pages on site are liked with ontology structure. In pepper we use Focused Crawler and Ontology for exact service related site.

2013

Finding useful information from the web which has a large and distributed structure requires efficient search strategies. Focused crawlers selectively retrieve Web documents that are relevant to a predefined set of topics. To intelligently make decisions about relevant URLs and web pages, different authors had proposed different strategies. In this paper we review and compare focused crawling strategies, studied and published during the past few years. We also give the details of various issues related to focused crawling. Keywords— Semantic web, crawler, focused, ontology.

2009

Manual ontology construction is costly, time- consuming, error-prone and inflexible to change. To address these problems, researchers hope that an automated process will result in faster and better ontology construction and enrichment. Ontology learning has become recently a major area of research whose goal is to facilitate the construction of ontologies by decreasing the amount of effort required to produce an ontology for a new domain. However, most of current approaches are dealing with some specific tasks or a part of the ontology learning process rather than providing complete support to users. There are few studies that attempt to automate the entire ontology learning process from the collection of domain-specific literature, filtering out documents irrelevant to the domain, to text mining to build new ontologies or enrich existing ones. In this paper, we present a complete framework for ontology learning that enables us to retrieve documents from the Web using focused crawli...

Journal of Computer Science and Technology, 2012

The Web comprises of voluminous rich learning content. The volume of ever growing learning resources however leads to the problem of information overload. A large number of irrelevant search results generated from search engines based on keyword matching techniques further augment the problem. A learner in such a scenario needs semantically matched learning resources as the search results. Keeping in view the volume of content and significance of semantic knowledge, our paper proposes a multi-threaded semantic focused crawler (SFC) specially designed and implemented to crawl on the WWW for educational learning content. The proposed SFC utilizes domain ontology to expand a topic term and a set of seed URLs to initiate the crawl. The results obtained by multiple iterations of the crawl on various topics are shown and compared with the results obtained by executing an open source crawler on the similar dataset. The results are evaluated using Semantic Similarity, a vector space model based metric, and the harvest ratio.

International Journal of Computer …, 2012

Finding meaningful information among the billions of information resources on the Web is a difficult task due to growing popularity of the Internet. The future of World Wide Web (WWW) is the Semantic Web, where ontologies are used to assign (agreed) meaning to the content of the Web. On the Semantic Web, data will inevitably be linked to many different ontologies, and information processing across ontologies is not possible without knowing the semantic mappings between them. As the resources on the Semantic Web are annotated using these ontologies, new search techniques are required to find specific information. For this, architecture has been proposed for ontology based semantic web crawler. This architecture can exploit the semantic metadata to efficiently discover and extract information from the Semantic Web. In this paper Semantic matching between content of downloaded web page and ontology is used to guide the crawler towards relevant information.

2017

As Web is massive, much less coherent, changes more rapidly, and is spread over geographically distributed computers. This requires new information retrieval techniques, or extensions to the old ones, to deal with the gathering of the information, to make index structures scalable and efficiently updateable, and to improve the discriminating ability of search engines. A crawler solves the Resource Discovery Problem in the context of WWW by retrieving information from remote sites using standard web protocols. The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called focused crawler or topical crawlers. --------------------------------------------------------------------------------------------------------------------------------------Date of Submission: 13-11-2017 Date of acceptance: 30-11-2017 ------------------------------------------...

2008

We present work in progress on automated and ontology-guided discovery, extraction and mapping of information sources on the Semantic Web. It concerns an ontology-guided focused crawler to discover and match different data sources. We have developed an automated ontology-matcher embedded in the crawler that relates semantic web documents found during the crawl to an initial topic ontology that describes the domain of interest of the crawl. Similarity coefficients resulting from the matching process are used to guide the crawler to the information sources relevant to the modelled domain of interest. In addition, the matching process also provides links between the different data sources, which helps in the integration of this data at a later stage. In this paper, the overall architecture, the various modules and methods actually implemented are presented.