Study on Web Content Extraction Techniques

2014

Web Mining is extracting information from the web resources and finding interesting patterns that can be useful from ever expanding database of World Wide Web. One of the subfield of Web mining is Web Content Mining. The objective of this paper is four folds. Firstly, this paper introduces Web Mining, Secondly, it tries to explain the interrelationship of Web mining with various other areas, thirdly, it explains various Web Content Mining techniques and finally the paper concludes with the analysis of these various techniques.

Wide Web has rich source of voluminous and heterogeneous information which The World continues to expand in size and complexity. Many Web pages are unstructured and semi structured, so it consists of noisy information like advertisement, links, headers, footers etc. This noisy information makes extraction of Web content tedious. Extracting main content from web page is the preprocessing of web information system. Many techniques that were proposed for Web content extraction are based on automatic extraction and hand crafted rule generation. A hybrid approach is proposed to extract main content from Web pages. A HTML Web page is converted to DOM tree and features are extracted and with the extracted features, rules are generated. Decision tree classification and Naive Bayes classification are machine learning methods used for rules generation.

2012

In last few years, we faced problem regarding extraction of data from web pages. In this paper we proposed to address problem of web data extraction techniques related to areas such as natural language processing, language and grammar, machine learning, information retrieval and Ontologies. As consequence they represent very distinct feature and capabilities which make direct comparison difficult to be done.

Since the INTERNET outburst, consumer perception turned into a complex issue to be measured. Non-traditional advertising methods and new product exhibition alternatives emerged. Forums and review sites allow end users to suggest, recommend or rate products according to their experiences. This gave raise to the study of such data collections. After analyze, store and process them properly, they are used to make reports used to assist in middle to high staff decision making. This research aims to implement concepts and approaches of artificial intelligence to this area. The framework proposed here (named GDARIM), is able to be parameterized and handled to other similar problems in different fields. To do that it first performs deep problem analysis to determine the specific domain variables and attributes. Then, it implements specific functionality for the current data collection and available storage. Next, data is analyzed and processed, using Genetic Algorithms to retro feed the keywords initially loaded. Finally, properly reports of the results are displayed to stakeholders.

International Journal of Computer Applications, 2011

Today internet has made the life of human dependent on it. Almost everything and anything can be searched on net. Web pages usually contain huge amount of information that may not interest the user, as it may not be the part of the main content of the web page. To extract the main content of the web page, data mining techniques need to be implemented. A lot of research has already been done in this field. Current automatic techniques are unsatisfactory as their outputs are not appropriate for the query of the user. In this paper, we are presenting an automatic approach to extract the main content of the web page using tag tree & heuristics to filter the clutter and display the main content. Experimental results have shown that the technique presented in this paper is able to outperform existing techniques dramatically.

World Wide Web as a global service that distributed widely and is a global

World Wide Web over the years has become an huge repository of data from which extracting useful information has become one of the biggest challenges. Furthermore the data is available in the form of structured, semi-structured, and unstructured formats. Much of the information on web is either semi-structured or unstructured. Extracting potential useful knowledge from these various formats has become a big area of research in the recent times. Web Content mining is a subset of Web mining which focuses on extracting useful patterns from the contents available in the web documents. This paper focuses on the various content mining techniques to be applied on the web documents.

2013

The World Wide Web plays an important role while searching for information in the data network. This paper deals with research in the area of automatic extraction of textual and non-textual information. Developed method consists of two data types of extractions i.e. image and text data extraction. The extraction is performed using the concepts of Document Object Model (DOM) tree. The paper presents a series of data filters to detect and remove irrelevant data from the web page. Many web applications adopt AJAX to enhance their user experience. But AJAX has a number of properties making it extremely difficult for traditional search engines to crawl. The paper proposed an AJAX crawling scheme based on DOM and breadth-first AJAX crawling algorithm. keyword: DOM, Extraction images, Content Extraction.

International Journal of Advanced Computer Science and Applications(IJACSA), 2021

In recent years, the emergence of WWW (World Wide Web) led to the accumulation of huge amount of information and data. Hence the web is found to consist of unstructured and structured information that impacts the day to day life of the society. Because of such availability of huge information, utilization of the required information becomes more challenging. This paper provided a comprehensive survey on the current situation and recent trends on web content mining (WCM) and its applications thereby contributing to the enhancement of the upcoming research in WCM. The paper focused mainly on the mining and retrieval techniques, various WCM approaches, challenges and process of information retrieval and information extraction. The paper describes the four major tasks of web content mining that is information retrieval, information extraction, generalization and validation in detail. WCM concentrates on orchestrating, sorting, classifying, collecting, congregating of web data and provide the improved data which can be easily accessed by the users. Web content mining tools were needed to scan text, images and HTML documents and provide results to the search engine. It guides the search engine to provide better productive results for every search based on their importance. The paper also analysed different web content mining tools for the extraction of relevant information from the corresponding web page.

isara solutions, 2020

As the information is growing and changing over the internet at a very fast rate hence search engines are used to search the required information but still there are some limitations likewise to receive fast, accurate and hidden information. Hence web data extraction systems are used which provide the information according to user need and in a fast way which improves the efficiency of search engines. Web Data Extraction systems are a broad class of software applications or methods targeting at extracting information from web sources like web pages, scripts or websites in an efficient manner.