Methods For Extracting Content Blocks From Web Pages

The Web has continued to grow up since its inception in volume of information, in the complexity of its topology, as well as in its diversity of content and services. This phenomenon was transformed the web in spite of his young age to an obscure media to take useful information. Today, they are billions of HTML documents, images and other media files on the Internet. Taking into account the wide variety of the web, the extraction of interesting content has become a necessity. Web mining came as a rescue for the above problem. Web content mining is a subdivision under web mining, which is defined as "the process of extracting useful information from the text, images and other forms of content that make up the pages" by eliminating noisy information .This extraction process can employ automatic techniques and hand-crafted rules. In this paper, we propose a method for web data extraction that uses hand-crafted rules developed in Java.

2002

In this paper, we propose a new approach to discover informative contents from a set of tabular documents (or Web pages) of a Web site. Our system, InfoDiscoverer, first partitions a page into several content blocks according to HTML tag <TABLE> in a Web page. Based on the occurrence of the features (terms) in the set of pages, it calculates entropy value of each feature. According to the entropy value of each feature in a content block, the entropy value of the block is defined. By analyzing the information measure, we propose a method to dynamically select the entropy-threshold that partitions blocks into either informative or redundant. Informative content blocks are distinguished parts of the page, whereas redundant content blocks are common parts.

2013

The World Wide Web plays an important role while searching for information in the data network. This paper deals with research in the area of automatic extraction of textual and non-textual information. Developed method consists of two data types of extractions i.e. image and text data extraction. The extraction is performed using the concepts of Document Object Model (DOM) tree. The paper presents a series of data filters to detect and remove irrelevant data from the web page. Many web applications adopt AJAX to enhance their user experience. But AJAX has a number of properties making it extremely difficult for traditional search engines to crawl. The paper proposed an AJAX crawling scheme based on DOM and breadth-first AJAX crawling algorithm. keyword: DOM, Extraction images, Content Extraction.

Journal of Engineering Science and Technology Review, 2020

Due to the emergent of technological development, Web has evolved as the most powerful digital weapon for mankind in recent days. As the size of the web is increasing rapidly, extracting the interesting content from the web become the supreme challenge. In the meantime, the retrieved web pages have many uninteresting content blocks that are not useful for the user which also degrades the performance of content extraction. These uninteresting blocks include advertisements, banners, copyrights, navigation bars etc., and are normally named as web page noise. Removing these noises from the web pages is considered to be the primary task in pre-processing. This paper presents an approach that eliminates the noise and near duplicates for extracting significant content from the web page. The proposed method has three steps. Initially, the web page is divided into various blocks and the block which is considered as noise is removed using tag analysis and Document Object Model Tree. Secondly, the elimination of redundant blocks is carried out by computing fingerprints using modified simhash algorithm with proximity measure. From the distinct blocks, several parameters such as Titlewords, Linkwords and Contentwords are extracted. Thus, the extraction of significant content is carried out by computing the scores for each block using a weighted block scoring mechanism. The blocks having higher score values are extracted and finally, the core content is extracted from the web page. The experimental analysis has been performed and the results show that the proposed method eliminates noise in an efficient way.

With the phenomenal growth of the Web, there is an ever increasing volume of data and information published in numerous Web pages. It is becoming a challenging task to extract and collect the required web pages/information from the web. Data mining is the form of retrieving the knowledge and information which is available in the internet.web mining is extracting information from the web resources and finding interesting patterns and is one of the forms of data mining technique. One of the subfield of Web mining is Web Content Mining. Web content mining is the process of identifying the user specific data from audio, image, text, and video which is already available in the web .This process is also called as web text mining. The web content mining consists of structured data, unstructured and semi-structured data. This paper focus on the web content mining tools , techniques and analyzing the methods for retrieving the data. Keywords: Data Mining, Web Mining, Web Content Mining, Mining Tools, Mining Techniques

Software: Practice and Experience, 2013

Classical Web crawlers make use of only hyperlink information in the crawling process. However, focused crawlers are intended to download only Web pages that are relevant to a given topic by utilizing word information before downloading the Web page. But, Web pages contain additional information that can be useful for the crawling process. We have developed a crawler, iCrawler (intelligent crawler), the backbone of which is a Web content extractor that automatically pulls content out of seven different blocks: menus, links, main texts, headlines, summaries, additional necessaries, and unnecessary texts from Web pages. The extraction process consists of two steps, which invoke each other to obtain information from the blocks. The first step learns which HTML tags refer to which blocks using the decision tree learning algorithm. Being guided by numerous sources of information, the crawler becomes considerably effective. It achieved a relatively high accuracy of 96.37% in our experiments of block extraction. In the second step, the crawler extracts content from the blocks using string matching functions. These functions along with the mapping between tags and blocks learned in the first step provide iCrawler with considerable time and storage efficiency. More specifically, iCrawler performs 14 times faster in the second step than in the first step. Furthermore, iCrawler significantly decreases storage costs by 57.10% when compared with the texts obtained through classical HTML stripping.

2013

Diving into the World Wide Web for the purpose of fetching precious stones (relevant information) is a tedious task under the limitations of current diving equipments (Current Browsers). While a lot of work is being carried out to improve the quality of diving equipments, a related area of research is to devise a novel approach for mining. This paper describes a novel approach to extract the web data from the hidden websites so that it can be used as a free service to a user for a better and improved experience of searching relevant data. Through the proposed method, relevant data (Information) contained in the web pages of hidden websites is extracted by the crawler and stored in the local database so as to build a large repository of structured and indexed and ultimately relevant data. Such kind of extracted data has a potential to optimally satisfy the relevant Information starving end user.

A commercial Web page typically contains many information blocks. Apart from the main content blocks, it usually has such blocks as navigation panels, copyright and privacy notices, and advertisements (for business purposes and for easy user access). We call these blocks that are not the main content blocks of the page, the noisy blocks. We show that the information contained in these noisy blocks can seriously harm Web data mining. Eliminating these noises is thus of great importance. In this project, we propose a noise elimination technique that uses a machine learning (ML) based method which compares HTML tag pairs to estimate how likely they present in the web pages. We use one of the ML techniques called J48 decision tree classifier since decision tree decides the target value (dependent variable) of a new sample based on various attribute values of the available data.

2020

By the fast growth of Communications knowledge and communication growth, in only Internet filled despite junk, also websites used as straightforward for navigation pane. Mainstream web page’s text data typically has granular permissions and could be separated interested in a short script sheet and a multiple-script sheet. To retrieve the webpage script data, the location of textual details can be identified correctly using the multiple text functionality and webpage design rules. Web page data extraction means to collect data from a website by submitting a link to the requested page, then combing different objects through the Markup and sorting the data. This article suggested a technique for obtaining site page script data founded on multiplefunction synthesis, according to the above features. Simulations based on a huge amount of information demonstrates that the method is standardized and highly accurate for extracting text information from solitary text, multiple-text webpages a...

A Web Page has large amount of information. Only some information in web pages is useful in real world applications. Web Page has some additional contents like hyperlinks, header footer, navigational panel; advertisements may cause the content extraction to be complicated. This irrelevant data is available with original content which is known as noisy data of website. This paper discusses various approaches for extracting informative content from web pages and removes noisy data.