Web News

description10 papers

group19 followers

lightbulbAbout this topic

Web news refers to news content that is published and disseminated through online platforms, utilizing the internet as a medium for distribution. It encompasses various formats, including articles, videos, and interactive content, and is characterized by its immediacy, accessibility, and the ability to engage audiences through multimedia and social sharing.

lightbulbAbout this topic

Key research themes

1. How can personalization methods predict and adapt to changing user interests for effective web news recommendation?

This research area investigates approaches for building personalized news recommendation systems that capture the dynamic nature of users’ news interests. As users consume news online, their preferences evolve rapidly owing to ongoing news events and individual changes, posing challenges for systems to accurately predict current interests and recommend relevant articles. Studies in this theme seek to model user behavior from click logs, incorporate collective news trends, and combine complementary recommendation techniques to enhance user engagement with web news.

Personalized News Recommendation Based on Click Behavior

by yvette tin

2017

Key finding: By analyzing anonymized Google News click logs, this paper revealed that while individual user interests vary over time, they generally follow broader news trends. Based on this, the authors developed a Bayesian framework... Read more

articleView Paper downloadDownload

2. What roles do interactivity, immediacy, and temporal dynamics play in shaping the production, presentation, and consumption of web news?

This theme explores how the digital nature of web news influences its distinct characteristics compared to traditional print news. Key features include interactivity—enabling user participation and modification of content; immediacy—allowing faster publication cycles without strict deadlines; and the complex temporalities associated with continuous news updates and accumulation of information online. Research investigates methodological challenges in analyzing these characteristics at the news story level, how time is textually constituted on news websites over years, and the consequences for journalistic practices and reader engagement.

FREEZING THE FLOW OF ONLINE NEWS

by Federico Gatti

2019

Key finding: The study identified methods to 'freeze' the continually updating flow of online news, enabling systematic content analyses focused on interactivity and immediacy at the news story level. Empirical comparison of online news... Read more

articleView Paper downloadDownload

The shifting temporalities of online news: The Guardian's website from 1996 to 2015

by Niels Brügger

2021

Key finding: This longitudinal analysis used archived versions of The Guardian’s website to reveal how news temporalities evolved alongside digital affordances. It demonstrated that online news constituted time through overlapping textual... Read more

articleView Paper downloadDownload

Journalism on the Web

by Michael Karlsson and

2016, Journalism Studies

Key finding: The paper conceptualized four core traits—interactivity, immediacy, hypertextuality, and multimodality—as defining features of web journalism. It highlighted the expansive research on interactivity, emphasizing how increased... Read more

articleView Paper downloadDownload

keyboard_arrow_downShow more

3. How do algorithms and code influence the curation, diversity, and distribution of web news, and what implications does this have for users and the news ecosystem?

This area investigates the embedded journalistic decisions and influence of algorithmic systems that curate web news content, impacting diversity, attention allocation, and ultimately public discourse. It includes audits of dominant algorithmic news curation engines, analyses of the code within news recommendation apps, and studies on the role of search engines in news distribution. These insights reveal the complex ways algorithms shape what news users see, how news is categorized and prioritized, and the potential biases or editorial decisions encoded within technological infrastructures.

Search as News Curator

by Daniel Trielli

2023, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Key finding: Through an audit of Google Search’s Top Stories box, this study found a strong concentration of news sources—just twenty sources contributed over half the impressions—indicating limited source diversity in algorithmic news... Read more

articleView Paper downloadDownload

Coding the News

by Matthew Weber and

2017, Digital Journalism

Key finding: Analyzing 59 open source mobile news apps, this work uncovered how embedded code functions as technological actants that filter and prioritize news, implementing journalistic decisions algorithmically. It demonstrated that... Read more

articleView Paper downloadDownload

Web News Extraction via Path Ratios

by Gongqing Wu

2013

Key finding: The authors introduced a novel web content extraction method using Text to Tag Path Ratio (TPR) and Extended TPR features based on DOM tree structures, enabling accurate differentiation of news content from non-news elements... Read more

articleView Paper downloadDownload

keyboard_arrow_downShow more

4. How do users integrate online and offline news sources in their consumption patterns, especially regarding news function and media combinations?

This research theme addresses the relationship between online and offline news consumption, analyzing whether users supplement or substitute traditional media with online sources. It explores user motivations and functions (e.g., broad overview, fast updates, background information) across different outlets, as well as patterns in combining multiple media types to compose personal news diets. Findings help to understand the complex media repertoires that shape modern news consumption and inform strategies for delivering news across platforms.

Investigating people’s news diets: How online users use offline news

by Klaus Schoenbach

2022, Communications

Key finding: Based on a large-scale survey in the Netherlands, the study found that users often employ offline and online news media for similar information functions such as obtaining a broad news overview or fast updates. Rather than... Read more

articleView Paper downloadDownload

5. How can educational settings leverage web news through support systems to facilitate News-In-Education (NIE) for elementary school learners?

This research area focuses on developing web news recommendation and browsing systems tailored to the needs of elementary school teachers and students within NIE programs. It includes methods to identify suitable web news articles based on readability, local relevance, and supportive features, as well as interface designs—such as map-based browsing—to enhance engagement and comprehension. These studies address pedagogical challenges such as article selection, language difficulty, and teacher workload, aiming to utilize web news effectively as teaching materials.

Examination of a Method for Determining Web News Suitable as Teaching Materials of NIE toward News Recommendation for Elementary School Teachers

by SDIWC Organization

2019, The Fifth International Conference on Electronics and Software Science (ICESS2019)

Key finding: By analyzing features from NIE worksheets and web news articles, this study developed a Support Vector Machine-based method incorporating Word2Vec representations and difficulty measures of words to accurately classify web... Read more

articleView Paper downloadDownload

Examination of a Method for Determining Web News Suitable for NIE using Features of NIE Worksheets

by SDIWC Organization

2019, The Fourth International Conference on Electronics and Software Science (ICESS2018)

Key finding: This paper analyzed 181 articles from NIE worksheets, focusing on content categories and readability. It identified characteristic features of suitable articles and proposed an SVM-based classification method to select web... Read more

articleView Paper downloadDownload

A Support System to Browse Web News using Google Maps for Elementary School Children in Japan

by SDIWC Organization

2015

Key finding: The authors developed a system that aggregates web news and displays them geographically on Google Maps, enabling elementary school students to interactively browse local news by area. The system addresses issues like news... Read more

articleView Paper downloadDownload

keyboard_arrow_downShow more

All papers in Web News

Tradução como contrapoder: representações de gênero em webnotícias

by Angelica Karim Garcia Simao

2024, Tradterm

Neste trabalho, analisamos a tradução de textos jornalísticos, entendendo a tradução como um fenômeno de “representação cultural” (ZIPSER E POLCHLOPEK, 2006), e observamos os critérios utilizados por tradutores para representar a... more

descriptionView Paper arrow_downwardDownload

Transgeneristas: representación discursiva en noticias virtuales de el Universal de Cartagena (2009 – 2012)

by Christian Howard Hooker

2023, Palobra

Esta investigación estudia la representación discursiva del sujeto transgenerista en las noticias publicadas en la versión web del periódico El Universal entre los años 2009 y 2012, a partir de un Análisis Crítico del Discursivo, para... more

descriptionView Paper arrow_downwardDownload

Tradução como contrapoder: representações de gênero em webnotícias

by Julia Diogo

2023, Tradterm

descriptionView Paper arrow_downwardDownload

Automatic Template Extraction from Heterogeneous Web Pages

by Sonali Rangdale

2023

In this paper, we will enlist the process of extracting template from heterogeneous Web Pages. Extracting structured information from semi-structured machine readable web pages automatically plays a major role these days, so some websites... more

descriptionView Paper arrow_downwardDownload

A study of diffusion and acceptance of an electronic newspaper: the case of an Ehime newspaper

by Shiro Uesugi

2023

The diffusion of portable network devices such as the iPad and tablet PCs, in addition to especially designed reading devices like Kindle, is transforming people's behaviour with printed media such as newspapers. Newspapers in electronic... more

descriptionView Paper arrow_downwardDownload

CETR - Content extraction via tag ratios

by Jiawei Han

2022

We present Content Extraction via Tag Ratios (CETR)a method to extract content text from diverse webpages by using the HTML document's tag ratios. We describe how to compute tag ratios on a line-by-line basis and then cluster the... more

descriptionView Paper arrow_downwardDownload

Methods For Extracting Content Blocks From Web Pages

by Rajashri Shinde

2022, International journal of engineering research and technology

The Web is perhaps the single largest data source in the world .The coverage of Web information is very wide and diverse. It has information which is of type required information by the user i.e. content blocks of the pages & the rest... more

descriptionView Paper arrow_downwardDownload

A study of diffusion and acceptance of an electronic newspaper: the case of an Ehime newspaper

by Shiro Uesugi

2022

The diffusion of portable network devices such as the iPad and tablet PCs, in addition to especially designed reading devices like Kindle, is transforming people’s behaviour with printed media such as newspapers. Newspapers in electronic... more

descriptionView Paper arrow_downwardDownload

CrimeStand

by Faizan Wajid

2021, Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems - GIS '16

Pursuing criminal activity is tied with understanding illegal or unlawful actions taken on opportunity within a geographic location. Mapping such activities can aid significantly in determining the health of a region, and the vicissitudes... more

descriptionView Paper arrow_downwardDownload

Methods For Extracting Content Blocks From Web Pages

by ashwini shinde

2021

descriptionView Paper arrow_downwardDownload

CLIFF-CLAVIN: Determining Geographic Focus for News Articles [Extended Abstract

by Rahul Bhargava

2021

The growing diversity of news sources available online has led to a significant methodological change in field of global news coverage. Studies of media attention and framing require sophisticated analytic tools to permit analysis of a... more

descriptionView Paper arrow_downwardDownload

Cultivation of Japan-France Exchange

by SDIWC Organization and

2019, The 8th International Conference on Informatics and Applications (ICIA2019)

This case study examines the process of revitalizing the once almost stagnated partnership between two universities in Japan and France. ERASMUS+ICM program has a great impact on the activating process, not only due to its financial... more

descriptionView Paper arrow_downwardDownload

A Case Study of Accepting an International Student from Turkey to Japan -The Trials of Advancing and Improving International Students Exchange Program

by SDIWC Organization and

2019, The 8th International Conference on Informatics and Applications (ICIA2019)

This case study focuses on the establishment of a new cooperative effort to accept a student enrolled at Middle East Technical University, Turkey who was nominated to Kagawa University, Japan under the Erasmus+ International Credit... more

Figure 1. Department of Computer Engineering syllabi

Figure 2. Computer Architecture Syllabus (in English)

Figure 3. Tentative Schedule for Fall 2019

7 LOGISTICAL ISSUES Table 2. List of Courses (Fall 2019) 7.1 Course Schedule, Travel Distance and Accommodation The nominee has applied to the Sanuki Program, and will take courses in Japanese language and culture, in addition to three courses in engineering [Table 2]. With the English syllabi for these courses, the nominee was able to gain approval from the Department of Computer Engineering.

descriptionView Paper arrow_downwardDownload

Multiclass Classification and Regression Analysis of Muscle Fatigue Using Wrist EMG

by SDIWC Organization

2019, The Fifth International Conference on Electronics and Software Science (ICESS2019)

Muscles can cause injury by training to improve physical performance. However, there are few ways to assess muscle fatigue currently. Therefore, in this paper, muscle fatigue is evaluated using surface EMG(ElectroMyoGram). The proposed... more

Figure 1. P-EMG plus type sensors are less expensive to run.

Figure 3. CNN network structure 2.4.2 Regression analysis Regression analysis is a kind of analysis in statistics. Methods of regression analysis are divided linear analysis and nonlinear analysis. In this paper, we used linear analysis. Linear analysis is a model that predicts the value of objective variables from the explanatory vari- ables. We used a least squares method to pre- dict values. Single regression analysis uses an explanatory variable, and multiple regression analysis uses two or more explanatory vari- ables. In this paper, we used multiple regres- sion analysis. Classification used all of the measured data converted to power spectral val- ues. However, regression analysis used data

Figure 7. Experimental data 3 days. Below is a data used for verification. tion operation was carried out 4 times a day for

Figure 8. Criteria for quantitative evaluation 3.2 Result

Table 1. Results obtained by Support Vector Machine (SVM)

Figure 9. Cross validation At first, we describe the result obtained by SVM. Below is how to use data for Cross vali- dation.

Table 3. Results obtained by Regression analysis 4 CONSIDERATION In this paper, we verified two ways based on classification and regression analysis.

Table 2. Results obtained by Convolutional Neural Net- work (CNN) 3.2.2 Regression analysis

Table 4. Total result From the above table, as a result, regression analysis obtained better accuracy than classifi- cation. Classification average resulted in low accuracy less than 35%, but regression anal- ysis average obtained more than 37% accu- racy. From these results, it is suggested that regression analysis is more useful for this pa- per. It is considered that the reason is that class classification learns and identifies dis- crete values, while regression analysis largely uses continuous values for learning discrimina- tion. Furthermore, result of regression analysis suggested that the state of muscle fatigue had some correlation with the state of decreasing number of muscle trainings.

descriptionView Paper arrow_downwardDownload

Examination of a Method for Determining Web News Suitable as Teaching Materials of NIE toward News Recommendation for Elementary School Teachers

by SDIWC Organization

2019, The Fifth International Conference on Electronics and Software Science (ICESS2019)

NIE (Newspaper in Education) is an educational approach that utilizes newspaper as teaching materials in classes. It is being implemented in elementary schools and junior high schools mainly. However, teachers’ workloads have increased by... more

However, considering of using news articles of NIE worksheets for the regional learning, only 14 local newspapers that publish NIE worksheets cannot cover all elementary schoo in Japan. This means that NIE cannot b implemented with the only existing NI worksheets. Therefore, it is necessary to recommend news articles suitable as teaching materials of NIE from Web news in various areas to the teachers. moa Mm Fig 1 Example of a worksheet without an article”!

Fig 2 Example of a worksheet with an article” Table 1 Investigation Result of Newspaper companies providing NIE worksheets

descriptionView Paper arrow_downwardDownload

Examination of a Method for Determining Web News Suitable for NIE using Features of NIE Worksheets

by SDIWC Organization

2019, The Fourth International Conference on Electronics and Software Science (ICESS2018)

NIE (Newspaper in Education) is an educational approach that utilizes newspaper as teaching materials in classes. It is being implemented in elementary schools and junior high schools mainly, however, adjusting time in general classes,... more

descriptionView Paper arrow_downwardDownload

Extraction of Product and Shop Names of Souvenirs from Blog Articles using Conditional Random Fields

by SDIWC Organization

2019, The Fourth International Conference on Electronics and Software Science (ICESS2018)

Demand for souvenirs that can be purchased only at the particular location or area is increasing, because anyone can purchase various souvenirs on online shopping sites. We call such souvenirs “local limited souvenirs” in this paper.... more

Table 1. Results labeled with baseline features for extracting names of products Table 2. Results labeled with baseline features for extracting names of shops

Table 3. Results labelled with baseline features and one of additional features for extracting unknown products and shops From Table 5, it turns out that the performances for unknown entities are improved when the method uses the baseline features and BIOES labeling scheme. It confirmed that the performance of extracting unknown entities is lower than that with the baseline features when the extraction method uses the baseline features and one of additional features other than “BIOES”. In particular, the performance of the method with "exBracket" or "correctPOS" for unknown entities is down about two or three points.

Table 4. Result labeled with baseline features, BIOES and inBracket

descriptionView Paper arrow_downwardDownload

Continuous Display of Images Searched by Keywords Extracted from a Brainstorming Session and Suggested by an Autosuggest Function

by SDIWC Organization and

2019, The Fourth International Conference on Electronics and Software Science (ICESS2018)

In this paper, we presented a system called SWISS (Search Websites’ Images using Search Suggestions) that searches images for brainstorming activities. Participants’ utterances are continuously input into SWISS and converted into text.... more

Figure 2. Position of the participants. Figure 1. System searches images with two words, which are extracted from a brainstorming and selected from candidate words with the autosuggest function.

descriptionView Paper arrow_downwardDownload

Modeling of Authors' Writing Styles to Detect Plagiarism in Japanese Academic Reports

by SDIWC Organization and

2019, The Fourth International Conference on Electronics and Software Science (ICESS2018)

In recent years, it has become easier for students to obtain well-written reports from the Internet in short time, or copy and paste the contents of their classmates' reports into their reports. Consequently, plagiarism that students... more

Figure 1. An example of trained writing model. The output probability indicates the probability of the report being produced by the author who provided training data for his/her writing mod- els. If a report is produced in a coding style which is close to the student’s one, the resulted output probability should be higher ratio. We represent a writing style of an author by a set of Np, writing models Mall defined as,

Figure 3. Basic Analysis: break down of token cate- gories for 16 Japanese report documents produced by each of the students.

Table 2. Examples of bias probability for state 2 of M7’. Table 3. Examples of bias probability for state 4 of 77’.

Table 4. Examples of transition probability for 1/77’.

make author identification instead of measur-

descriptionView Paper arrow_downwardDownload

A Survey of Web Information Extraction Systems

by Mos Zhang

2017

The Internet presents a huge amount of useful information which is usually formatted for its users, which makes it difficult to extract relevant data from various sources. Therefore, the availability of robust, flexible Information... more

descriptionView Paper arrow_downwardDownload

A Prototype System to Browse Web News using Maps for NIE in Elementary Schools in Japan

by SDIWC Organization and

2017

In many elementary schools around the world, NIE (Newspaper In Education) that uses newspapers as teaching tools has been implemented. However, the contents of newspaper articles are difficult for elementary school children. It is also... more

descriptionView Paper arrow_downwardDownload

Multilingual Improvement of an e-Learning System for Packet Routing Visualization

by SDIWC Organization and

2017

Kagawa university had already developed an e-Learning system with self-learning oriented facility of application program for the sake of studying network/communication/packet-controlling. Important facilities of that system were to... more

descriptionView Paper arrow_downwardDownload

A Support System for Extensive Reading using English News on the Web

by SDIWC Organization

2016

On language learning for English as a second language (ESL) learners, extensive reading is one of methods that can improve reading comprehension ability. Extensive reading is a method to read a lot of sentences that learners can fluently... more

descriptionView Paper arrow_downwardDownload

Using the DOM Tree for Content Extraction

by David Insa and

2015, Electronic Proceedings in Theoretical Computer Science

The main information of a webpage is usually mixed between menus, advertisements, panels, and other not necessarily related information; and it is often difficult to automatically isolate this information. This is precisely the objective... more

Figure 1: Blocks of a webpage from the New York Times website

of the distribution of the code between the lines of the HTML webpage, our technique is able to work with any webpage independently of how the webpage was generated or formatted.

The interesting property of this definition, is that it considers nodes as blocks where the internal information is grouped and indivisible using the DOM structure. Therefore, the CNR of an internal node, takes into account all the text and tags included in its descendants. Note also that the CNR of a node n, CNR(n), with a single child n; is always smaller than CNR(n;) because n can not contain text. However, if n has several children nj...n-, then CNR(n) can be greater than CNR(7) depending on the amount of text in the other children. This is very useful, because it allows us to detect blocks of relevant content, even if some nodes without text belong to the block.

Figure 4: CNRs of the DOM nodes associated with the HTML code in Example 3.2 After we have computed the CNRs we take the top rated nodes. Let us consider that the dark nodes in Figure [3| represent the top rated nodes. Then, we use Algorithm|2|to identify the most relevant blocks in the webpage. Initially, all the dark nodes are in the set of blocks. Then, because nodes 7 and 8 are brothers, in the first iteration, the algorithm removes nodes 7 and 8, and it adds node 6 to the set. In the second iteration, nodes 5 and 6 are removed, and node 2 is added. Finally, in the third iteration, nodes 2 and 3 are removed, and node I is added. Therefore, the final set of nodes computed by Algorithm {2 only contains the dashed nodes (1 and 4). The node that contains more text is selected as the block with the main content of the webpage. Observe that due to the structure of the DOM tree, the final node is often a container tag. Note also that all the nodes of this container are part of the final block, even if they do not contain text. Therefore, the final block is a block with all the information of the initial webpage that was placed together by the designer including related but non-textual elements such as images. The information of other blocks is not mixed with the information of the main block due to the structure of the DOM tree. For instance, node 4 corresponds to the footer, and it contains a lot of text. However, although it is textually adjacent in the source code to some nodes included in the main block; it is outside the container selected as the main content block (node 1). Therefore, node 4 is finally discarded. Algorithm 2 Identifying main content blocks

The first scenario is the most common situation. It happens when both the precision and the recall are 100%, thus the content extracted is exactly the main content. This situation happens around 50% of the times. As an example, observe in Figure|5}that the main webpage of Wikipedia has been filtered and both the horizontal and vertical menus have been removed. All the advertisements have been also removed. Figure 5: Main webpage of Wikipedia (left) and its filtered version (right)

Algorithm |1}recursively obtains the CNR of each node starting at the root node of the DOM tree. At each node it adds three new attributes to the node with the computed weight (weight), the number of characters it contains (textLength), and the CNR (CNR). The number of characters is computed ignoring special characters such as spaces or line breaks. This makes the algorithm independent of the formatting of the webpage (e.g., those webpages that organize the code using several spaces do not influence the CNRs). The algorithm distinguishes between three kinds of nodes, namely textNode which is a kind of DOM node that contains plain text and that is always a leaf, thus, it has weight 1; nonContentNode that repre- sents irrelevant nodes with a CNR of 0; and the rest of nodes that represent all kinds of tags. All methods (such as addAttribute) and attributes (such as innerText) used in the algorithm are standard in the DOM model and have the usual meaning.

descriptionView Paper arrow_downwardDownload

Web Template Extraction Based on Hyperlink Analysis

by Josep Silva and

2015, Electronic Proceedings in Theoretical Computer Science

Web templates are one of the main development resources for website engineers. Templates allow them to increase productivity by plugin content into already formatted and prepared pagelets. For the final user templates are also useful,... more

nodes and E is a set of edges between nodes in N (see Figure [I). root(T) denotes the root node of T.
Given a node n € N, link(n) denotes the hyperlink of n when n is a node that represents a hyperlink
(HTML label <a>). parent(n) represents node n’ € N such that (n’,n) € E. Similarly, children(n) repre-
sents the set {n’ € N | (n,n') € E}. subtree(n) denotes the subtree of T whose root is n € N. path(n) is
a non-empty sequence of nodes that represents a DOM path; it can be defined as path(n) = non, ...nn
such that Vi,0 <i < m.n; = parent(nj,1).
In order to identify the part of the DOM tree that is common in a set of webpages, our technique uses
an algorithm that is based on the notion of mapping. A mapping establishes a correspondence between
the nodes of two trees.

Example 4.1 Consider the BBC website. Two of its webpages are shown in Figure|2| In this website all
webpages share the same template, and this template has a main menu that is present in all webpages,
and a submenu for each item in the main menu. The site map of the BBC website may be represented
with the topology shown in Figure|3}

As in Definition we left the algorithm parametric with respect to the equality function =. This
is done on purpose, because this relation is the only parameter that is subjective and thus, it is a good
design decision to leave it open. For instance, a researcher can decide that two DOM nodes are equal
if they have the same label and attributes. Another researcher can relax this restriction ignoring some
attributes (i.e, the template can be the same, even if there are differences in colors, sizes, or even positions
of elements. It usually depends on the particular use of the extracted template). Clearly, = has a direct
influence on the precision and recall of the technique. The more restrictive, the more precision (and less

recall). Note also that the algorithm uses n, ~ ny to indicate that ny and ny maximize function =.
max

descriptionView Paper arrow_downwardDownload

Development of the Function which Collects Viewing History in the Support System for Dividing and Viewing Lecture Contents

by SDIWC Organization

2015

The authors develop the support system for dividing and viewing lecture contents by using lecture contents metadata. This system has index function, unit/ content viewing function, and play-list function. This system can divide the... more

descriptionView Paper arrow_downwardDownload

A Support System to Browse Web News using Google Maps for Elementary School Children in Japan

by SDIWC Organization

2015

Education that takes advantage of the newspaper as teaching materials called NIE (Newspaper In Education) is attracting attention in many elementary schools around the world. In general, teachers often take up the features and the events... more

descriptionView Paper arrow_downwardDownload

A Survey of Web Information Extraction Systems

by Mohammed Kayed

2015, IEEE Transactions on Knowledge and Data Engineering

Fig. 2. A Semi-structured page containing data records (in rectangular box) to be extracted. Muslea, who maintains the RISE (Repository of Online to the type of in traction patterns tha rules since the JE task HTML pages. Finally, t straints. Information Sources Used in Information Extraction Tasks) Web site, classified IE tools into 3 different classes according put documents and the struc- ture/constraints of the extraction patterns [11]. The first class includes tools that process IE from free text using ex- are mainly based on_ syntac- tic/semantic constraints. The second class is called Wrapper induction systems which rely on the use of delimiter-based processes online documents such as he third class also processes IE from online documents; however the patterns of these tools are based on both delimiters and syntactic/semantic con-

Fig. 3. Structurization of various documents.

The goal of WI is to automatically generate a wrapper that is used to extract the targets for an information resource. Let us consider the way how user interacts with WI sys- tems. Earlier systems are designed to facilitate program- mers in writing extraction rules, while later systems intro- duce machine learning for automatic rule generalization. Therefore, the user interaction has evolved from writing extraction rules to labeling target extraction data. In recent years, more efforts are devoted to reducing labeling and creating WI systems with unlabelled training examples. Following this trend, we can classify WI systems into the four classes manually-constructed TE Systems, supervised IE Systems, semi-supervised IE Systems and unsupervised IE Systems.

Fig. 7. A Minerva grammar in ENBF style.

Fig. 8. A WebOQL hypertree for the page p,, in Figure 4.

uses multi-pass scans to handle missing attributes and mul- inear for the he da Figure remaining examp ta structure of the ure 12(a), where some o 12(b). For examp racted whole by first applying which begins with “<ol iple permutations. The extraction rules are generated by using of a sequential covering algorithm, which starts from andmark automata to cover as many positive exam- ples as possible, and then tries to generate new automata es. A Stalker EC tree that describes running example is shown in Fig- f the extraction rules are shown in e, the reviewer ratings can be ex- the List(Reviewer) extraction rule >” and ends with “</ol>”) to the document, and t hen the Rating extraction rule to each individual reviewer, which is obtained by applying the iteration rule for List(Reviewer). In a way, STALKER is equivalent to multi-pass Softmealy [30]. However, the ex- traction patterns for each attribute can be sequential as op- posed to the continuous patterns used by Softmealy.

Fig. 12. An EC tree (a), and a Stalker extraction rule (b).

Fig. 13. Matching the first two pages of the running example (taken from [22)).

Fig. 14. The tag tree (left) and the DOM tree (as a comparison) for page p, in Figure 4.

ANALYSIS BASED ON THE TASK DOMAINS Non-HTML Support (NHS): The support for non-HTML inputs depends on the features or background knowledge used by the IE systems. Thus, when an IE system fails to generalize extraction rules for an IE task, we (the program- mers) know how to or what to adjust the system for such a task. Most supervised systems can support non-HTML documents by modifying the generalization hierarchy (e.g. Softmealy) or adding new token features (e.g. SRV). Manual systems such as Minerva and TSIMMIS, where extraction rules are written by hand, can be adapted by the wrapper developer to handle non-HTML documents. Some wrap- pers, e.g. WebOQL, W4F, XWrap, and DEPTA, rely heavily on the use of DOM trees information in their systems, so they cannot support non-HTML documents, while se- quence based approaches, such as IEPAD, OLERA, Road- Runner, and DeLa can be adapted to handle non-HTML documents by adding proper encoding schemes. The equivalence class technology of EXALG also supports non- HTML documents, but the success depends on token role differentiation. Extraction Level: IE tasks can be classified into four catego- ries: field-level, record-level, page-level and site-level. Ra- pier and SRV are designed to extract single-slot records, or equivalently field-level extractions. Wrappers in and RoadRunner extract the embedded data o EXALG bjects in whole pages which may contain records of multiple kindes, so wrappers in these systems are page-level. The maining systems in Table 1 are examples of record other re- -level IE tasks, although some can be extended for page-level extrac- tion, e.g. NoDoSE, STALKER, etc. Most record-level IE sys- tems discover record boundaries and then divide t hem into separate items, while the bottom-up extraction strategy in DEByE extracts a set of attributes and then assembles them to form a record. So far, there are no site-level IE systems.

ANALYSIS BASED ON THE TECHNIQUES USED context and the exact delimiters of the phrase to be extracted. DEByE and NoDoSE all require a large amount of support from users to model the data in the documents. They focus on the interface design and apply very simple methods to earn extraction patterns, ie. common prefix and suffix of he data values to be extracted. On the other hand, Stalker and SoftMealy use Ad-hoc generalization methods for earning extraction rules. They focus on the learning tech- niques and the extractor architecture and use a hierarchy of oken classes for token generalization, which is quite differ- ent from NoDoSE and DEByE where the extraction rules are simply based on superficial or literal words. TABLE 2 Features Used: Earlier IE systems are designed to handle non- emplate based Web pages, say computer science de- partment Web pages from various universities. Therefore, they iter-b possi have used both HTML tags and literal words as delim- ased constraints. For template-based Web pages, it is ble to use DOM tree paths to denote a specific piece of information in a Web page. For example, W4F, XWrap and other commercial products use DOM tree paths to address a We rule | used ALG b page. Since the data to be extracted are often co- ocated in the same path of the DOM tree, this makes the earning process much easier. For free text information extraction, natural language processing techniques such as part-of-speech tagger and Word-Net semantic classes are as additional features. SRV also uses orthographic features, token’s length, and link grammars. Finally, EX- exploits statistical information of the tokens in Web pages to generate their wrappers.

ANALYSIS BASED ON AUTOMATION DEGREE lection). For IEPAD and OLERA, although they require no labeling before pattern discovery, post-labeling is needed to sift desired data, while the work of distinguishing template tokens from data tokens is accomplished by unsupervised IE systems. Strictly speaking, the label of the data extracted by unsupervised IE systems remains to be assigned, and only DeLa has dealt with this problem.

descriptionView Paper arrow_downwardDownload

Criteria for Evaluating Information Extraction Systems

by Khaled Shaalan

2015, International Conference on Informatics and Systems

descriptionView Paper arrow_downwardDownload

A Survey of Web Information Extraction Systems

by Khaled Shaalan

2015, A Survey of Web Information Extraction Systems

descriptionView Paper arrow_downwardDownload

Methods For Extracting Content Blocks From Web Pages

by Rajashri Shinde

2015

An example of compressed structure tree is given in Figure 2, which compresses the DOM trees d1 and d2.

descriptionView Paper arrow_downwardDownload

A Survey of Web Data Extraction Techniques

by International Journal of Advance Research in Computer Science and Management Studies [IJARCSMS] ijarcsms.com

2014

extraction.

descriptionView Paper arrow_downwardDownload

Automatic Template Extraction from Heterogeneous Web Pages

by IJSRP Journal

2014

Figure 1: System model for template extraction Essential paths of document

descriptionView Paper arrow_downwardDownload

January2014Paper A1

by Lavanya Damodaram

2014

This paper probes various methods for extracting data from various HTML sites, web page that group several structured records, where we are going to look into the use of automatically generated wrappers. The huge volume of information on... more

descriptionView Paper arrow_downwardDownload

HTML Pattern Generator-Automatic Data Extraction from Web Pages

by Mihai Gabroveanu

2014

Existing methods of information extraction from HTML documents include manual approach, supervised learning and automatic techniques. The manual method has high precision and recall values but it is difficult to apply it for large number... more

descriptionView Paper arrow_downwardDownload

Web News Extraction via Path Ratios

by Gongqing Wu

2013

In addition to the news content, most web news pages also contain navigation panels, advertisements, related news links etc. These non-news items not only exist outside the news region, but are also present in the news content region.... more

websites’ layouts or structures. It is difficult for traditional models and techniques to distinguish non-news items from content segments. Let’s take a news article from the New York Times as an example, which appeared on October 23, 2012, as shown in Figure 1. This is a typical web news page. The content in the solid frame is web news, and the content in the dotted boxes is noise. Obviously, the banners, related links and advertisements occupy over one half of the page. Besides, we notice that the news in the solid frame also contains two non-news areas. In this paper, we propose a highly effective content extraction algorithm for distinguishing news content from non-news content effectively. Our news extraction technique is based on the potential relevance between web content layouts and their tag paths on the parsing trees, which is summarized as follows: on a typical web news page, news segments commonly have similar tag paths and contain long text and more punctuations, whereas the noise items usually have similar tag paths and contain brief text and less punctuations.

Figure 2. The DOM tree of Example 1 For convenience of description, this paper adopts the labeled ordered tree model to formalize an HTML page. According to the extraction demands, we extend the labeled ordered tree model, and meanwhile, ensure all operations could be operated on the DOM tree conveniently. If no confusion in the context, we treat the extended labeled ordered tree and the DOM tree as the same concept. Definition 1 (Extended Labeled Ordered Tree): L = {l, h, 15, ...} is a set of finite alphabetic elements, and each /; is a tag of the HTML language. The extended labeled ordered tree on L is defined as a 7-tuple expression T = (V, E, vo, <, L, U(*), c(*)). In this expression, V is a set of nodes, E is a set of edges, ve V, < is the sibling relationship set of nodes, and G = (V, E, vo, <) is an ordered tree with vo as the root. Mapping /: V — L is a label function, Vve V, I(v) is the label of node v. Mapping c: V > String is a content function, Vve V, c(v) represents the content of node v.

Figure 5. TPR of each node from the web news in Figure 1

Figure 6. ETPR of each node from the web news in Figure 1 A. CONTENT EXTRACTION Figure 6 shows the ETPR-histogram ordering by preorder traversal of the text nodes for the page in Figure 1. It is similar to Figure 5, the tag paths having a common ETPR value gather together, which indicates that content nodes gather together and noise nodes also gather together. Compared to Figure 5, ETPR is better suited for identifying web news because of the more obvious difference between sections. The difference between the two calculation methods is discussed in detail in Section 5.

Compared to Figure 6, H' shown in Figure 7 is better suited for distinguishing content segments from noise segments. H' has a lower variance because outlying peaks and valleys are smoothed. Similarly, the important short texts, such as internal-links, that may occupy a single low-ETPR node among many high-ETPR nodes, are smoothed to a higher value than the threshold. Therefore smoothing increases the cohesiveness within segments and also the differences between segments. 4.3 Extraction Algorithm CEPR

Figure 9. Precision, Recall and F-score tradeoff for NY Post as the threshold coefficient (4) increases from 0 to 2 After one webpage is parsed into a tree, we can calculate a TPR or ETPR histogram easily. Content nodes are often with high values and noise nodes have low values. But how to make a good use of high values from low values? In other words, how to distinguish content nodes from noise nodes? In this paper, we use a threshold technique. According to the above definitions of precision, recall and F-score, we find that with the increase of threshold 1, there might be a corresponding increase of precision and a sharp decrease of recall. Then how to select an available t to balance the recall and precision to get the highest F-score is considered to be a key issue. Common threshold selection methods include the middle value, the average value and the standard deviation. However, from ETPR histograms, we can figure out that the number of lower values in ETPR is far more than the higher ones. The middle value might be affected by low values, and the average value might be affected by high values, so we choose the standard deviation o to set the threshold. We set t=Jo, where 4 is a threshold parameter. Figure 9 shows the tradeoff of precision, recall and F-score along with 2 when extracting the source of the NY Post corpora. We can see that with the J increasing, the recall reduces and the precision increases. When A=0 all text nodes are

The results shown in Figure 10 demonstrate that the smoothed extractions outperform unsmoothed extraction. During the smoothing process, with the increase of a, the average P increases but average R decreases, and the highest average F (92.60%) is obtained when a is set as 1.

Table 1. The top 5 most popular news websites and their estimated unique monthly visitors Besides news content, a typical web news page contains title banners, advertisements, related links, copyrights and disclaimer notices. These additional non-news items, which are also known as noise, are responsible for roughly 40-50% of content on news websites [1]. Therefore, many web applications need to clean up news web pages. For instance, with highly-cleaned news inputted, the quality of web news summaries will be improved [2]; pocket- sized devices with small screens like mobile phones or PADs will achieve a better effect of user experience. In addition, the noise costs more data storage space and more computing time in web information retrieval, web content management and analysis, and reduces the quality of service at the same time.

CEPR is implemented in Java, and uses JTidy and HTMLParser components for cleaning and parsing web pages. Incorrect tags, wrong HTML coding and other errors that exist on web pages often make HTMLParser fail to parse. We use JTidy to fix these errors and translate them into standard XHTML documents. After parsing a webpage into a DOM tree, we calculate the text to tag path ratios by ignoring empty text nodes. In the extraction process. we remove the HTML tags, redundant blanks and empty lines from the content. Combining threshold setting with Gaussian smoothing weighted by the tag path edit distance together, we give our content extraction algorithm with the TPR/ETPR basis in Figure 8.

Table 2. Precision, Recall, F and Score for each algorithm on each source. W inners are in bold.

descriptionView Paper arrow_downwardDownload

Joint optimization of wrapper generation and template detection

by Di Wu

2013

Many websites have large collections of pages generated dynamically from an underlying structured source like a database. The data of a category are typically encoded into similar pages by a common script or template. In recent years,... more

descriptionView Paper arrow_downwardDownload

Joint optimization of wrapper generation and template detection

by Di Wu

2013

descriptionView Paper arrow_downwardDownload

Information extraction for search engines using fast heuristic techniques

by Eu-Gene Siew

2012, Data & Knowledge Engineering

We study the structured records of web pages and the relevant problems associated with the extraction and alignment of these structured records. Current automatic wrappers are complicated because they take into consideration the problems... more

descriptionView Paper arrow_downwardDownload

Web News

Key research themes

1. How can personalization methods predict and adapt to changing user interests for effective web news recommendation?

2. What roles do interactivity, immediacy, and temporal dynamics play in shaping the production, presentation, and consumption of web news?

3. How do algorithms and code influence the curation, diversity, and distribution of web news, and what implications does this have for users and the news ecosystem?

4. How do users integrate online and offline news sources in their consumption patterns, especially regarding news function and media combinations?

5. How can educational settings leverage web news through support systems to facilitate News-In-Education (NIE) for elementary school learners?

Related Topics

All papers in Web News