Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
The Web Data Model
Data Model
Unstructured Access to Structured Data
Web Data Warehousing
Preliminary Findings
Related Work
Conclusions
References
All Topics
Computer Science
Database Systems

Structured Querying of Web Text Data: A Technical Challenge

Profile image of Dan SuciuDan Suciu

2007, Cidr

Abstract

The Web contains a huge amount of text that is currently beyond the reach of structured access tools. This unstructured data often contains a substantial amount of implicit structure, much of which can be captured using information extraction (IE) algorithms. By combining an IE system with an appropriate data model and query language, we could enable structured access to all of the Web's unstructured data. We propose a general-purpose query system called the extraction database, or ExDB, which supports SQL-like structured queries over Web text. We also describe the technical challenges involved, motivated in part by our experiences with an early 90M-page prototype.

Figures (6)
Figure 1: The top-ranked results for a query to our EXxDB prototype. The query here is q(?a, ?b, ?c) :- invented(?a, ?b), died-in(?a, <year> ?c). This query took 30 seconds to process on a database of 90M Web pages. {mjc, chrisre, suciu, etzioni, banko}@cs.washington.edu
Figure 1: The top-ranked results for a query to our EXxDB prototype. The query here is q(?a, ?b, ?c) :- invented(?a, ?b), died-in(?a, <year> ?c). This query took 30 seconds to process on a database of 90M Web pages. {mjc, chrisre, suciu, etzioni, banko}@cs.washington.edu
Figure 2: Constructing the ExDB requires several processing steps. In step 1, we run information extractors over the downloaded web text, as described in Section 3. In step 2, the extracted information is stored in the ExDB data model, described in Section 2. Finally, applications can query the EXDB middleware and probabilistic RDBMS. Sections 4 and 5 describe query processing and possible applications.
Figure 2: Constructing the ExDB requires several processing steps. In step 1, we run information extractors over the downloaded web text, as described in Section 3. In step 2, the extracted information is stored in the ExDB data model, described in Section 2. Finally, applications can query the EXDB middleware and probabilistic RDBMS. Sections 4 and 5 describe query processing and possible applications.
Table 3: Top-10 prototype-ExDB results for q(?s) :- invented((scientist) ?s, ?x), and Google search results for scientist invented. The goal is to retrieve a list of practical-minded scientists. Only one document returnec by Google arguably contains an answer to the query; it is still embedded in unstructured text. Note that some of the ExDB entries are duplicates and should be merged; object synonyms will make this possible.
Table 3: Top-10 prototype-ExDB results for q(?s) :- invented((scientist) ?s, ?x), and Google search results for scientist invented. The goal is to retrieve a list of practical-minded scientists. Only one document returnec by Google arguably contains an answer to the query; it is still embedded in unstructured text. Note that some of the ExDB entries are duplicates and should be merged; object synonyms will make this possible.
Figure 4: Construction pipeline for the Schema Extraction Model. We still run an IE system over downloaded text, but use the resulting extractions to compute a single traditional relational database.
Figure 4: Construction pipeline for the Schema Extraction Model. We still run an IE system over downloaded text, but use the resulting extractions to compute a single traditional relational database.
Figure 5: The simple construction pipeline for the Text Query Model.
Figure 5: The simple construction pipeline for the Text Query Model.

Related papers

Structured queries over web text
Dan Suciu

2006

The Web contains a vast amount of text that can only be queried using simple keywords-in, documentsout search queries. But Web text often contains structured elements, such as hotel location and price pairs embedded in a set of hotel reviews. Queries that process these structural text elements would be much more powerful than our current document-centric queries. Of course, text does not contain metadata or a schema, making it unclear what a structured text query means precisely. In this paper we describe three possible models for structured queries over text, each of which implies different query semantics and user interaction.

View PDFchevron_right
HWPDE: Novel Approach for Data Extraction from Structured Web Pages
Manpreet Sehgal

2013

Diving into the World Wide Web for the purpose of fetching precious stones (relevant information) is a tedious task under the limitations of current diving equipments (Current Browsers). While a lot of work is being carried out to improve the quality of diving equipments, a related area of research is to devise a novel approach for mining. This paper describes a novel approach to extract the web data from the hidden websites so that it can be used as a free service to a user for a better and improved experience of searching relevant data. Through the proposed method, relevant data (Information) contained in the web pages of hidden websites is extracted by the crawler and stored in the local database so as to build a large repository of structured and indexed and ultimately relevant data. Such kind of extracted data has a potential to optimally satisfy the relevant Information starving end user.

View PDFchevron_right
A Framework for Extracting Information from Semi-Structured Web Data Sources
Shadi Hasan

2008

Nowadays, many users use web search engines to find and gather information. User faces an increasing amount of various semi-structured information sources. The issue of correlating, integrating and presenting related information to users becomes important. When a user uses a search engine such as Yahoo and Google to seek a specific information, the results are not only information about the availability of the desired information, but also information about other pages on which the desired information is mentioned. The number of selected pages is enormous. Therefore, the performance capabilities, the overlap among results for the same queries and limitations of web search engines are an important and large area of research. Extracting information from the web data sources also becomes very important because the massive and increasing amount of diverse semi-structured information sources in the Internet that are available to users, and the variety of web pages making the process of information extraction from web a challenging problem. This paper proposes a framework for extracting, classifying and browsing semi-structured web data sources. The framework is able to extract relevant information from different web data sources, and classify the extracted information based on the standard classification of Nokia products.

View PDFchevron_right
EXTRACT AND ANALYSIS OF SEMI STRUCTURED DATA FROM WEBSITES AND DOCUMENTS
prashant ahire, Yogesh Pawar

Discovering into the W3 consortium and portable documents for the purpose of fetching useful information is a hectic task under the limitations of current available browsers. While a huge amount of work is being carried out to improve the efficiency. The huge amount of information on web and portable digital document is stored in backend databases which are not indexed by traditional search engines such databases are known as Semi structured Databases and extraction and analysis of web content and documents is a time consuming and complex task. Hence, there has been increased interest in retrieval and integration of semi structured web data and digital document data with a view to improve quality information to the users who wish to analyze the data. This paper states an approach that identifies web page templates and the tag structures of a portable document in order to sort semi structured data from web pages and documents and analyze the fetching data as per user requirement using various SQL queries. Keyword: Web page extraction, Analysis, Web Page Service, portable documents

View PDFchevron_right
Information Extraction From Semi-structured and Unstructured Sources
Stefan Dumitrescu
View PDFchevron_right
DEiXTo: A Web Data Extraction Suite
Fotis Kokkoras, Nick Bassiliades

Web data extraction (or web scraping) is the process of collecting unstructured or semi-structured information from the World Wide Web, at different levels of automation. It is an important, valuable and practical approach towards web reuse while at the same time can serve the transition of the web to the semantic web, by providing the structured data required by the latter. In this Paper we present DEiXTo, a web data extraction suite that provides anarsenal of features aiming at designing and deploying wellengineered extraction tasks. We focus on presenting the core pattern matching algorithm and the overall architecture, which allows programming of custom-made solutions for hard extraction tasks. DEiXTo consists of both freeware and open source components.

View PDFchevron_right
Web Data Extraction, Applications and Techniques: A Survey
linda wijaya

The World Wide Web contains a huge amount of unstructured and semi-structured information, that is exponentially increasing with the coming of the Web 2.0, thanks to User-Generated Contents (UGC). In this paper we intend to briefly survey the fields of application, in particular enterprise and social applications, and techniques used to approach and solve the problem of the extraction of information from Web sources: during last years many approaches were developed, some inherited from past studies on Information Extraction (IE) systems, many others studied ad hoc to solve specific problems.

View PDFchevron_right
Information Extraction: Distilling Structured Data from Unstructured Text
Andrew McCallum
View PDFchevron_right
Extracting Semistructured Information from the Web
Hèctor Garcia

1997

We describe a configurable tool for extracting semistructured data from a set of HTML pages and for converting the extracted information into database objects. The input to the extractor is a declarative specification that states where the data of interest is located on the HTML pages, and how the data should be "packaged" into objects. We have implemented the Web extractor using the Python programming language stressing efficiency and ease-of-use. We also describe various ways of improving the functionality of our current prototype. The prototype is installed and running in the TSIMMIS testbed as part of a DARPA I 3 (Intelligent Integration of Information) technology demonstration where it is used for extracting weather data form various WWW sites.

View PDFchevron_right
WEIDJ: Development of a new algorithm for semi-structured web data extraction
TELKOMNIKA JOURNAL

TELKOMNIKA Telecommunication Computing Electronics and Control, 2021

In the era of industrial digitalization, people are increasingly investing in solutions that allow their process for data collection, data analysis and performance improvement. In this paper, advancing web scale knowledge extraction and alignment by integrating few sources by exploring different methods of aggregation and attention is considered in order focusing on image information. The main aim of data extraction with regards to semi-structured data is to retrieve beneficial information from the web. The data from web also known as deep web is retrievable but it requires request through form submission because it cannot be performed by any search engines. As the HTML documents start to grow larger, it has been found that the process of data extraction has been plagued with lengthy processing time. In this research work, we propose an improved model namely wrapper extraction of image using document object model (DOM) and JavaScript object notation data (JSON) (WEIDJ) in response to the promising results of mining in a higher volume of image from a various type of format. To observe the efficiency of WEIDJ, we compare the performance of data extraction by different level of page extraction with VIBS, MDR, DEPTA and VIDE. It has yielded the best results in Precision with 100, Recall with 97.93103 and F-measure with 98.9547.

View PDFchevron_right

References (41)

  1. E. Agichtein and L. Gravano. Snowball: Extracting relations from large plain-text collections. In Proceedings of the 2001 ACM SIGMOD International Conference on Digital Libraries, 2000.
  2. S. Agrawal, S. Chaudhuri, and G. Das. Dbxplorer: A system for keyword-based search over relational databases. In ICDE, 2002.
  3. P. Andritsos, A. Fuxman, and R. Miller. Clean answers over dirty databases: A probabilistic approach. In ICDE, 2006.
  4. M. Banko, M. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. Open information extraction from the web. In IJCAI, 2007.
  5. O. Benjelloun, A. D. Sarma, A. Halevy, and J. Widom. Uldbs: Databases with uncertainty and lineage. In VLDB, 2006.
  6. S. Brin. Extracting patterns and relations from the world wide web. In WebDB Workshop at EDBT '98, 1998.
  7. P. Buneman, S. Khanna, and W. C. Tan. Why and where: A characterization of data provenance. In ICDT, pages 316-330, 2001.
  8. M. Cafarella and O. Etzioni. A search engine for natural language applications. In WWW, 2005.
  9. A. Chandel, P. C. Nagesh, and S. Sarawagi. Efficient batch top-k search for dictionary-based entity recognition. In ICDE, 2006.
  10. K. C.-C. Chang, B. He, and Z. Zhang. Toward large scale integration: Building a metaquerier over databases on the web. In CIDR, pages 44-55, 2005.
  11. J. Cho and S. Rajagopalan. A fast regular expression indexing engine. In ICDE, 2002.
  12. W. Cohen. Information extraction and integration: An overview, 2004.
  13. Y. Cui and J. Widom. Practical lineage tracing in data warehouses. In ICDE, pages 367-378, 2000.
  14. N. Dalvi, C. Ré, and D. Suciu. Query evaluation on probabilistic databases. In IEEE Data Engineering Bulletin, 2006.
  15. N. Dalvi and D. Suciu. Efficient query evaluation on probabilistic databases. In VLDB, 2004.
  16. A. Doan, R. Ramakrishnan, F. Chen, P. DeRose, Y. Lee, R. McCann, M. Sayyadian, and W. Shen. Community information management. In IEEE Data Engineering Bulletin, 2006.
  17. A. Doan, R. Ramakrishnan, and S. Vaithyanathan. Managing information extraction, 2006.
  18. X. Dong, A. Halevy, and J. Madhavan. Reference reconciliation in complex information spaces. In SIGMOD, 2005.
  19. D. Downey, S. Soderland, and O. Etzioni. A probabilistic model of redundancy in information extraction. In IJCAI, 2005.
  20. O. Etzioni, M. Cafarella, D. Downey, S. Kok, A. Popescu, T. Shaked, S. Soderland, D. Weld, and A. Yates. Unsupervised named-entity extraction from the web: An experimental study. Artificial Intelligence, 165(1):91-134, 2005.
  21. R. Fagin, A. Lotem, and M. Naor. Optimal aggregation algorithms for middleware. J. Comput. Syst. Sci., 66(4):614-656, 2003.
  22. C. Fellbaum. English verbs as a semantic net. International Journal of Lexicography, 3(4):278-301, 1990.
  23. L. Gravano, P. G. Ipeirotis, H. V. Jagadish, N. Koudas, S. Muthukrishnan, and D. Srivastava. Approximate string joins in a database (almost) for free. In VLDB, pages 491-500, 2001.
  24. M. Gubanov and P. A. Bernstein. Structural text search and comparison using automatically extracted schema. In WebDB, 2006.
  25. A. Y. Halevy, O. Etzioni, A. Doan, Z. G. Ives, J. Madhavan, L. McDowell, and I. Tatarinov. Crossing the structure chasm. In CIDR, 2003.
  26. V. Hristidis and Y. Papakonstantinou. Discover: Keyword search in relational databases. In VLDB, 2002.
  27. T. Jayram, R. Krishnamurthy, S. Raghavan, S. Vaithyanathan, and H. Zhu. Avatar information extraction system. In IEEE Data Engineering Bulletin, 2006.
  28. D. Konopnicki and O. Shmueli. W3QS -A System for WWW Querying. In 13th International Conference on Data Engineering (ICDE'97), 1997.
  29. D. Lin and P. Pantel. Discovery of inference rules from text. In KDD, pages 323-328, 2001.
  30. J. Liu, X. Dong, and A. Halevy. Answering structured queries on unstructured data. In WebDB, 2006.
  31. I. R. Mansuri and S. Sarawagi. Integrating unstructured data into relational databases. In ICDE, 2006.
  32. A. O. Mendelzon, G. A. Mihalia, and T. Milo. Querying the World Wide Web. International Journal on Digital Libraries, 1996.
  33. G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. Miller. Introduction to wordnet: An on-line lexical database. International Journal of Lexicography, 3(4):235-312, 1990.
  34. A. Natsev, Y.-C. Chang, J. R. Smith, C.-S. Li, and J. S. Vitter. Supporting incremental join queries on ranked inputs. In VLDB, pages 281-290, 2001.
  35. D. V. K. Reynold Cheng and S. Prabhakar. Evaluating probabilistic queries over imprecise data. In SIGMOD, pages 551-562, 2003.
  36. A. D. Sarma, O. Benjelloun, A. Halevy, and J. Widom. Working models for uncertain data. In ICDE, April 2006.
  37. Y. Shinyama and S. Sekine. Preemptive information extraction using unrestricted relation discovery. In Proceedings of the Human Language Technology and North American Chapter of Association of Computational Linguistics Conference (HLT/NAACL-06), 2006.
  38. E. Spertus and L. A. Stein. Squeal: A Structured Query Language for the Web. In WWW, pages 95-103, 2000.
  39. M. Theobald, G. Weikum, and R. Schenkel. Top-k query evaluation with probabilistic guarantees. In VLDB, pages 648-659, 2004.
  40. P. D. Turney. Expressing implicit semantic relations without supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL-2006), 2006.
  41. J. Widom. Trio: A system for integrated management of data, accuracy, and lineage. In CIDR, pages 262-276, 2005.

Related papers

Structured querying of Web text
Dan Suciu

2007

The Web contains a huge amount of text that is currently beyond the reach of structured access tools. This unstructured data often contains a substantial amount of implicit structure, much of which can be captured using information extraction (IE) algorithms. By combining an IE system with an appropriate data model and query language, we could enable structured access to all of the Web's unstructured data. We propose a general-purpose query system called the extraction database, or ExDB, which supports SQL-like structured queries over Web text. We also describe the technical challenges involved, motivated in part by our experiences with an early 90M-page prototype.

View PDFchevron_right
WEB-SCALE INFORMATION EXTRACTION FROM UNSTRUCTURED AND UNGRAMMATICAL DATA SOURCES
TJPRC Publication

TJPRC, 2014

Information Extraction (IE) is the task of automatically extracting knowledge from text. The massive body of text now available on the World Wide Web presents an unprecedented opportunity for information extraction. However, information extraction on the Web is challenging due to the enormous variety of distinct concepts and structured expressed. The explosive growth and popularity of the worldwide web has resulted in a huge amount of information sources on the Internet. However, due to the heterogeneity and the lack of structure of Web information sources, access to this huge collection of information has been limited to browsing and searching. Information extraction from unstructured and ungrammatical text on the Web, such as classified Ads, Auction listings, and web postings forums. Since the data is unstructured and ungrammatical, this information extraction precludes the use of rule-based methods that rely on consistent structures within the text or natural language processing techniques that rely on grammar. Posts are full of useful information, as defined by the attributes that compose the entity within the post. Currently accessing the data within posts does not go much beyond keyword search. This is precisely because the ungrammatical and unstructured nature of posts makes extraction difficult, so the attributes remain embedded within the posts. These data sources are ungrammatical, since they do not conform to the proper rules of written language. Therefore, Natural Language Processing (NLP) based information extraction techniques are not appropriate. As more and more information comes online, the ability to process and understand this information becomes more and more crucial. Data integration attacks this problem by letting users query heterogeneous data sources within a unified query framework, combining the results to ease understanding. However, while data integration can integrate data from structured sources such as databases, semi-structured sources such as that extracted from Web pages, and even Web Services, this leaves out a large class of useful information: unstructured and ungrammatical data sources. We proposed a system based Machine Learning technique to obtain the structured data records from different unstructured and non-template based websites. The proposed approach will be implemented by collection of known entities and their attributes, which refer as “reference set," A reference set can be constructed from structured sources, such as databases, or scraped from semi-structured sources such as collections of Web pages. A reference set can even be constructed automatically from the unstructured, ungrammatical text itself. This project implements methods to exploit reference sets for extraction using machine learning techniques. The machine learning approach provides higher accuracy extractions and deals with ambiguous extractions, although at the cost of requiring human effort to label training data.

View PDFchevron_right
Just-in-time information extraction using extraction views
Mina Farid

Proceedings of the ACM SIGMOD International Conference on Management of Data, 2012

View PDFchevron_right
Information Extraction in Semantic, Highly-Structured, and Semi-Structured Web Sources
Juan Manuel Santos Gago

Polibits, 2014

The evolution of the Web from the original proposal made in 1989 can be considered one of the most revolutionary technological changes in centuries. During the past 25 years the Web has evolved from a static version to a fully dynamic and interoperable intelligent ecosystem. The amount of data produced during these few decades is enormous. New applications, developed by individual developers or small companies, can take advantage of both services and data already present on the Web. Data, produced by humans and machines, may be available in different formats and through different access interfaces. This paper analyses three different types of data available on the Web and presents mechanisms for accessing and extracting this information. The authors show several applications that leverage extracted information in two areas of research: recommendations of educational resources beyond content and interactive digital TV applications.

View PDFchevron_right
Scaling the Information Extraction from Unstructured and Ungrammatical Data Sources on Web
shubhangi vaikole

2014

Information Extraction (IE) on the web is the task of automatically extracting knowledge from text. Web Information Extraction (WIE) systems have recently been able to extract massive quantities of relational data from online text. This massive body of text which are now available on the World Wide Web do presents an unparalleled opportunity for information extraction. However, this information extraction on the Web is challenging due to the vast variety of distinct concepts and structured expressed. The explosive growth and popularity of the worldwide web has resulted in a huge amount of information sources on the Internet. However, due to the heterogeneity, diversity and the lack of structure of Web information sources, access to this huge collection of information has been limited to browsing and searching.

View PDFchevron_right

Related topics

  • Structured Query Language
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved