Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
New Hypothesis
Conclusion
References

A theory of information matching

Profile image of Jagadeesh GorlaJagadeesh Gorla

Abstract

* The theory and the mathematical modelling presented in this report has not been published elsewhere. However, different applications of the theory are under review.

Related papers

Matching data detection for the integration system
International Journal of Electrical and Computer Engineering (IJECE), M. El Abassi

International Journal of Electrical and Computer Engineering (IJECE), 2023

The purpose of data integration is to integrate the multiple sources of heterogeneous data available on the internet, such as text, image, and video. After this stage, the data becomes large. Therefore, it is necessary to analyze the data that can be used for the efficient execution of the query. However, we have problems with solving entities, so it is necessary to use different techniques to analyze and verify the data quality in order to obtain good data management. Then, when we have a single database, we call this mechanism deduplication. To solve the problems above, we propose in this article a method to calculate the similarity between the potential duplicate data. This solution is based on graphics technology to narrow the search field for similar features. Then, a composite mechanism is used to locate the most similar records in our database to improve the quality of the data to make good decisions from heterogeneous sources.

View PDFchevron_right
MEDD: An Approximate Matching Technology for Database Searching, Linking, and De-Duplicating
Arthur Goldberg

2001

When you need to combine multiple, error-filled data feeds into a single, highly accurate database, the hardest problem is matching corresponding records. How do you match, for instance, "Thomas J. Hanks" with "Tom Hank" or "International Business Machines" with "Intl. Bus. Mach."? We present an innovative, accurate system that employs a powerful, patentpending, machine learning technique to determine the probability that two database records correspond to the same person or company. We start by showing why record matching is such a difficult problem and describe the basics of the record matching process. As an example, we discuss the New York City Department of Health, where we removed 300,000 duplicate records from a 2.1 million record children's health database. MEDD is built around "comparison functions". Comparison functions check whether a pair of records has a certain matching or non-matching characteristic. Examples include "First names match", "First names match using the 'Soundex' phoneticization technique", or "Birthday does not match". MEDD uses a training process called "maximum entropy modeling" to infer the relative importance of the different comparison functions from a small set of record-pairs which have been hand-marked as "same" or "different". Out of this process comes a "weight" which is assigned to each feature. At runtime, MEDD operates as a function which takes a set of fields (a "search record") as an input and returns a list of database ID's which might match the search record. The ID's are ranked by a probability of match which is computed by MEDD's weighted comparison functions.

View PDFchevron_right
Semantic matching
Pavel Shvaiko

The Knowledge Engineering Review, 2003

We think of match as an operator that takes two graph-like structures (e.g. database schemas or ontologies) and produces a mapping between elements of the two graphs that correspond semantically to each other. The goal of this paper is to propose a new approach to matching, called semantic matching. As its name indicates, in semantic matching the key intuition is to exploit the model-theoretic information, which is codified in the nodes and the structure of graphs. The contributions of this paper are (i) a rational reconstruction of the major matching problems and their articulation in terms of the more generic problem of matching graphs, (ii) the identification of semantic matching as a new approach for performing generic matching and (iii) a proposal for implementing semantic matching by testing propositional satisfiability.

View PDFchevron_right
A Concept-Based Approach for Entity Matching-IJAERDV04I0965828.pdf
Editor IJAERD

One of the central problems of database integration is entity matching, that is, the identification of similar data elements in two or more databases or other data sources. This approach considers the correspondences across the attributes of various databases. Moreover, it uses different matchers to combines multiple databases. To solve the heterogeneity matching problem, we have proposed an improved approach for entity matching to increase accuracy in matching databases.

View PDFchevron_right
Record Matching to Improve Data Quality
Vassilios Verykios

1999

Data Quality is de ned in TB98] as tness for use, which implies that quality is relative to the use of data. Problems with data quality tend to fall into two categories: inconsistency among systems and inconsistency with reality. Format/syntax, semantic and value inconsistencies are representative of inconsistency among systems whereas incorrect and missing values are representative of inconsistencies with reality.

View PDFchevron_right
A generalized cost optimal decision model for record matching
Vassilios Verykios

Proceedings of the 2004 international workshop on Information quality in informational systems - IQIS '04, 2004

Record (or entity) matching or linkage is the process of identifying records in one or more data sources, that refer to the same real world entity or object. In record linkage, the ultimate goal of a decision model is to provide the decision maker with a tool for making decisions upon the actual matching status of a pair of records (i.e., documents, events, persons, cases, etc.). Existing models of record linkage rely on decision rules that minimize the probability of subjecting a case to clerical review, conditional on the probabilities of erroneous matches and erroneous non-matches. In practice though, (a) the value of an erroneous match is, in many applications, quite different from the value of an erroneous non-match, and (b) the cost and the probability of a misclassification, which is associated with the clerical review, is ignored in this way. In this paper, we present a decision model which is optimal, based on the cost of the record linkage operation, and general enough to accommodate multi-class or multi-decision case studies. We also present an example along with the results from applying the proposed model to large comparison spaces.

View PDFchevron_right
Similarity matching
Ramesh Jain

Recent Developments in Computer Vision, 1996

Image databases will force us to rethink many of the concepts that led us so far. One of these is matching. We argue that the fundamental operation in a content-indexed image database should not be matching the query against the images in the database in search of a "target" image that best matches the query. The basic operation in query-by-content will be ranking portions of the database with respect to similarity with the query. What kind of similarity measure should be used is a problem we begin exploring in this paper. We let psychological experiments guide us in the quest for a good similarity measure, and devise a measure derived from a set-theoretic measure proposed in the psychological literature, modified by the introduction of fuzzy logic.

View PDFchevron_right
The Field Matching Problem: Algorithms and Applications
Alvaro Monge

1996

To combine information from heterogeneous sources, equivalent data in the multiple sources must be identied. This task is the eld matching problem. Specically, the task is to determine whether or not two syntactic values are alternative designations of the same semantic entity. For example the addresses and UCSD, Computer Science and Engineering Department, CA 92093-0114 do designate the same department. This paper describes three eld matching algorithms, and evaluates their performance on real-world datasets. One proposed method is the well-known Smith-Waterman algorithm for comparing DNA and protein sequences. Several applications of eld matching in knowledge discovery are described brie y, including webfind, which is a new software tool that discovers scienti c papers published on the worldwide web. webfind uses external information sources to guide its search for authors and papers. Like many other worldwide web tools, webfind needs to solve the eld matching problem in order to navigate between information sources.

View PDFchevron_right
Automatic threshold estimation for data matching applications
Leandro Krug Wives

2010

Several advanced data management applications, such as data integration, data deduplication, and similarity querying rely on the application of similarity functions. A similarity function requires the definition of a threshold value in order to decide whether two different data instances match, ie, if they represent the same real world object. In this context, threshold definition is a central problem. This paper proposes a method for estimating the quality of a similarity function.

View PDFchevron_right
Statistical matching of multiple sources: A look through coherence
Barbara Vantaggi

International Journal of Approximate Reasoning, 2008

In several applications there is the need to consider different data sources and to integrate information: a specific case is the so-called statistical matching, where data sources have just a set of common variables and inference is required on the other variables. The traditional way to cope with such situations is to combine the available data with assumptions strong enough to identify pointwise the joint probability. Such assumptions cannot always be justified and inference should take into account all the set of compatible probabilities. In this paper, we show how statistical matching problems can be managed by means of coherent conditional probability: coherence allows us to combine the knowledge coming from different multiple sources, included those given from field experts, without necessarily assuming further hypothesis. Moreover, inferences and decisions can be dealt with by taking in consideration also logical constraints among the variables, which arise naturally in the applications. An example showing advantages and drawbacks of the proposed method is given.

View PDFchevron_right

References (28)

  1. BLEI, D. M., NG, A. Y., AND JORDAN, M. I. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (Mar. 2003).
  2. BRIN, S., AND PAGE, L. The anatomy of a large-scale hypertextual web search engine. In WWW (1998).
  3. BURGES, C. J. C., RAGNO, R., AND LE, Q. V. Learning to rank with nonsmooth cost functions. In NIPS (2006), pp. 193-200.
  4. CRASWELL, N., ROBERTSON, S., ZARAGOZA, H., AND TAYLOR, M. Relevance weight- ing for query independent evidence. In SIGIR (2005), SIGIR '05.
  5. DEMPSTER, A., LAIRD, N., AND RUBIN, D. Maximum likelihood from incomplete data via the em algorithm. J. Royal Statistical Society, Series B (1977).
  6. DIEBOLT, J., AND ROBERT, C. P. Estimation of finite mixture distributions through bayesian sampling. J. R. Statist. Soc B, 2 (1994), 363-375.
  7. GAO, J., YUAN, W., LI, X., DENG, K., AND NIE, J.-Y. Smoothing clickthrough data for web search ranking. In SIGIR (2009), pp. 355-362.
  8. HARTER, S. A probabilistic approach to automatic keyword indexing. Journal of the Amer- ican Society for Information Retrieval Science (1975).
  9. HOFMANN, T. Probabilistic latent semantic indexing. In SIGIR (1999), SIGIR '99.
  10. HOFMANN, T. Latent semantic models for collaborative filtering. ACM Trans. Inf. Syst. (2004).
  11. HUANG, X., AND CROFT, W. B. A unified relevance model for opinion retrieval. In In CIKM (2009), CIKM '09.
  12. JONES, K. S. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28 (1972), 11-21.
  13. KOREN, Y., BELL, R. M., AND VOLINSKY, C. Matrix factorization techniques for recom- mender systems. IEEE Computer (2009).
  14. LAFFERTY, J. D., AND ZHAI, C. Document language models, query models, and risk minimization for information retrieval. In SIGIR (2001), pp. 111-119.
  15. MARON, M. E., AND KUHNS, J. L. On relevance, probabilistic indexing and information retrieval. J. ACM (1960).
  16. MEI, Q., FANG, H., AND ZHAI, C. A study of poisson query generation model for infor- mation retrieval. In In SIGIR (2007), SIGIR '07.
  17. MIZARRO, S. Relevance: The whole history. Journal of the American Society for Informa- tion Science 48(9) (1997), 321-343.
  18. PONTE, J. M., AND CROFT, W. B. A language modeling approach to information retrieval. In SIGIR (1998).
  19. ROBERTSON, S. The unified model revisited. In SIGIR 2003 Workshop on Mathemati- cal/Formal Models in Information Retrieval (2003).
  20. ROBERTSON, S., MARON, M. E., AND COOPER, W. S. Probability of relevance: a unifi- cation of two competing models for document retrieval. Information Technology: Research and Development (1982).
  21. ROBERTSON, S., AND SPARK JONES, K. Relevance weighting of search terms. Journal of the American Society for Information Science (1976).
  22. ROBERTSON, S., AND ZARAGOZA, H. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. (2009).
  23. ROBERTSON, S. E., VAN RIJSBERGEN, C. J., AND PORTER, M. F. Probabilistic models of indexing and searching. In SIGIR (1980).
  24. ROBERTSON, S. E., AND WALKER, S. Some simple effective approximations to the 2- poisson model for probabilistic weighted retrieval. In SIGIR (1994).
  25. SARWAR, B., KARYPIS, G., KONSTAN, J., AND REIDL, J. Item-based collaborative filter- ing recommendation algorithms. In WWW (2001).
  26. SCHOLER, F., AND WILLIAMS, H. E. Query association for effective retrieval. In CIKM (2002), CIKM '02.
  27. SHARDANAND, U., AND MAES, P. Social information filtering: algorithms for automating ẅord of mouth ¨. In CHI'95 (1995).
  28. WANG, J., DE VRIES, A. P., AND REINDERS, M. J. Unified relevance models for rating prediction in collaborative filtering. ACM Trans. on Information System (TOIS) (2008).

Related papers

A Unified Theory of Information Matching
Stephen Robertson

2012

In this work, we propose a theory for information matching. It is motivated by the observation that retrieval is about the relevance matching between two sets of properties (features), namely, the information need representation and information item representation. However, many probabilistic retrieval models rely on fixing one representation and optimizing the other (e.g. fixing the single information need and tuning the document) but not both. Therefore, it is difficult to use the available related information on both the document and the query at the same time in calculating the probability of relevance. In this paper, we address the problem by hypothesizing the relevance as a logical relationship between the two sets of properties; the relationship is defined on two separate mappings between these properties. By using the hypothesis we develop a unified probabilistic relevance model which is capable of using all the available information. We validate the proposed theory by formulating and developing probabilistic relevance ranking functions for both ad-hoc text retrieval and collaborative filtering. Our derivation in text retrieval illustrates the use of the theory in the situation where no relevance information is available. In collaborative filtering, we show that the resulting recommender model unifies the user and item information into a relevance ranking function without applying any dimensionality reduction techniques or computing explicit similarity between two different users (or items), in contrast to the state-of-the-art recommender models.

View PDFchevron_right
Automatic record matching in cooperative information systems
Luca De Santis

2003

The paper describes an algorithm for automatic record matching in cooperative information systems. The algorithm is based on the well-known Sorted Neighborhood Method. The original contribution is on the proposal of a method to automatically choose the key for record matching. The performances of the algorithm are compared with previous literature results; the algorithm detects a higher percentage of correct duplicates and a lower percentage of false positive, on a single key sorting pass.

View PDFchevron_right
Adaptive Name Matching in Information Integration
pradeep a

IEEE Expert / IEEE Intelligent Systems, 2003

I n f o r m a t i o n I n t e g r a t i o n o n t h e W e b (s, t, i, j), which is the edit distance between the first i letters in s and the first j letters in t. Let s i denote the ith letter of s, and, similarly, let t j be the jth letter of t. It's easy to see that we can define D(s, t, i, j) recursively, where D(s, t, 0, 0) = 0 and Identifying approximately duplicate database records that refer to the same entity is essential for information integration. The authors compare and describe methods for combining and learning similarity measures for name matching.

View PDFchevron_right
Record matching: Past, present and future
M. Cochinwala, V. Verykios

2001

Data quality often manifests itself as inconsistencies between systems or inconsistencies with reality. The latter can be explained as a mismatch between real world objects and the way they are represented in databases. The former, as it is exemplified by data heterogeneity -semantic and structural -replicated data, and data integrity problems, is the main cause of duplicate records. Often the same real world entity is represented by two or more records. Many times, duplicate records do not share a common key and/or they contain errors that makes matching them a difficult problem. This specific problem is the subject of this expository study. This study relies on a thorough analysis of the literature, on an intimate knowledge of an application and on many years of working on data quality with sensitive telecommunication systems. This paper also presents a taxonomy for record matching algorithms and proposed solutions. This paper is limited to record matching and addresses the general problem of data quality only in passing. This should be and has been the topic of many other survey papers.

View PDFchevron_right
On the accuracy and completeness of the record matching process
mahmoud elfeky

2000

Record matching or linking is one of the phases of the data quality improvement process, in which, records from different sources, are cleansed and integrated in a centralized data store to be used for various purposes. Both, earlier and recent studies in data quality and record linkage focus on various statistical models, which make strong assumptions on the probabilities of attribute errors. In this study, we evaluate different models for record linkage, which are built based on data only. We use a program that generates data with known error distributions and we train classification models, which we use to estimate the accuracy and the completeness of the record linking process. The results indicate that the automated learning techniques are adequate for this process and that both their accuracy and their completeness are comparable to the accuracy and the completeness of other, mostly manual, processes.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved