Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Figures
Introduction
Adaptive Sorted Neighborhood Method
Experimental Results Evaluation Metrics
Datasets -
Restaurant Dataset
Real Customer Dataset
Conclusion
References

Near Duplicate Detection In Relational Database

Profile image of Bhagyashri KelkarBhagyashri Kelkar

2013

Abstract

Near Duplicate detection is an important precess for many database based applications. Accurately identifying duplicate entities between multiple data sources is a big challenge to organizations and researchers. To detect the approximately duplicate records that refer to the same real-world entity is important to make the database more concrete and achieve higher data quality. In this process, ideally each record must be compared with every other record in dataset for finding duplicates. It is possible to reduce search space for record comparisons by using mutual exclusion property of tuples. In this research paper we analyze two types of blocking algorithms, namely, the adaptive sorted neighborhood method (ASNM), and iterative blocking and their combination with Jaro Winkler distance for string matching. Experimental evaluation on real dataset shows that, adaptive sorted neighborhood method along with Jaro Winkler distance algorithm outperforms in terms of precision and recall and requires very less number of comparisons than iterative blocking method. The experiments also highlight that, strings matching threshold gives optimal results if value is in range of 85% to 90%.

Key takeaways
sparkles

AI

  1. Adaptive Sorted Neighborhood Method with Jaro-Winkler outperforms iterative blocking in precision and recall.
  2. Optimal similarity threshold for restaurant dataset is 85%, while 90% generally yields best results across datasets.
  3. Blocking reduces search space, enhancing efficiency from O(n^2) to more manageable comparisons.
  4. Effective attribute weight allocation significantly influences duplicate detection outcomes.
  5. Nearly 1% to 5% duplicate records are common in integrated databases, impacting data quality.
Figures (9)
Figure 3. RECORD COMPARISONS REQUIRED FOR ITERATIVE BLOCKING(UB) AND ADAPTIVE SORTED NEIGHBORHOOD(ASNM) IN COMBINATION WITH JARO-WINKLER(JW) AND SOUNDEX(SND) SIMILARITY
Figure 3. RECORD COMPARISONS REQUIRED FOR ITERATIVE BLOCKING(UB) AND ADAPTIVE SORTED NEIGHBORHOOD(ASNM) IN COMBINATION WITH JARO-WINKLER(JW) AND SOUNDEX(SND) SIMILARITY
TABLE I. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD WITH JARO- WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS
TABLE I. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD WITH JARO- WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS
Implementation of adaptive sorted neighborhood with Jaro- Winkler string similarity:
Implementation of adaptive sorted neighborhood with Jaro- Winkler string similarity:
Figure 2. COMPARATIVE CHART OF ITERATIVE BLOCKING(IB) AND ADAPTIVE SORTED NEIGHBORHOOD(ASNM) IN COMBINATION WITH JARO-WINKLER(JW) AND SOUNDEX(SND) SIMILARITY
Figure 2. COMPARATIVE CHART OF ITERATIVE BLOCKING(IB) AND ADAPTIVE SORTED NEIGHBORHOOD(ASNM) IN COMBINATION WITH JARO-WINKLER(JW) AND SOUNDEX(SND) SIMILARITY
TABLE V. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD USING JARO- WINKLER SIMILARITY AND UNEQUAL FIELD WEIGHTS Dbgen Dataset:
TABLE V. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD USING JARO- WINKLER SIMILARITY AND UNEQUAL FIELD WEIGHTS Dbgen Dataset:
TABLE III-RESULTS OF ITERATIVE BLOCKING USING JARO-WINKLER SIMILARITY AND UNEQUAL FIELD WEIGHTS Comparison of table II with table III indicate that, assigning unequal weights for forming similarity index results in improved precision, recall & F-measure and increase in number of record comparisons.
TABLE III-RESULTS OF ITERATIVE BLOCKING USING JARO-WINKLER SIMILARITY AND UNEQUAL FIELD WEIGHTS Comparison of table II with table III indicate that, assigning unequal weights for forming similarity index results in improved precision, recall & F-measure and increase in number of record comparisons.
TABLE IV. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD WITH JARO- WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS If highest weight-age is given to name field, following results are obtained.
TABLE IV. RESULTS OF ADAPTIVE SORTED NEIGHBORHOOD METHOD WITH JARO- WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS If highest weight-age is given to name field, following results are obtained.
TABLE II: RESULTS OF ITERATIVE BLOCKING WITH JARO-WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS
TABLE II: RESULTS OF ITERATIVE BLOCKING WITH JARO-WINKLER SIMILARITY AND EQUAL FIELD WEIGHTS

Related papers

A STUDY AND SURVEY ON VARIOUS PROGRESSIVE DUPLICATE DETECTION MECHANISMS
eSAT Journals

One of the serious problems faced in several applications with personal details management, customer affiliation management, data mining, etc is duplicate detection. This survey deals with the various duplicate record detection techniques in both small and large datasets. To detect the duplicity with less time of execution and also without disturbing the dataset quality, methods like Progressive Blocking and Progressive Neighborhood are used. Progressive sorted neighborhood method also called as PSNM is used in this model for finding or detecting the duplicate in a parallel approach. Progressive Blocking algorithm works on large datasets where finding duplication requires immense time. These algorithms are used to enhance duplicate detection system. The efficiency can be doubled over the conventional duplicate detection method using this algorithm. Several different methods of data analysis are studied here with various approaches for duplicate detection.

View PDFchevron_right
Duplicates Detection Within Incomplete Data Sets Using Blocking and Dynamic Sorting Key Methods
Awsan Thabet

International Journal of Advanced Computer Science and Applications, 2018

In database records duplicate detection, blocking method is commonly used to reduce the number of comparisons between the candidate record pairs. The main procedure in this method requires selecting attributes that will be used as sorting keys. Selection accuracy is essential in clustering candidates records that are likely matched in the same block. Nevertheless, the presence of missing values affects the creation of sorting keys and this is particularly undesirable if it involves the attributes that are used as the sorting keys. This is because, consequently, records that are supposed to be included in the duplicate detection procedure will be excluded from being examined. Thus, in this paper, we propose a method that can deal with the impact of missing values by using a dynamic sorting key. Dynamic sorting is an extension of blocking method that essentially works on two functions namely uniqueness calculation function (UF) (to choose unique attributes) and completeness function (CF) (to search for missing values). We experimented a particular blocking method called as sorted neighborhood with a dynamic sorting key on a restaurant data set (that consists of duplicate records) obtained from earlier research in order to evaluate the method's accuracy and speed. Hypothetical missing values were applied to testing data set used in the experiment, where we compare the results of duplicate detection with (and without) dynamic sorting key. The result shows that, even though missing values are present, there is a promising improvement in the partitioning of duplicate records in the same block.

View PDFchevron_right
Fast Semantic Duplicate Detection Techniques in Databases
IBRAHIM MOUKOUOP NGUENA

Journal of Software Engineering and Applications

Semantic duplicates in databases represent today an important data quality challenge which leads to bad decisions. In large databases, we sometimes find ourselves with tens of thousands of duplicates, which necessitates an automatic deduplication. For this, it is necessary to detect duplicates, with a fairly reliable method to find as many duplicates as possible and powerful enough to run in a reasonable time. This paper proposes and compares on real data effective duplicates detection methods for automatic deduplication of files based on names, working with French texts or English texts, and the names of people or places, in Africa or in the West. After conducting a more complete classification of semantic duplicates than the usual classifications, we introduce several methods for detecting duplicates whose average complexity observed is less than O(2n). Through a simple model, we highlight a global efficacy rate, combining precision and recall. We propose a new metric distance between records, as well as rules for automatic duplicate detection. Analyses made on a database containing real data for an administration in Central Africa, and on a known standard database containing names of restaurants in the USA, have shown better results than those of known methods, with a lesser complexity.

View PDFchevron_right
Deduplication Methods Using Levenshtein Distance Algorithm
Eugene Valeriano

Deleted Journal, 2024

The study aimed to propose methods to improve the data integrity of the Relational databases such as MS SQL, MySQL and PostgreSQL via record duplication detection. The FODORS and ZAGAT Restaurant database benchmark datasets have been utilized to facilitate the processes involved in preparing and delivering high-quality data. Furthermore, the Levenshtein distance algorithm was used to propose three (3) methods namely: default, eliminating equal string, and knowledge-based libraries to cut duplicate records in the database. In the 70% selected threshold, the average detected duplicate records of 88 out of 112 records in the restaurant dataset. Finally, to efficiently detect duplicate records in the database, depend on the data being analyzed and threshold selected.

View PDFchevron_right
Detection of near duplicates in tables based on the Locality-Sensitive Hashing Method and the Nearest Neighbor Method
Alexander Kuchansky, Лариса Чала

A hybrid method for the detection of near duplicates in tables is proposed. This method allows the identification of similarities between text and numeric data of tables separately, and then it generalized the results obtained. For the text data, sequences of words are formed in the canonized form, from which, based on the method of locality-sensitive hashing, the bit sequences are constructed. A similarity between data in this case is determined by the Hamming distance at the assigned threshold value. The identification of similarities between numeric data in tables is implemented based on the method of the nearest neighbours with assigned metric distances. The method makes it possible to identify near duplicates, present in data in the input table, relative to a set of tables, which are selected from the scientific publications and dissertations and theses papers. It should be noted that the method is designed for finding near duplicates in tables that contain only text and numeric data. In the case of availability in the content of examined tables of pictures and formulas, these objects are examined separately by using specific methods. The method proposed might be implemented in the systems that are intended for running intelligent analysis of information represented by text and tables to identify similarities and detect near-duplicates, in particular, antiplagiarism-systems.

View PDFchevron_right
Performance Analysis of Duplicate Record Detection Techniques
Syed Hasnain Adil

Engineering, Technology & Applied Science Research, 2019

In this paper, a comprehensive performance analysis of duplicate data detection techniques for relational databases has been performed. The research focuses on traditional SQL based and modern bloom filter techniques to find and eliminate records which already exist in the database while performing bulk insertion operation (i.e. bulk insertion involved in the loading phase of the Extract, Transform, and Load (ETL) process and data synchronization in multisite database synchronization). The comprehensive performance analysis was performed on several data sizes using SQL, bloom filter, and parallel bloom filter. The results show that the parallel bloom filter is highly suitable for duplicate detection in the database.

View PDFchevron_right
Improving database quality through eliminating duplicate records
Mingzhen Wei

Data Science Journal, 2006

Redundant or duplicate data are the most troublesome problem in database management and applications. Approximate field matching is the key solution to resolve the problem by identifying semantically equivalent string values in syntactically different representations. This paper considers token-based solutions and proposes a general field matching framework to generalize the field matching problem in different domains. By introducing a concept of String Matching Points (SMP) in string comparison, string matching accuracy and efficiency are improved, compared with other commonly-applied field matching algorithms. The paper discusses the development of field matching algorithms from the developed general framework. The framework and corresponding algorithm are tested on a public data set of the NASA publication abstract database. The approach can be applied to address the similar problems in other databases.

View PDFchevron_right
Effective and efficient similarity search in databases
Vardas Pavarde
View PDFchevron_right
Validation of Deduplication in Data using Similarity Measure
varsha wandhekar

International Journal of Computer Applications, 2015

Deduplication is the process of determining all categories of information within a data set that signify the same real life / world entity. The data gathered from various resources may have data high quality issues in it. The concept to identify duplicates by using windowing and blocking strategy. The objective is to achieve better precision, good efficiency and also to reduce the false positive rate all are in accordance with the estimated similarities of records. Various Similarity metrics are commonly used to recognize the similar field entries. So the main focus of this paper is to applying appropriate similarity measure on appropriate data to properly identifying the duplicates.

View PDFchevron_right
Effective Progressive Algorithm for Duplicate Detection on Large Dataset
International Journal of Scientific Research in Science and Technology IJSRST

Effective progressive algorithm for duplicate detection on large dataset is the process of detecting unwanted files in the document in very short time.It does not affect the file without any quality change and also it used for cleaning process. The main advantage is very efficient and very speed.It has unlimited large data sets. The system alerts the user about potential duplicates when the user tries to create new records or update existing records. To maintain data quality, you can schedule a duplicate detection job to check for duplicates for all records that match a certain criteria. You can clean the data by deleting, deactivating, or merging the duplicates reported by a duplicate detection. We propose two novel, progressive duplicate detection algorithms namely progressive sorted neighborhood method (PSNM), which performs best on small and almost clean datasets, and progressive blocking (PB), which performs best on large and very dirty datasets. Both enhance the efficiency of duplicate detection even on very large datasets. which expose different strengths and outperform current approaches. We exhaustively evaluate on several real-world datasets testing our own and previous algorithms.

View PDFchevron_right

References (8)

  1. Xiaochun Yang Bin Wang Guoren Wang Ge Yu Key "RSEARCH: Enhancing Keyword Search in Relational Databases Using Nearly Duplicate Records" ( 2010 IEEE. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering )
  2. Ahmed K. Elmagarmid, Senior Member, IEEE, Panagiotis G. Ipeirotis, Member, IEEE Computer Society, and Vassilios S. Verykios, Member, IEEE Computer Society Ahmed K. Elmagarmid, Panagiotis G. Ipeirotis, and Vassilios S.Verykios. "Duplicate Record Detection: A Survey"( IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 19, NO. 1, JANUARY 2007 )
  3. Su Yan, Dongwon Lee, Min-Yen Kan, and Lee C. Giles. "Adaptive sorted neighborhood methods for efficient record linkage." In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, 2007.
  4. Steven Euijong Whang, David Menestrina, Georgia Koutrika, Martin Theobald, and Hector Garcia-Molina. "Entity resolution with iterative blocking". In Proceedings of the ACM International Conference on Management of Data (SIGMOD), 2009.
  5. A. E. Monge and C. Elkan, "An efficient domain- independent algorithm for detecting approximately S.E. Whang, D. Menestrina, G. Koutrika, M. Theobald, H. Garcia-Molina, Entity resolution with iterative blocking, in: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data (SIGMOD '09), 2009, pp. 219-232
  6. Bilenko, R. Mooney, W. Cohen, P. Ravikumar, and S. Fienberg. "Adaptive name matching in information integration" IEEE Intelligent Systems, 18(5):16-23, Sep/Oct 2003. AUTHORS
  7. Bhagyashri A. Kelkar received the B.E. in Computer Sci. & Engg. from Walchand College of Engg., Sangli in 1995. She is currently pursuing M.E. degree in Comp. Science & Engg. Prof. K. B. Manwade received M.Tech. degree in computer science from Shivaji University, Kolhapur. He is working as an Head of Computer sceience Department in Ashokrao Mane Group of Institutes, Wathar.
  8. Prof. G. A. Patil received ME degree in computer sci. & engg. From Walchand College Sangli. He is working as head of department in D. Y. Patil College of engg. & tech., Kolhapur .

Related papers

Novel and Efficient approach for Duplicate Record Detection
IJAERS Journal

—Similarity check of real world entities is a necessary factor in these days which is named as Data Replica Detection. Time is an critical factor today in tracking Data Replica Detection for large data sets, without having impact over quality of Dataset. In this system primarily introduce two Data Replica Detection algorithms, where in these contribute enhanced procedural standards in finding Data Replication at limited execution periods. This system contribute better improvised state of time than conventional techniques. We propose two Data duplicate record detection algorithms namely progressive sorted neighbourhood method (PSNM), which performs best on small and almost clean datasets, progressive blocking (PB), and parallel sorted neighbourhood method which performs best on large and very grimy datasets. Both enhance the efficiency of duplicate detection even on very large datasets.

View PDFchevron_right
An Efficient Domain-Independent Algorithm for Detecting Approximately Duplicate Database Records
Alvaro Monge

1997

Detecting database records that are approximate duplicates, but not exact duplicates, is an important task. Databases may contain duplicate records concerning the same realworld entity because of data entry errors, because of unstandardized abbreviations, or because of di erences in the detailed schemas of records from multiple databases, among other reasons. In this paper, we present an e cient algorithm for recognizing clusters of approximately duplicate records. Three key ideas distinguish the algorithm presented. First, a version of the Smith-Waterman algorithm for computing minimum edit-distance is used as a domainindependent method to recognize pairs of approximately duplicate records. Second, the union/ nd algorithm is used to keep track of clusters of duplicate records incrementally, as pairwise duplicate relationships are discovered. Third, the algorithm uses a priority queue of cluster subsets to respond adaptively to the size and homogeneity of the clusters discovered as the database is scanned. This typically reduces by over 75% the number of times that the expensive pairwise record matching (Smith-Waterman or other) is applied, without impairing accuracy. Comprehensive experiments on synthetic databases and on a real database of bibliographic records con rm the e ectiveness of the new algorithm.

View PDFchevron_right
Database Record Duplicate Detection System using Simil Algorithm
Jumoke Soyemi

International Journal on Computer Science and Engineering

As more data is populated into the database table, there is the tendency for the table to store duplicate or redundant record which results in the consumption of data spaces in the database and also in the storage device where the database resides. Despite the ever-increasing memory capacities of devices, significant benefits can still be realized by reducing the bytes size needed to represent an object when it is stored or retrieved from the database. This is quite beneficial to mobile devices with limited storage, reference data, e-mail, where sequences of large bytes are repeated and data transmitted over lowbandwidth or congested links. Reducing bytes equates to eliminating unneeded data, and there are numerous techniques for reducing redundancy when objects are stored or sent. This study implemented a database record duplicate detection system using simil algorithm as the reduction technique to achieve efficiency in detecting and reducing the presence of duplicate records in a database and hence provides an automated means of executing database record optimization. Keywords-database record; simil algorithm; prediction technique; duplicate detection system I. INTRODUCTION The need to detect and remove duplicate records that have to do with the same entity within a dataset is a task that is crucial. Linking data to detect duplicates is good in improving the quality and integrity of data which allow re-uses of existing data sources for future research work [1]. Duplicate detection is different from other Information Retrieval (IR) considering how it defines the similarity check between two or more documents. In many IR document, similarity refers to semantic relevance, which could be syntactically very different but still relevant. In contrast, the appearance of similarity in duplicate detection in early database research is quite conventional, and what it does it to discover syntactically almost-identical documents [2][3][4]. For other tasks that need to detect documents with intermediate level of similarity, there has not been much research done. The major problem that does arise is that, as more data is populated into a database table, there is the tendency for the table to store duplicate or redundant record which results in the consumption of data spaces in the database and also in the storage device the database resides. In reality, entities in the database have two and more representations. This is because records rarely share a common key and they exhibit errors thus making duplicate matching a task that is difficult [5][6]. Errors are introduced as the result of transcription errors, incomplete information, lack of standard formats, or any combination of these factors. In this paper, duplicate record detection system is proposed. The similarity metrics that are commonly used to detect similar field entries are covered with some algorithm used for duplicate detection to find approximately duplicates records in a database. This study is designed based on Simil algorithm to identify the presence of duplicate records in a MySQL database local server installed on a computer system. It is meant to serve as a means of reducing the quantity of disk space that MySQL database server is hosting on a system and provide an automated means of executing database record optimization. The Database record duplicate detection system is designed using Java Object Oriented Programming Language and MySQL ODBC J-Connector Library for connecting to the database server of MySQL. The rest of the study is divided into sections, with section 1 introducing the study, section 2 reviews the literature. Section 3 discusses the system design and its implementation and section 4 concludes the study. II. REVIEW OF LITERATURE The problem of finding duplicated documents is a research interest in database and web-search communities for quite some years now. The key applications developed related to this area include plagiarism detection in web publishing and redundancy detection in large datasets. The popular duplicate detection techniques are grouped into Fingerprint-based and Full text-based techniques [7].

View PDFchevron_right
Duplicate Record Detection: A Survey
first name lastname

IEEE Transactions on Knowledge and Data Engineering, 2007

Often, in the real world, entities have two or more representations in databases. Duplicate records do not share a common key and/or they contain errors that make duplicate matching a difficult task. Errors are introduced as the result of transcription errors, incomplete information, lack of standard formats, or any combination of these factors. In this paper, we present a thorough analysis of the literature on duplicate record detection. We cover similarity metrics that are commonly used to detect similar field entries, and we present an extensive set of duplicate detection algorithms that can detect approximately duplicate records in a database. We also cover multiple techniques for improving the efficiency and scalability of approximate duplicate detection algorithms. We conclude with coverage of existing tools and with a brief discussion of the big open problems in the area.

View PDFchevron_right
A Survey: Detection of Duplicate Record
Anurag Jain

2012

The problem of identifying approximately duplicate record in database is an essential step for data cleaning & data integration process. A dynamic web page is displayed to show the results as well as other relevant advertisements that seem relevant to the query. The real world entities have two or more representation in databases. When dealing with large amount of data it is important that there be a well defined and tested mechanism to filter out duplicate result. This keeps the result relevant to the queries. Duplicate record exists in the query result of many web databases especially when the duplicates are defined based on only some of the fields in a record. Using exact matching technique Records that are exactly same can be detected. The system that helps user to integrate and compares the query results returned from multiple web databases matches the different sources records that referred to the same real world entity. In this paper, we analyze the literature on duplicate re...

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved