Academia.eduAcademia.edu

Outline

Title
Abstract
Goals and Objectives
Introduction and Motivation
Intended Audience and Material
References
All Topics
Computer Science
Database Systems

Entity Resolution: Past, Present and Yet-to-Come

Profile image of Ekaterini IoannouEkaterini Ioannou

2020

https://doi.org/10.5441/002/EDBT.2020.85
visibility

…

description

4 pages

link

1 file

Abstract

Entity Resolution (ER) lies at the core of data integration, with a bulk of research focusing on its effectiveness and its time efficiency. Most past relevant works were crafted for addressing Veracity over structured (relational) data. They typically rely on schema, expert and external knowledge to maximize accuracy. Part of these methods have been recently extended to process large volumes of data through massive parallelization techniques, such as the MapReduce paradigm. With the present advent of Big Web Data, the scope moved towards Variety, aiming to handle semi-structured data collections, with noisy and highly heterogeneous information. Relevant works adopt a novel, loosely schema-aware functionality that emphasizes scalability and robustness to noise. Another line of present research focuses on Velocity, i.e., processing data collections of a continuously increasing volume. In this tutorial, we present the ER generations by discussing past, present, and yet-to-come mechanis...

Related papers

Entity resolution for uncertain data
Hamideh Afsarmanesh

2012

Entity resolution (ER), also known as duplicate detection or record matching, is the problem of identifying the tuples that represent the same real world entity. In this paper, we address the problem of ER for uncertain data, which we call ERUD. We propose two different approaches for the ERUD problem based on two classes of similarity functions, i.e. context-free and context-sensitive. We propose a PTIME algorithm for context-free similarity functions, and a Monte Carlo algorithm for context-sensitive similarity functions. Existing context-sensitive similarity functions need at least one pass over the database to compute some statistical features of data, which makes it very inefficient for our Monte Carlo algorithm. Thus, we propose a novel context-sensitive similarity function that makes our Monte Carlo algorithm more efficient. To further improve the efficiency of our proposed Monte Carlo algorithm, we propose a parallel version of it using the MapReduce framework. We validated our algorithms through experiments over both synthetic and real datasets. Our performance evaluation shows the effectiveness of our algorithms in terms of success rate and response time.

View PDFchevron_right
Learning-based entity resolution with MapReduce
Lars Kolb, Erhard Rahm

Proceedings of the third international workshop on Cloud data management - CloudDB '11, 2011

Entity resolution is a crucial step for data quality and data integration. Learning-based approaches show high effectiveness at the expense of poor efficiency. To reduce the typically high execution times, we investigate how learningbased entity resolution can be realized in a cloud infrastructure using MapReduce. We propose and evaluate two efficient MapReduce-based strategies for pair-wise similarity computation and classifier application on the Cartesian product of two input sources. Our evaluation is based on real-world datasets and shows the high efficiency and effectiveness of the proposed approaches.

View PDFchevron_right
JedAI: The Force Behind Entity Resolution
George Giannakopoulos

Lecture Notes in Computer Science, 2017

We present JedAI, a toolkit for Entity Resolution that can be used in three different ways: as an open-source Java library that implements numerous state-of-the-art, domain-independent methods, as a workbench that facilitates the evaluation of their relative performance and as a desktop application that offers out-of-the-box ER solutions. JedAI bridges the gap between the database and the Semantic Web communities, offering solutions that are applicable to both relational and RDF data. It also conveys a modular architecture that facilitates its extension with more methods and with more comprehensive workflows.

View PDFchevron_right
Simplifying Entity Resolution on Web Data with Schema-Agnostic, Non-Iterative Matching
Vassilis Christophides

2018 IEEE 34th International Conference on Data Engineering (ICDE), 2018

Entity Resolution (ER) aims to identify different descriptions in various Knowledge Bases (KBs) that refer to the same entity. ER is challenged by the Variety, Volume and Veracity of descriptions published in the Web of Data. To address them, we propose the MinoanER framework that fulfills full automation and support of highly heterogeneous entities. MinoanER leverages a token-based similarity of entities to define a new metric that derives the similarity of neighboring entities from the most important relations, indicated only by statistics. For high efficiency, similarities are computed from a set of schema-agnostic blocks and processed in a non-iterative way that involves four threshold-free heuristics. We demonstrate that the effectiveness of MinoanER is comparable to existing ER tools over real KBs exhibiting low heterogeneity in terms of entity types and content. Yet, MinoanER outperforms state-of-the-art ER tools when matching highly heterogeneous KBs.

View PDFchevron_right
Efficient entity resolution for large heterogeneous information spaces
George Papadakis

Proceedings of the fourth ACM international conference on Web search and data mining - WSDM '11, 2011

We have recently witnessed an enormous growth in the volume of structured and semi-structured data sets available on the Web. An important prerequisite for using and combining such data sets is the detection and merge of information that describes the same real-world entities, a task known as Entity Resolution. To make this quadratic task efficient, blocking techniques are typically employed. However, the high dynamics, loose schema binding, and heterogeneity of (semi-)structured data, impose new challenges to entity resolution. Existing blocking approaches become inapplicable because they rely on the homogeneity of the considered data and a-priory known schemata. In this paper, we introduce a novel approach for entity resolution, scaling it up for large, noisy, and heterogeneous information spaces. It combines an attribute-agnostic mechanism for building blocks with intelligent block processing techniques that boost blocks with high expected utility, propagate knowledge about identified matches, and preempt the resolution process when it gets too expensive. Our extensive evaluation on real-world, large, heterogeneous data sets verifies that the suggested approach is both effective and efficient.

View PDFchevron_right
D-Swoosh: A Family of Algorithms for Generic, Distributed Entity Resolution
Tait Larson

27th International Conference on Distributed Computing Systems (ICDCS '07), 2007

Entity Resolution (ER) matches and merges records that refer to the same real-world entities, and is typically a compute-intensive process due to complex matching functions and high data volumes. We present a family of algorithms, D-Swoosh, for distributing the ER workload across multiple processors. The algorithms use generic match and merge functions, and ensure that new merged records are distributed to processors that may have matching records. We perform a detailed performance evaluation on a testbed of 15 processors, for cases where application knowledge can eliminate some comparisons and where all records must be matched. Our experiments use actual comparison shopping data provided by Yahoo!.

View PDFchevron_right
MinoanER: Schema-Agnostic, Non-Iterative, Massively Parallel Resolution of Web Entities
Vassilis Christophides

2019

Entity Resolution (ER) aims to identify different descriptions in various Knowledge Bases (KBs) that refer to the same entity. ER is challenged by the Variety, Volume and Veracity of entity descriptions published in the Web of Data. To address them, we propose the MinoanER framework that simultaneously fulfills full automation, support of highly heterogeneous entities, and massive parallelization of the ER process. MinoanER leverages a token-based similarity of entities to define a new metric that derives the similarity of neighboring entities from the most important relations, as they are indicated only by statistics. A composite blocking method is employed to capture different sources of matching evidence from the content, neighbors, or names of entities. The search space of candidate pairs for comparison is compactly abstracted by a novel disjunctive blocking graph and processed by a non-iterative, massively parallel matching algorithm that consists of four generic, schema-agnost...

View PDFchevron_right
Collective entity resolution in relational data
Lise Getoor

2007

Abstract Many databases contain uncertain and imprecise references to real-world entities. The absence of identifiers for the underlying entities often results in a database which contains multiple references to the same entity. This can lead not only to data redundancy, but also inaccuracies in query processing and knowledge extraction. These problems can be alleviated through the use of entity resolution. Entity resolution involves discovering the underlying entities and mapping each database reference to these entities.

View PDFchevron_right
Online collective entity resolution
Lise Getoor

2007

Abstract Entity resolution is a critical component of data integration where the goal is to reconcile database references corresponding to the same real-world entities. Given the abundance of publicly available databases that have unresolved entities, we motivate the problem of quick and accurate resolution for answering queries over such 'unclean'databases.

View PDFchevron_right
Query-time entity resolution
Indrajit Bhattacharya

Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '06, 2006

Entity resolution is the problem of reconciling database references corresponding to the same real-world entities. Given the abundance of publicly available databases that have unresolved entities, we motivate the problem of query-time entity resolution: quick and accurate resolution for answering queries over such 'unclean' databases at query-time. Since collective entity resolution approaches-where related references are resolved jointly-have been shown to be more accurate than independent attribute-based resolution for off-line entity resolution, we focus on developing new algorithms for collective resolution for answering entity resolution queries at query-time. For this purpose, we first formally show that, for collective resolution, precision and recall for individual entities follow a geometric progression as neighbors at increasing distances are considered. Unfolding this progression leads naturally to a two stage 'expand and resolve' query processing strategy. In this strategy, we first extract the related records for a query using two novel expansion operators, and then resolve the extracted records collectively. We then show how the same strategy can be adapted for query-time entity resolution by identifying and resolving only those database references that are the most helpful for processing the query. We validate our approach on two large real-world publication databases where we show the usefulness of collective resolution and at the same time demonstrate the need for adaptive strategies for query processing. We then show how the same queries can be answered in real-time using our adaptive approach while preserving the gains of collective resolution. In addition to experiments on real datasets, we use synthetically generated data to empirically demonstrate the validity of the performance trends predicted by our analysis of collective entity resolution over a wide range of structural characteristics in the data.

View PDFchevron_right

References (78)

  1. Parag Agrawal, Omar Benjelloun, Anish Das Sarma, Chris Hayworth, Shubha U. Nabar, Tomoe Sugihara, and Jennifer Widom. 2006. Trio: A System for Data, Uncertainty, and Lineage. In VLDB. 1151-1154.
  2. Hotham Altwaijry, Dmitri Kalashnikov, and Sharad Mehrotra. 2013. Query- Driven Approach to Entity Resolution. PVLDB 6, 14 (2013), 1846-1857.
  3. Hotham Altwaijry, Dmitri Kalashnikov, and Sharad Mehrotra. 2017. QDA: A Query-Driven Approach to Entity Resolution. TKDE (2017).
  4. Hotham Altwaijry, Sharad Mehrotra, and Dmitri V. Kalashnikov. 2015. QuERy: A Framework for Integrating Entity Resolution with Query Processing. PVLDB 9, 3 (2015), 120-131.
  5. Omar Benjelloun, Hector Garcia-Molina, David Menestrina, Qi Su, Steven Eui- jong Whang, and Jennifer Widom. 2009. Swoosh: a generic approach to entity resolution. VLDB J. 18, 1 (2009), 255-276.
  6. Philip A. Bernstein, Jayant Madhavan, and Erhard Rahm. 2011. Generic Schema Matching, Ten Years Later. PVLDB 4, 11 (2011), 695-701.
  7. Indrajit Bhattacharya and Lise Getoor. 2007. Collective entity resolution in relational data. TKDD 1, 1 (2007), 5.
  8. Guilherme Dal Bianco, Marcos André Gonçalves, and Denio Duarte. 2018. BLOSS: Effective meta-blocking with almost no effort. Inf. Syst. 75 (2018), 75-89.
  9. Christoph Böhm, Gerard de Melo, Felix Naumann, and Gerhard Weikum. 2012. LINDA: distributed web-of-data-scale entity matching. In CIKM. 2104-2108.
  10. Chengliang Chai, Guoliang Li, Jian Li, Dong Deng, and Jianhua Feng. 2018. A partial-order-based framework for cost-effective crowdsourced entity resolu- tion. VLDB J. 27, 6 (2018), 745-770.
  11. Peter Christen. 2012. Data Matching. Springer.
  12. Vassilis Christophides, Vasilis Efthymiou, and Kostas Stefanidis. 2015. Entity Resolution in the Web of Data. Morgan & Claypool Publishers.
  13. Sanjib Das, Paul Suganthan G. C., AnHai Doan, Jeffrey F. Naughton, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, Vijay Raghavendra, and Youngchoon Park. 2017. Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services. In SIGMOD. 1431-1446.
  14. Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data pro- cessing on large clusters. Commun. ACM 51, 1 (2008), 107-113.
  15. Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. 2013. Large-scale linked data integration using probabilistic reasoning and crowdsourcing. VLDB J. 22, 5 (2013), 665-687.
  16. Xin Dong, Alon Y. Halevy, and Jayant Madhavan. 2005. Reference Reconcilia- tion in Complex Information Spaces. In SIGMOD. 85-96.
  17. Xin Luna Dong and Divesh Srivastava. 2013. Big Data Integration. PVLDB 6, 11 (2013), 1188-1189.
  18. Xin Luna Dong and Divesh Srivastava. 2015. Big Data Integration. Morgan & Claypool Publishers.
  19. Songyun Duan, Achille Fokoue, Oktie Hassanzadeh, Anastasios Kementsiet- sidis, Kavitha Srinivas, and Michael J. Ward. 2012. Instance-Based Matching of Large Ontologies Using Locality-Sensitive Hashing. In ISWC. 49-64.
  20. Muhammad Ebraheem, Saravanan Thirumuruganathan, Shafiq R. Joty, Mourad Ouzzani, and Nan Tang. 2018. Distributed Representations of Tuples for Entity Resolution. PVLDB 11, 11 (2018), 1454-1467.
  21. Vasilis Efthymiou, George Papadakis, George Papastefanatos, Kostas Stefani- dis, and Themis Palpanas. 2017. Parallel meta-blocking for scaling entity resolution over big heterogeneous data. Inf. Syst. (2017).
  22. Vasilis Efthymiou, George Papadakis, Kostas Stefanidis, and Vassilis Christophides. 2019. MinoanER: Schema-Agnostic, Non-Iterative, Massively Parallel Resolution of Web Entities. In EDBT. 373-384.
  23. Ahmed K. Elmagarmid, Panagiotis G. Ipeirotis, and Vassilios S. Verykios. 2007. Duplicate Record Detection: A Survey. TKDE 19, 1 (2007), 1-16.
  24. Donatella Firmani, Sainyam Galhotra, Barna Saha, and Divesh Srivastava. 2018. Robust Entity Resolution Using a CrowdOracle. IEEE Data Eng. Bull. 41, 2 (2018), 91-103.
  25. Donatella Firmani, Barna Saha, and Divesh Srivastava. 2016. Online Entity Resolution Using an Oracle. PVLDB 9, 5 (2016), 384-395.
  26. Jeffrey Fisher, Peter Christen, Qing Wang, and Erhard Rahm. 2015. A Clustering-Based Framework to Control Block Sizes for Entity Resolution. In KDD. 279-288.
  27. Avigdor Gal. 2014. Tutorial: Uncertain Entity Resolution. PVLDB 7, 13 (2014), 1711-1712.
  28. Sainyam Galhotra, Donatella Firmani, Barna Saha, and Divesh Srivastava. 2018. Robust Entity Resolution using Random Graphs. In SIGMOD. 3-18.
  29. Lise Getoor and Ashwin Machanavajjhala. 2012. Entity Resolution: Theory, Practice & Open Challenges. PVLDB 5, 12 (2012), 2018-2019.
  30. Chaitanya Gokhale, Sanjib Das, AnHai Doan, Jeffrey F. Naughton, Narasimhan Rampalli, Jude W. Shavlik, and Xiaojin Zhu. 2014. Corleone: hands-off crowd- sourcing for entity matching. In SIGMOD. 601-612.
  31. Behzad Golshan, Alon Halevy, George Mihaila, and Wang-Chiew Tan. 2017. Data Integration: After the Teenage Years. In PODS. 101-106.
  32. Yash Govind, Erik Paulson, Palaniappan Nagarajan, Paul Suganthan G. C., AnHai Doan, Youngchoon Park, Glenn Fung, Devin Conathan, Marshall Carter, and Mingju Sun. 2018. CloudMatcher: A Hands-Off Cloud/Crowd Service for Entity Matching. PVLDB 11, 12 (2018), 2042-2045.
  33. Anja Gruenheid, Xin Luna Dong, and Divesh Srivastava. 2014. Incremental Record Linkage. PVLDB 7, 9 (2014), 697-708.
  34. Oktie Hassanzadeh, Fei Chiang, Renée J. Miller, and Hyun Chul Lee. 2009. Framework for Evaluating Clustering Algorithms in Duplicate Detection. PVLDB 2, 1 (2009), 1282-1293.
  35. Ekaterini Ioannou and Minos Garofalakis. 2015. Query Analytics over Proba- bilistic Databases with Unmerged Duplicates. TKDE 27, 8 (2015), 2245-2260.
  36. Ekaterini Ioannou, Wolfgang Nejdl, Claudia Niederée, and Yannis Velegrakis. 2010. On-the-Fly Entity-Aware Query Processing in the Presence of Linkage. PVLDB 3, 1 (2010), 429-438.
  37. Mayank Kejriwal and Daniel P. Miranker. 2013. An Unsupervised Algorithm for Learning Blocking Schemes. In ICDM. 340-349.
  38. Lars Kolb, Andreas Thor, and Erhard Rahm. 2012. Dedoop: Efficient Dedupli- cation with Hadoop. PVLDB 5, 12 (2012), 1878-1881.
  39. Lars Kolb, Andreas Thor, and Erhard Rahm. 2012. Load Balancing for MapReduce-based Entity Resolution. In ICDE. 618-629.
  40. Pradap Konda, Sanjib Das, Paul Suganthan G. C., AnHai Doan, Adel Ardalan, Jeffrey R. Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeffrey F. Naughton, Shishir Prasad, Ganesh Krishnan, Rohit Deep, and Vijay Raghavendra. 2016. Magellan: Toward Building Entity Matching Management Systems. PVLDB 9, 12 (2016), 1197-1208.
  41. Hanna Köpcke, Andreas Thor, and Erhard Rahm. 2010. Evaluation of entity resolution approaches on real-world match problems. PVLDB 3, 1 (2010), 484-493.
  42. Simon Lacoste-Julien, Konstantina Palla, Alex Davies, Gjergji Kasneci, Thore Graepel, and Zoubin Ghahramani. 2013. SIGMa: simple greedy matching for aligning large knowledge bases. In KDD. 572-580.
  43. Guoliang Li, Yudian Zheng, Ju Fan, Jiannan Wang, and Reynold Cheng. 2017. Crowdsourced Data Management: Overview and Challenges. In SIGMOD. 1711-1716.
  44. Juanzi Li, Jie Tang, Yi Li, and Qiong Luo. 2009. RiMOM: A Dynamic Multi- strategy Ontology Alignment Framework. TKDE 21, 8 (2009), 1218-1232.
  45. Jayant Madhavan, Philip A. Bernstein, and Erhard Rahm. 2001. Generic Schema Matching with Cupid. In VLDB. 49-58.
  46. Ruhaila Maskat, Norman W. Paton, and Suzanne M. Embury. 2016. Pay-as-you- go Configuration of Entity Resolution. T. Large-Scale Data-and Knowledge- Centered Systems (2016), 40-65.
  47. Davide Mottin, Matteo Lissandrini, Yannis Velegrakis, and Themis Palpanas. 2016. Exemplar queries: a new way of searching. VLDB J. 25, 6 (2016), 741-765.
  48. Sidharth Mudgal, Han Li, Theodoros Rekatsinas, AnHai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, and Vijay Raghavendra. 2018. Deep Learning for Entity Matching: A Design Space Exploration. In SIGMOD. 19-34.
  49. Kevin O'Hare, Anna Jurek, and Cassio de Campos. 2018. A new technique of selecting an optimal blocking method for better record linkage. Inf. Syst. 77 (2018), 151-166.
  50. George Papadakis, George Alexiou, George Papastefanatos, and Georgia Koutrika. 2015. Schema-agnostic vs Schema-based Configurations for Blocking Methods on Homogeneous Data. PVLDB 9, 4 (2015), 312-323.
  51. George Papadakis, Ekaterini Ioannou, Claudia Niederée, Themis Palpanas, and Wolfgang Nejdl. 2012. Beyond 100 million entities: large-scale blocking-based resolution for heterogeneous data. In WSDM. 53-62.
  52. George Papadakis, Ekaterini Ioannou, Themis Palpanas, Claudia Niederée, and Wolfgang Nejdl. 2013. A Blocking Framework for Entity Resolution in Highly Heterogeneous Information Spaces. TKDE 25, 12 (2013), 2665-2682.
  53. George Papadakis, Georgia Koutrika, Themis Palpanas, and Wolfgang Nejdl. 2014. Meta-Blocking: Taking Entity Resolutionto the Next Level. TKDE 26, 8 (2014), 1946-1960.
  54. George Papadakis and Wolfgang Nejdl. 2011. Efficient entity resolution meth- ods for heterogeneous information spaces. In ICDE Workshops. 304-307.
  55. George Papadakis, George Papastefanatos, and Georgia Koutrika. 2014. Su- pervised Meta-blocking. PVLDB 7, 14 (2014), 1929-1940.
  56. George Papadakis, Jonathan Svirsky, Avigdor Gal, and Themis Palpanas. 2016. Comparative Analysis of Approximate Blocking Techniques for Entity Reso- lution. PVLDB 9, 9 (2016), 684-695.
  57. George Papadakis, Leonidas Tsekouras, Emmanouil Thanos, George Gian- nakopoulos, Themis Palpanas, and Manolis Koubarakis. 2018. The return of JedAI: End-to-End Entity Resolution for Structured and Semi-Structured Data. PVLDB 11, 12 (2018), 1950-1953.
  58. Thorsten Papenbrock, Arvid Heise, and Felix Naumann. 2015. Progressive Duplicate Detection. TKDE 27, 5 (2015), 1316-1329.
  59. Vibhor Rastogi, Nilesh N. Dalvi, and Minos N. Garofalakis. 2011. Large-Scale Collective Entity Matching. PVLDB 4, 4 (2011), 208-218.
  60. Orion Fausto Reyes-Galaviz, Witold Pedrycz, Ziyue He, and Nick J. Pizzi. 2017. A supervised gradient-based learning algorithm for optimized entity resolution. DKE (2017).
  61. Anish Das Sarma, Ankur Jain, Ashwin Machanavajjhala, and Philip Bohannon. 2012. An automatic blocking mechanism for large-scale de-duplication tasks. In CIKM. 1055-1064.
  62. Giovanni Simonini, Sonia Bergamaschi, and H. V. Jagadish. 2016. BLAST: a Loosely Schema-aware Meta-blocking Approach for Entity Resolution. PVLDB 9, 12 (2016), 1173-1184.
  63. Giovanni Simonini, George Papadakis, Themis Palpanas, and Sonia Berga- maschi. 2019. Schema-Agnostic Progressive Entity Resolution. IEEE Trans. Knowl. Data Eng. 31, 6, 1208-1221.
  64. Yannis Sismanis, Ling Wang, Ariel Fuxman, Peter J. Haas, and Berthold Rein- wald. 2009. Resolution-Aware Query Answering for Business Intelligence. In ICDE. 976-987.
  65. Kostas Stefanidis, Vasilis Efthymiou, Melanie Herschel, and Vassilis Christophides. 2014. Entity resolution in the web of data. In WWW.
  66. Fabian M. Suchanek, Serge Abiteboul, and Pierre Senellart. 2011. PARIS: Probabilistic Alignment of Relations, Instances, and Schema. PVLDB 5, 3 (2011), 157-168.
  67. Vasilis Verroios and Hector Garcia-Molina. 2015. Entity Resolution with crowd errors. In ICDE. 219-230.
  68. Vasilis Verroios, Hector Garcia-Molina, and Yannis Papakonstantinou. 2017. Waldo: An Adaptive Human Interface for Crowd Entity Resolution. In SIGMOD. 1133-1148.
  69. Norases Vesdapunt, Kedar Bellare, and Nilesh N. Dalvi. 2014. Crowdsourcing Algorithms for Entity Resolution. PVLDB 7, 12 (2014), 1071-1082.
  70. Jiannan Wang, Tim Kraska, Michael J. Franklin, and Jianhua Feng. 2012. Crow- dER: Crowdsourcing Entity Resolution. PVLDB 5, 11 (2012), 1483-1494.
  71. Jiannan Wang, Sanjay Krishnan, Michael J. Franklin, Ken Goldberg, Tim Kraska, and Tova Milo. 2014. A sample-and-clean framework for fast and accurate query processing on dirty data. In SIGMOD. 469-480.
  72. Jiannan Wang, Guoliang Li, Tim Kraska, Michael J. Franklin, and Jianhua Feng. 2013. Leveraging transitive relations for crowdsourced joins. In SIGMOD. 229-240.
  73. Sibo Wang, Xiaokui Xiao, and Chun-Hee Lee. 2015. Crowd-Based Deduplica- tion: An Adaptive Approach. In SIGMOD. 1263-1277.
  74. Steven Euijong Whang and Hector Garcia-Molina. 2014. Incremental entity resolution on rules and data. VLDB J. 23, 1 (2014), 77-102.
  75. Steven Euijong Whang, Peter Lofgren, and Hector Garcia-Molina. 2013. Ques- tion Selection for Crowd Entity Resolution. PVLDB 6, 6 (2013), 349-360.
  76. Steven Euijong Whang, David Marmaros, and Hector Garcia-Molina. 2013. Pay-As-You-Go Entity Resolution. TKDE 25, 5 (2013), 1111-1124.
  77. Wei Yan, Yuan Xue, and Bradley Malin. 2013. Scalable load balancing for mapreduce-based record linkage. In IPCCC. 1-10.
  78. Chen Jason Zhang, Rui Meng, Lei Chen, and Feida Zhu. 2015. CrowdLink: An Error-Tolerant Model for Linking Complex Records. In ExploreDB. 15-20.

Related papers

An Overview of End-to-End Entity Resolution for Big Data
Vassilis Christophides

ACM Computing Surveys

One of the most critical tasks for improving data quality and increasing the reliability of data analytics is Entity Resolution (ER), which aims to identify different descriptions that refer to the same real-world entity. Despite several decades of research, ER remains a challenging problem. In this survey, we highlight the novel aspects of resolving Big Data entities when we should satisfy more than one of the Big Data characteristics simultaneously (i.e., Volume and Velocity with Variety). We present the basic concepts, processing steps, and execution strategies that have been proposed by database, semantic Web, and machine learning communities in order to cope with the loose structuredness , extreme diversity , high speed, and large scale of entity descriptions used by real-world applications. We provide an end-to-end view of ER workflows for Big Data, critically review the pros and cons of existing methods, and conclude with the main open research directions.

View PDFchevron_right
Flexible and Efficient Distributed Resolution of Large Entities
András József Molnár

Lecture Notes in Computer Science, 2012

Entity resolution (ER) is a computationally hard problem of data integration scenarios, where database records have to be grouped according to the real-world entities they belong to. In practice these entities may consist of only a few records from different data sources with typos or historical data. In other cases they may contain significantly more records, especially when we search for entities on a higher level of a concept hierarchy than records. In this paper we give theoretical foundation of a variety of practically important match functions. We show that under these formulations, ER with large entities can be solved efficiently with algorithms based on MapReduce, a distributed computing paradigm. Our algorithm can efficiently incorporate probabilistic and similarity-based record match, enabling flexible match function definition. We demonstrate the usability of our model and algorithm in a real-world insurance ER scenario, where we identify household groups of client records.

View PDFchevron_right
Large-Scale Entity Resolution for Semantic Web Data Integration
Gustavo De Assis Costa, Jose M Parente Oliveira

Despite all the advances, one of the main challenges for the consolidation of the Web of Data is data integration, a key aspect to semantic web data management. Most of the solutions make use of entity resolution, a task that deals with identifying and linking different manifestations of the same real world object in one or more datasets. However, data are usually incomplete, inconsistent and contain outliers and, to overcome these limitations, it is necessary to explore as much as possible the existent patterns in data. One way to extrapolate the commonly used technique of pair-wise matching is to explore the relationship structure between entities. Moreover, with the billions of RDF triples being published in the Web, scale has become a problem, posing some new challenges. Only recently some works started to consider new strategies that can deal with the problem of entity resolution in high scale datasets. In this paper we describe a Map-Reduce strategy for a relational learning a...

View PDFchevron_right
End-to-End Entity Resolution for Big Data: A Survey
Vassilis Christophides

ArXiv, 2019

One of the most important tasks for improving data quality and the reliability of data analytics results is Entity Resolution (ER). ER aims to identify different descriptions that refer to the same real-world entity, and remains a challenging problem. While previous works have studied specific aspects of ER (and mostly in traditional settings), in this survey, we provide for the first time an end-to-end view of modern ER workflows, and of the novel aspects of entity indexing and matching methods in order to cope with more than one of the Big Data characteristics simultaneously. We present the basic concepts, processing steps and execution strategies that have been proposed by different communities, i.e., database, semantic Web and machine learning, in order to cope with the loose structuredness, extreme diversity, high speed and large scale of entity descriptions used by real-world applications. Finally, we provide a synthetic discussion of the existing approaches, and conclude with...

View PDFchevron_right
An Effective Entity Resolution Approach for Big Data
Ali El-bastawissy

International Journal of Innovative Technology and Exploring Engineering

Entity Resolution (ER) is defined as the process 0f identifying records/ objects that correspond to real-world objects/ entities. To define a good ER approach, the schema of the data should be well-known. In addition, schema alignment of multiple datasets is not an easy task and may require either domain expert or ML algorithm to select which attributes to match. Schema agnostic meta-blocking tries to solve such a problem by considering each token as a blocking key regardless of the attributes it appears in. It may also be coupled with meta-blocking to reduce the number of false negatives. However, it requires the exact match of tokens which is very hard to occur in the actual datasets and it results in very low precision. To overcome such issues, we propose a novel and efficient ER approach for big data implemented in Apache Spark. The proposed approach is employed to avoid schema alignment as it treats the attributes as a bag of words and generates a set of n-grams which is transf...

View PDFchevron_right

Related topics

  • Computer Science
  • Relational Database
  • Information Retrieval
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved