Managing data quality by identifying the noisiest data samples

Data quality (DQ) assessment and improvement in larger information systems would often not be feasible without using suitable " DQ methods " , which are algorithms that can be automatically executed by computer systems to detect and/or correct problems in datasets. Currently, these methods are already essential, and they will be of even greater importance as the quantity of data in organisational systems grows. This paper provides a review of existing methods for both DQ assessment and improvement and classifies them according to the DQ problem and problem context. Six gaps have been identified in the classification, where no current DQ methods exist, and these show where new methods are required as a guide for future research and DQ tool development.

Annual Conference of the PHM Society, 2013

In large industries usage of advanced technological methods and modern equipment comes with the problem of storing, interpreting and analyzing huge amount of information. Handling information becomes more complicated and important at the same time. So, data quality is one of major challenges considering a rapid growth of information, fragmentation of information systems, incorrect data formatting and other issues. The aim of this paper is to describe industrial data processing and analytics on the realworld use case. The most crucial data quality issues are described, examined and classified in terms of Data Quality Dimensions. Factual industrial information supports and illustrates each encountered data deficiency. In addition, we describe methods for elimination data quality issues and data analysis techniques, which are applied after cleaning data procedure. In addition, an approach to address data quality problems in large-scale industrial datasets is proposed. This techniques and methods comprise several well-known techniques, which come from both worlds of mathematical logic and also statistics, improving data quality procedure and cleaning results.

International Journal of Knowledge-Based Organizations, 2011

The quality of real world data that is being fed into a data warehouse is a major concern of today. As the data comes from a variety of sources before loading the data in the data warehouse, it must be checked for errors and anomalies. There may be exact duplicate records or approximate duplicate records in the source data. The presence of incorrect or inconsistent data can significantly distort the results of analyses, often negating the potential benefits of information-driven approaches. This paper addresses issues related to detection and correction of such duplicate records. Also, it analyzes data quality and various factors that degrade it. A brief analysis of existing work is discussed, pointing out its major limitations. Thus, a new framework is proposed that is an improvement over the existing technique.

The value of information depends directly on the quality of the data used. Decisions are no better than the data on which they are based. How can organizations assess the quality of their information? How can they know if their data are useful?

30

… Science and Information Technology, 2008. IMCSIT …, 2008

Data cleaning is a process of maintaining data quality in information systems. Current data cleaning solutions require reference data to identify incorrect or duplicate entries. This article proposes usage of data mining in the area of data cleaning as effective in discovering reference data and validation rules from the data itself. Two algorithms designed by the author for data attribute correction have been presented. Both algorithms utilize data mining methods. Experimental results show that both algorithms can effectively clean text attributes without external reference data.

Proceedings of the 16th International Conference on Applied Computing 2019, 2019

Today, the quantity of data continues to increase; furthermore, the data are distributed and heterogeneous, from multiple sources (structured, semi-structured and unstructured) and with different levels of quality. Therefore, it is very likely to manipulate data without knowledge about their structures and their semantics. The subject covered in this paper aims at assisting the user in its quality approach. The data must be related to its semantic meaning, data types, constraints, comments and origin. We deal with the semantic schema recognition of a data source. It consists of categorizing the data by assigning it to a category and possibly a sub-category, and secondly, of establishing relations between columns and possibly discovering the semantics of the manipulated data source. These links detected between columns offer a better understanding of the source and the alternatives for correcting data.

School of Electrical Engineering Computer Science Science Engineering Faculty, 2014

The quality of the data in organisational information systems is a critical issue for organisations. The rapid growth in the volume and complexities of technologies related to databases and data warehouses has enabled executives and decision makers to more readily store, access, analyse and retrieve massive amounts of information to support business needs. For the most part, advantages, such as significantly improved organisational capacity, increased performance efficiency and customer satisfaction, as well as reduced operational times and costs have been the principal result. However, despite these obvious benefits, a major challenge remains, obstructing the proper delivery of these goals; this is the data quality. It is estimated that the immediate cost of a 1-5% error in data is approximately 10% of revenue. Poor data quality also has a severe impact on customer satisfaction, operating costs, effective decision making, and strategy execution. Data quality affects many applications that facilitate data mining, database and business management. Such applications contain many interesting algorithms and techniques with which to approach the multi-dimensional problems that impact on data quality. While these methods have contributed somewhat to solving some data quality problems, they have serious limitations particularly for mining outlier data. Mining massive volumes of data to expose infrequent outlier values i based on traditional data mining tasks, such as association rule is computationally expensive. The problems that have arisen are that traditional data mining tasks are mainly designed to manage frequent data, and the focus of most data mining research is on post-process tasks; in which improving the accuracy of the data mining algorithms is desirable. Such solutions are inadequate and inapplicable when it comes to providing continuous automated solutions for detecting outlier data. This research describes the development of a robust and novel prototype to address the quality problems that relate to the dimension of outlier data. It thoroughly investigates the associated problems with regards to detecting, assessing and determining the severity of the problem of outlier data. To address these problems, the study proposes new techniques based on granule mining, as an alternative to association rule mining, which is based on decision table theory. The proposed method is innovative and significant and has the potential to reduce the time and the costs associated with mining and assessing outlier data. Substantial experiments have demonstrated that the proposed algorithms for outlier data outperform state-of-the-art algorithms. The contribution of the experiments described herein will be to enable organisations to effectively detect outlier data and thereby assess the behaviour of data quality in a continuous automated or (a semi automated) way. ii I would like to dedicate this thesis to my loving family.

2009

Abstract. Poor quality data may be detected and corrected by performing various quality assurance activities that rely on techniques with different efficacy and cost. In this paper, we propose a quantitative approach for measuring and comparing the effectiveness of these data quality (DQ) techniques. Our definitions of effectiveness are inspired by measures proposed in Information Retrieval.

Autonomy Heterogeneity no yes totally semi DIS DW & MIS VMS CIS RS P2P no