Poisoning AI Models: New Frontiers in Data Manipulation Attacks

Machine Learning, 2010

Machine learning's ability to rapidly evolve to changing and complex situations has helped it become a fundamental tool for computer security. That adaptability is also a vulnerability: attackers can exploit machine learning systems. We present a taxonomy identifying and analyzing attacks against machine learning systems. We show how these classes influence the costs for the attacker and defender, and we give a formal structure defining their interaction. We use our framework to survey and analyze the literature of attacks against machine learning systems. We also illustrate our taxonomy by showing how it can guide attacks against SpamBayes, a popular statistical spam filter. Finally, we discuss how our taxonomy suggests new lines of defenses.

AI in Cybersecurity, 2018

Artificial intelligence and machine learning capabilities are growing at an unprecedented rate. These technologies have many widely beneficial applications, ranging from machine translation to medical image analysis. Countless more such applications are being developed and can be expected over the long term. Less attention has historically been paid to the ways in which artificial intelligence can be used maliciously. This report surveys the landscape of potential security threats from malicious uses of artificial intelligence technologies, and proposes ways to better forecast, prevent, and mitigate these threats. We analyze, but do not conclusively resolve, the question of what the long-term equilibrium between attackers and defenders will be. We focus instead on what sorts of attacks we are likely to see soon if adequate defenses are not developed.

2019 1st International Informatics and Software Engineering Conference (UBMYK), 2019

At the present time, machine learning methods have been becoming popular and the usage areas of these methods have also increased with this popularity. The machine learning methods are expected to increase in the cyber security components like firewalls, antivirus software etc. Nowadays, the use of this type of machine learning methods brings with it various risks. Attackers develop different methods to manipulate different systems, not only cyber security components, but also image detection systems. Therefore, securing machine learning models has become critical. In this paper, we demonstrate a data poisoning attack towards classification method of machine learning models and we also proposed a defense algorithm which makes machine learning models more robust against data poisoning attacks. In this study, we have conducted data poisoning attacks on MNIST, a widely used character detection data set. Using the poisoned MNIST dataset, we built classification models more reliable by using a generative model such as AutoEncoder.

2020

Artificial intelligence (AI) applications driven by machine learning (ML) are transformational technologies within the international nuclear security regime. Advancements realized by AI-faster and improved data insights, more efficient and automated processes, reductions in human error-enable nuclear security applications such as behavior analysis for insider threat mitigation, source tracking of stolen nuclear material, and facial recognition software for physical protection. In addition to the advantages, however, there are also inherent vulnerabilities and threats associated with its use and risk mitigations must be built into any AI/ML-enabled systems. This work provides a background on AI and ML and different data types used in the field, including open-source intelligence information (OSINT) that is discoverable by AI tools and application data that are used by AI tools for decision-making and automation. Current and potential AI applications and vulnerabilities related to their use within the nuclear security regime are also discussed. Key findings include: i. Data harvesting of OSINT can be performed offensively by adversaries for reconnaissance and future attack development as well as defensively for proactive identification of data and system security lapses. Nuclear security regimes must apply information security standards and best practices to reduce these vulnerabilities. ii. AI systems often use large amounts of data, whose quality are very important. Poor quality data can yield, at best, inaccurate results leading to improper decisions or, at worst, inaccurate (and dangerous) results leading to nuclear sabotage, theft of nuclear material, or personnel injury. iii. While the use of AI within nuclear security regimes is currently limited, many new applications could be developed introducing new vulnerabilities in the regime. It is imperative that AI application development follows best practices in AI application and data security to reduce the introduction of unmitigated threats and vulnerabilities. The principal recommendation for INS is to include information on the unique and enhanced data vulnerabilities introduced by AI/ML applications in appropriate cyber trainings, workshops, and technical exchanges.

Ubiquity, 2018

While machine learning has proven to be promising in several application domains, our understanding of its behavior and limitations is still in its nascent stages. One such domain is that of cybersecurity, where machine learning models are replacing traditional rule based systems, owing to their ability to generalize and deal with large scale attacks which are not seen before. However, the naive transfer of machine learning principles to the domain of security needs to be taken with caution. Machine learning was not designed with security in mind and as such is prone to adversarial manipulation and reverse engineering. While most data based learning models rely on a static assumption of the world, the security landscape is one that is especially dynamic, with an ongoing never ending arms race between the system designer and the attackers. Any solution designed for such a domain needs to take into account an active adversary and needs to evolve over time, in the face of emerging thre...

Computer Vision – ECCV 2020, 2020

Data poisoning attacks on machine learning models have attracted much recent attention, wherein poisoning samples are injected at the training phase to achieve adversarial goals at test time. Although existing poisoning techniques prove to be effective in various scenarios, they rely on certain assumptions on the adversary knowledge and capability to ensure efficacy, which may be unrealistic in practice. This paper presents a new, practical targeted poisoning attack method on neural networks in vision domain, namely BlackCard. BlackCard possesses a set of critical properties for ensuring attacking efficacy in practice, which has never been simultaneously achieved by any existing work, including knowledge-oblivious, clean-label, and clean-test. Importantly, we show that the effectiveness of BlackCard can be intuitively guaranteed by a set of analytical reasoning and observations, through exploiting an essential characteristic of gradient-descent optimization which is pervasively adopted in DNN models. We evaluate the efficacy of BlackCard for generating targeted poisoning attacks via extensive experiments using various datasets and DNN models. Results show that BlackCard is effective with a rather high success rate while preserving all the claimed properties.

2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS)

Data poisoning is a type of adversarial attack on training data where an attacker manipulates a fraction of data to degrade the performance of machine learning model. Therefore, applications that rely on external data-sources for training data are at a significantly higher risk. There are several known defensive mechanisms that can help in mitigating the threat from such attacks. For example, data sanitization is a popular defensive mechanism wherein the learner rejects those data points that are sufficiently far from the set of training instances. Prior work on data poisoning defense primarily focused on offline setting, wherein all the data is assumed to be available for analysis. Defensive measures for online learning, where data points arrive sequentially, have not garnered similar interest. In this work, we propose a defense mechanism to minimize the degradation caused by the poisoned training data on a learner's model in an online setup. Our proposed method utilizes an influence function which is a classic technique in robust statistics. Further, we supplement it with the existing data sanitization methods for filtering out some of the poisoned data points. We study the effectiveness of our defense mechanism on multiple datasets and across multiple attack strategies against an online learner.

International Journal of General Systems, 2019

Data integrity is a key component of effective Bayesian network structure learning algorithms, namely PC algorithm, design and use. Given the role that integrity of data plays in these outcomes, this research demonstrates the importance of data integrity as a key component in machine learning tools in order to emphasize the need for carefully considering data integrity during tool development and utilization. To meet this purpose, we study how an adversary could generate a desired network with the PC algorithm. Given a Bayesian network B 1 and a database DB 1 generated by B 1 and a second Bayesian network, B 2 , which is equal to B 1 , except for a minor change like a missing link, a reversed link, or an additional link, we explore and analyze what is the minimal number of changes such as additions, deletions, substitutions to DB 1 that lead to a database DB 2 that, when given as input to PC algorithm, results in B 2 .

Sustainability, 2020

With the increasing popularity of the Internet of Things (IoT) platforms, the cyber security of these platforms is a highly active area of research. One key technology underpinning smart IoT systems is machine learning, which classifies and predicts events from large-scale data in IoT networks. Machine learning is susceptible to cyber attacks, particularly data poisoning attacks that inject false data when training machine learning models. Data poisoning attacks degrade the performances of machine learning models. It is an ongoing research challenge to develop trustworthy machine learning models resilient and sustainable against data poisoning attacks in IoT networks. We studied the effects of data poisoning attacks on machine learning models, including the gradient boosting machine, random forest, naive Bayes, and feed-forward deep learning, to determine the levels to which the models should be trusted and said to be reliable in real-world IoT settings. In the training phase, a lab...