Multimodal Deception Detection

Wiley Handbook of Science and Technology for Homeland Security, 2008

Human intelligence is the key to stopping terrorism, and therefore it is essential to know when the information obtained is false. This chapter briefly outlines the research on behavioral clues to deception, as well as research on people's abilities to spot deception once it has happened. We find that there is no clue or clue pattern that is specific to deception, although there are clues specific to emotion and cognition. In general, behavioral clues are only limited in their abilities to identify deception and that there are still behavioral measurement issues that may plague research on deception. Moreover, a closer examination of the laboratory research suggests many research studies are not relevant to security contexts, thus the research literature may underestimate the usefulness of behavioral information-particularly for the utility of identifying emotional and cognitive states. We also find that most people, unaided by technology, cannot detect lies from behavior information, but that some groups do show significantly higher levels of accuracy-although more research is needed to understand why. We conclude that more directed interaction with scientists and practitioners-in both lab work and in the real world, in creating real world databases, in identifying base rates for malfeasant behavior in security settings, in optimizing training and in identifying pre existing excellence within security organizations can more rapidly capitalize on the usefulness of behavioral information in security settings.

2012

We thank those who kindly volunteered to participate in the study. We also thank David McCormick for encouragement and support and Aldert Vrij, University of Portsmouth, for helpful discussions and advice. 2 Summary This paper presents the development of a computerised, non-invasive psychological profiling system, „Silent Talker‟, for the analysis of nonverbal behaviour. Nonverbal signals hold rich information about mental, behavioural and/or physical states. Previous attempts to extract individual signals and to classify an overall behaviour have been time-consuming, costly, biased, error-prone and complex. Silent Talker overcomes these problems by the use of Artificial Neural Networks. The testing and validation of the system was undertaken by detecting processes associated with “deception ” and “truth”. In a simulated theft scenario thirty-nine participants „stole‟ (or didn‟t) money, and were interviewed about its location. Silent Talker was able to detect different behaviour pat...

Engineering and TechnologyJournal, 2025

ABSTRACT:Within the realm of deception detection research,this comparative study investigates the use of machine learning, artificial intelligence, and multimodal data processing. From the year 2020 to the year 2024, it focuses on twenty-four studies that show the growing potential of AI-driven systems in terms of enhancing the consistency, scalability, and accuracy of fraud detection. In order to identify deceit in a variety of data types, such as facial expressions, audio signals, written language, and behavioral abnormalities, different techniques have showed promise. Some of these techniques include Support Vector Machines (SVM), Long Short-Term Memory (LSTM) networks, Convolutional Neural Networks (CNNs), and hybrid models. On the other hand, issues like as adversary manipulation, biases in training datasets,and the potential for deception cues to be generalized across linguistic, cultural, and social contexts continue to be a concern. To further complicate the deployment of deception detection systems, there is a dearth of real-world validation, and the present models have little adaptability in dynamic environments. The article places an emphasis on the need of openness in the design of artificial intelligence, ethical concerns about user privacy, and the development of systems that have properties that are sensitive to cultural and environmental factors. The integration of concepts from other disciplines, the ability to withstand assaults from adversaries, and the development of ways to decrease prejudice should be the primary focus of research in the future

Research Square (Research Square), 2024

Detecting deception reliably has been a sought-after goal for researchers since the early 20th century. This is due to its high stakes nature, especially when considering the setting under which a deception detection system would be utilized, such as law enforcement sectors, judicial bodies, and criminal investigation bodies. Therefore, recent literature has greatly focused on developing systems with ever increasing accuracies to minimize false positives. Attempts to diversify the sources of data being analyzed and classi ed as well as the reliance on arti cial intelligence technologies have shown great success. But little attention was paid to the applicability of these systems in real-life scenarios. an idle deception detection system needs to exhibit accuracy but also perform in real-time, a feature that is lacking in the current state-of-the-art. In this work, a non-invasive real-time multimodal deception detection system is developed using advanced machine learning techniques. It combines data sources from video and audio streams to extract visual, acoustic, and linguistic features. It also utilizes parallel computing techniques to ensure high performance adequate for real-time usage. Furthermore, a userfriendly graphical user interface was built to facilitate the use of the system. Multiple experiments were conducted to determine its accuracy under various circumstances and combinations of features, the system had a detection accuracy of 99.83% under real-life, high-stakes scenarios using visual and acoustic features, and 91.03% accuracy under controlled environments, while an 89.54% accuracy was achieved using mixed environments and using visual, acoustic, and linguistic features.

2019

Recently, detecting liars and extracting features which distinguish them from truth-tellers have been the focus of a wide range of disciplines. To the author's best knowledge, most of the work has been done on facial expressions and body gestures but only few works have been done on the language used by both liars and truth-tellers. This paper sheds light on four axes. The first axis copes with building an audio corpus for deceptive and truthful speech for Egyptian Arabic speakers. The second axis focuses on examining the human perception of lies and proving our need for computational linguistic-based methods to extract features which characterize truthful and deceptive speech. The third axis is concerned with building a linguistic analysis program that could extract from the corpus the inter- and intra-linguistic cues for deceptive and truthful speech. The program built here is based on selected categories from the Linguistic Inquiry and Word Count program. Our results demonstr...

2004

An assumption undergirding decision making and negotiation regarding human interaction, that people's communication is determined to be truthful, has increasingly come under assault by daily reports of duplicity, scams, exaggerated claims, chicanery, fraud, exposed secrets, false identities, and sundry other forms of deception. Misrepresentations of facts, concealment of discrediting information, fraudulent reports, equivocal or strategically ambiguous messages, verbalisms meant to baffle, irrelevant comments intended to derail lines of inquiry-these and more are means by which deceit can infiltrate and sabotage interpersonal interactions, group decisions and negotiations. It is this timely issue that is the topic of the current issue, which is the fourth in a series examining deception in interpersonal and group interactions in face-to-face and online environments. Also part of this focus on deception are computer-aided tools for detecting deceit that may be of utility to group decisions and negotiations, whether used by researchers, analysts, or the participants themselves. The three preceding special issues on this topic included 14 articles from a wide array of authors representing such disciplines as information systems, communication, psychology, linguistics and computer science. The papers addressed topics ranging from theoretical models of deception and its detection, to behavioral indicators of deception and computer-aided tools for their detection, to the impact of such moderators on detection success as communication modality, group size, web experience, and motivation. Virtually all of the articles published so far either advanced a theory or were guided by a theoretical stance. In the first issue, Carlson, George, Burgoon, Adkins and White led with "Deception in Computer-Mediated Communication." Their theoretical synthesis merged interpersonal deception theory with channel expansion theory and

PLOS ONE, 2023

Several studies applying Machine Learning to deception detection have been published in the last decade. A rich and complex set of settings, approaches, theories, and results is now available. Therefore, one may find it difficult to identify trends, successful paths, gaps, and opportunities for contribution. The present literature review aims to provide the state of research regarding deception detection with Machine Learning. We followed the PRISMA protocol and retrieved 648 articles from ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. 540 of them were screened (108 were duplicates). A final corpus of 81 documents has been summarized as mind maps. Metadata was extracted and has been encoded as Python dictionaries to support a statistical analysis scripted in Python programming language, and available as a collection of Jupyter Lab Notebooks in a GitHub repository. All are available as Jupyter Lab Notebooks. Neural Networks, Support Vector Machines, Random Forest, Decision Tree and K-nearest Neighbor are the five most explored techniques. The studies report a detection performance ranging from 51% to 100%, with 19 works reaching accuracy rate above 0.9. Monomodal, Bimodal, and Multimodal approaches were exploited and achieved various accuracy levels for detection. Bimodal and Multimodal approaches have become a trend over Monomodal ones, although there are high-performance examples of the latter. Studies that exploit language and linguistic features, 75% are dedicated to English. The findings include observations of the following: language and culture, emotional features, psychological traits, cognitive load, facial cues, complexity, performance, and Machine Learning topics. We also present a dataset benchmark. Main conclusions are that labeled datasets from real-life data are scarce. Also, there is still room for new approaches for deception detection with Machine Learning, especially if focused on languages and cultures other than English-based. Further research would greatly contribute by providing new labeled and multimodal datasets for deception detection, both for English and other languages.

2008 Cairo International Biomedical Engineering Conference, 2008

Current lie detection methods, based on the polygraph technique, rely upon the measurement of several physiological characteristics to discriminate whether a truth or a lie is expressed. Direct electro-encephalographic (EEG) measurements of event-related brain potentials (ERP) to particular questions or situations can also be used. The purpose of this study is to evaluate RT (Reaction Time) analysis and combined RT and ERP to identify participants is possessing specific guilty knowledge. The accuracy of BART (bootstrapped analysis of reaction time) method is 81% which are comparable to 80% accuracy of ERP analysis (Wavelet classifier method) in previous study. BART-BAD method has the best accuracy compared to multimodal methods and has superior results compared to the single mode analysis (BAD method). BAD-BART method proposed as best approach for Assess "Guilty Knowledge".

Human Communication Research, 2004

This experiment examined children's and undergraduates' verbal and nonverbal deceptive behavior, and the extent to which their truths and lies could be correctly classified by paying attention to these responses. A total of 196 participants (aged 5-6, 10-11, 14-15, and undergraduates) participated in an erasing the blackboard event, and told the truth or lied about the event afterwards. Nonverbal and verbal responses were coded, the latter with Criteria-Based Content Analysis and Reality Monitoring. Although children and undergraduates demonstrated different behaviors (for example, the children obtained lower CBCA scores and made more movements), actual cues to deceit were remarkably similar across different age groups (for example, both 5-6-year-olds and undergraduates obtained lower CBCA scores and made fewer movements while lying). A combination of verbal and nonverbal lie detection methods resulted in more correct classifications of liars and truth tellers than the verbal and nonverbal lie detection methods individually, with the combined method obtaining hit rates as high as 88%. Deceptive responses 2 Detecting Deceit Via Analyses of Verbal and Nonverbal Behavior in Children and Adults Although it sounds reasonable to suggest that looking at a combination of nonverbal and verbal behaviors ("speech content") will lead to more accurate classifications of liars and truth tellers than investigating nonverbal and verbal behaviors separately, researchers rarely investigate both types simultaneously. Vrij, Edward, Roberts, and Bull (2000) examined (adult) participants' nonverbal and verbal behavior (the latter investigated with Criteria-Based Content Analysis and Reality Monitoring) and obtained the most accurate classifications of liars and truth tellers when both the nonverbal and verbal behaviors were taken into account. Similarly, in the present study, both nonverbal and verbal responses of liars and truth tellers were investigated. However, the present study differs from Vrij et al.'s (2000) study in several ways. Perhaps the most important difference was that in the present study apart from adults (undergraduates) children also participated, and this seems the first study in which truthful and deceptive verbal and nonverbal behaviors of both children and adults were investigated and compared. Research examining children's nonverbal behavior relating to deceit is rare, and less than a handful of studies have been published to date (see Vrij, 2002, for a review). Child deception research has mainly focused on questions such as whether children lie, why they lie and when they lie (Frank, 1992). At the same time, examining children's behavior while lying is important for both theoretical and practical reasons, the latter due at least in part to the fact that police officers, social workers, judges and juries are sometimes confronted with the question of whether a child is lying or telling the truth. Theoretical Reasons to Explain Deceptive Nonverbal Behavior Three perspectives (the emotional, cognitive complexity and attempted behavioral control perspectives) seem particularly important for predicting and explaining nonverbal

Deception in computer-mediated communication is defined as a message knowingly and intentionally transmitted by a sender to foster a false belief or conclusion by the perceiver. Stated beliefs about deception and deceptive messages or incidents are content analyzed in a sample of 324 computer-mediated communications. Relevant stated beliefs are obtained through systematic sampling and querying of the blogosphere based on 80 English words commonly used to describe deceptive incidents. Deception is conceptualized broader than lying and includes a variety of deceptive strategies: falsification, concealment (omitting material facts) and equivocation (dodging or skirting issues). The stated beliefs are argued to be valuable toward the creation of a unified multi-faceted ontology of deception, stratified along several classificatory facets such as (1) contextual domain (e.g., personal relations, politics, finances & insurance), (2) deception content (e.g., events, time, place, abstract notions), (3) message format (e.g., a complaint: they lied to us, a victim story: I was lied to or tricked, or a direct accusation: you're lying), and (4) deception variety, each tied to particular verbal cues (e.g., misinforming, scheming, misrepresenting, or cheating). The paper positions automated deception detection within the field of library and information science (LIS), as a feasible natural language processing (NLP) task. Key findings and important constructs in deception research from interpersonal communication, psychology, criminology, and language technology studies are synthesized into an overview. Deception research is juxtaposed to several benevolent constructs in LIS research: trust, credibility, certainty, and authority. "If falsehood, like truth, had only one face, we would be in a better shape. For we would take as certain the opposite of what the liar said. But the reverse of truth has a hundred thousand shapes and a limitless field." (Of Liars, p.24).