Human Behavior Analysis by Means of Multimodal Context Mining
https://doi.org/10.3390/S16081264Abstract
There is sufficient evidence proving the impact that negative lifestyle choices have on people's health and wellness. Changing unhealthy behaviours requires raising people's self-awareness and also providing healthcare experts with a thorough and continuous description of the user's conduct. Several monitoring techniques have been proposed in the past to track users' behaviour; however, these approaches are either subjective and prone to misreporting, such as questionnaires, or only focus on a specific component of context, such as activity counters. This work presents an innovative multimodal context mining framework to inspect and infer human behaviour in a more holistic fashion. The proposed approach extends beyond the state-of-the-art, since it not only explores a sole type of context, but also combines diverse levels of context in an integral manner. Namely, low-level contexts, including activities, emotions and locations, are identified from heterogeneous sensory data through machine learning techniques. Low-level contexts are combined using ontological mechanisms to derive a more abstract representation of the user's context, here referred to as high-level context. An initial implementation of the proposed framework supporting real-time context identification is also presented. The developed system is evaluated for various realistic scenarios making use of a novel multimodal context open dataset and data on-the-go, demonstrating prominent context-aware capabilities at both low and high levels. The last World Health Organization (WHO) global status report on noncommunicable diseases reveals that illnesses associated with lifestyle choices are currently the leading cause of death worldwide [1]. As a matter of fact, non-communicable diseases are responsible for more than two-thirds of the world's deaths, with more than 40% of them representing premature deaths under the age of 70 years. Recognizing this seriously worrying epidemic scenario, the WHO has defined a clear roadmap to alter the course of the so-called " slow-moving public health disaster ". Most of the policies
References (53)
- WHO. Global Status Report on Noncommunicable Diseases 2014; Technical Report; World Health Organization: Geneva, Switzerland, 2014.
- Burke, L.E.; Swigart, V.; Turk, M.W.; Derro, N.; Ewing, L.J. Experiences of Self-Monitoring: Successes and Struggles During Treatment for Weight Loss. Qual. Health Res. 2009, 19, 815-828.
- Shephard, R.J. Limits to the measurement of habitual physical activity by questionnaires. Br. J. Sports Med. 2003, 37, 197-206.
- Swan, M. Health 2050: The realization of personalized medicine through crowdsourcing, the Quantified Self, and the participatory biocitizen. J. Pers. Med. 2012, 2, 93-118.
- Fitbit Surge. Available online: https://www.fitbit.com/surge (accessed on 30 November 2015).
- Jawbone Up. Available online: https://jawbone.com/up (accessed on 30 November 2015).
- Misfit Shine. Available online: http://www.misfitwearables.com/products/shine (accessed on 30 November 2015).
- Garmin Vivoactive. Available online: http://sites.garmin.com/en-US/vivo/vivofit/ (accessed on 1 April 2016).
- Empatica Embrace. Available online: https://www.empatica.com/product-embrace (accessed on 30 November 2015).
- Oresko, J.; Jin, Z.; Cheng, J.; Huang, S.; Sun, Y.; Duschl, H.; Cheng, A.C. A Wearable Smartphone-Based Platform for Real-Time Cardiovascular Disease Detection Via Electrocardiogram Processing. IEEE Trans. Inf. Technol. Biomed. 2010, 14, 734-740.
- Banos, O.; Villalonga, C.; Damas, M.; Gloesekoetter, P.; Pomares, H.; Rojas, I. PhysioDroid: Combining Wearable Health Sensors and Mobile Devices for a Ubiquitous, Continuous, and Personal Monitoring. Sci. World J. 2014, 2014, 1-11.
- Patel, S.; Mancinelli, C.; Healey, J.; Moy, M.; Bonato, P. Using Wearable Sensors to Monitor Physical Activities of Patients with COPD: A Comparison of Classifier Performance. In Proceedings of the 6th International Workshop on Wearable and Implantable Body Sensor Networks, Berkeley, CA, USA, 3-5 June 2009; pp. 234-239.
- Pierce, W.D.; Cheney, C.D. Behavior Analysis and Learning; Psychology Press: New York, NY, USA; London, UK, 2013.
- Pung, H.; Gu, T.; Xue, W.; Palmes, P.; Zhu, J.; Ng, W.L.; Tang, C.W.; Chung, N.H. Context-aware middleware for pervasive elderly homecare. IEEE J. Sel. Areas Commun. 2009, 27, 510-524.
- Xu, J.; Wang, Y.; Barrett, M.; Dobkin, B.; Pottie, G.; Kaiser, W. Personalized, Multi-Layer Daily Life Profiling through Context Enabled Activity Classification and Motion Reconstruction: An Integrated Systems Approach. IEEE J. Biomed. Health Inform. 2015, 20, 177-188.
- Gaggioli, A.; Pioggia, G.; Tartarisco, G.; Baldus, G.; Corda, D.; Cipresso, P.; Riva, G. A Mobile Data Collection Platform for Mental Health Research. Pers. Ubiquitous Comput. 2013, 17, 241-251.
- Apple HealthKit. Available online: https://developer.apple.com/healthkit/ (accessed on 30 November 2015).
- Google Fit. Available online: https://developers.google.com/fit/overview (accessed on 30 November 2015).
- SAMI, S. Available online: https://developer.samsungsami.io/sami/sami-documentation/ (accessed on 30 November 2015).
- Banos, O.; Amin, M.; Khan, W.; Ali, T.; Afzal, M.; Kang, B.; Lee, S. Mining Minds: An innovative framework for personalized health and wellness support. In Proceedings of the 2015 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), Istanbul, Turkey, 20-23 May 2015; Volume 15.
- Villalonga, C.; Banos, O.; Ali Khan, W.; Ali, T.; Razzaq, M.A.; Lee, S.; Pomares, H.; Rojas, I. High-Level Context Inference for Human Behavior Identification. In International Work-Conference on Ambient Assisted Living an Active Ageing (IWAAL 2015), Lecture Notes in Computer Science; Cleland, I., Guerrero, L., Bravo, J., Eds.; Springer International Publishing: Puerto Varas, Chile, 2015; Volume 9455, pp. 164-175.
- Banos, O.; Amin, M.; Khan, W.; Hussain, M.; Afzal, M.; Kang, B.; Lee, S. The Mining Minds Digital Health and Wellness Framework. BioMed. Eng. OnLine 2016, 15, 165-186.
- Roggen, D.; Troester, G.; Lukowicz, P.; Ferscha, L.; del R. Millan, J.; Chavarriaga, R. Opportunistic Human Activity and Context Recognition. Computer 2013, 46, 36-45.
- Amin, M.B.; Banos, O.; Khan, W.A.; Muhammad Bilal, H.S.; Gong, J.; Bui, D.M.; Cho, S.H.; Hussain, S.; Ali, T.; Akhtar, U.; et al. On Curating Multimodal Sensory Data for Health and Wellness Platforms. Sensors 2016, 16, doi:10.3390/s16070980.
- Banos, O.; Galvez, J.M.; Damas, M.; Pomares, H.; Rojas, I. Window Size Impact in Human Activity Recognition. Sensors 2014, 14, 6474-6499.
- Preece, S.J.; Goulermas, J.Y.; Kenney, L.P.J.; Howard, D. A Comparison of Feature Extraction Methods for the Classification of Dynamic Activities From Accelerometer Data. IEEE Trans. Biomed. Eng. 2009, 56, 871-879.
- Cover, T.M.; Hart, P.E. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 1967, 13, 21-27.
- Banos, O.; Damas, M.; Pomares, H.; Prieto, A.; Rojas, I. Daily living activity recognition based on statistical feature quality group selection. Expert Syst. Appl. 2012, 39, 8013-8021.
- Banos, O.; Damas, M.; Pomares, H.; Rojas, I. On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition. Sensors 2012, 12, 8039-8054.
- Banos, O.; Calatroni, A.; Damas, M.; Pomares, H.; Rojas, I.; Sagha, H.; del R Millan, J.; Troster, G.; Chavarriaga, R.; Roggen, D. Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems across Sensor Modalities. In Proceeding of the 2012 16th International Symposium on Wearable Computers (ISWC), Newcastle, Australia, 18-22 June 2012; pp. 92-99.
- Han, M.; Vinh, L.T.; Lee, Y.K.; Lee, S. Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone. Sensors 2012, 12, 12588-12605.
- Chaaraoui, A.A.; Padilla-López, J.R.; Climent-Pérez, P.; Flórez-Revuelta, F. Evolutionary joint selection to improve human action recognition with RGB-D devices. Expert Syst. Appl. 2014, 41, 786-794.
- Han, J.; Shao, L.; Xu, D.; Shotton, J. Enhanced computer vision with microsoft kinect sensor: A review. IEEE Trans. Cybern. 2013, 43, 1318-1334.
- Soomro, K.; Zamir, A.R. Action recognition in realistic sports videos. In Computer Vision in Sports; Springer: New York, NY, USA, 2014; pp. 181-208.
- Zhang, H.; Parker Lynne, E. CoDe4D: Color-depth local spatio-temporal features for human activity recognition from RGB-D videos. IEEE Trans. Circ. Syst. Video Technol. 2016, 26, 541-555.
- Gupta, S.; Mooney, R.J. Using Closed Captions as Supervision for Video Activity Recognition. Available online: http://www.cs.utexas.edu/ ml/papers/gupta.aaai10.pdf (accessed on 9 August 2016).
- Bang, J.; Lee, S. Call Speech Emotion Recognition for Emotion based Services. J. KIISE Softw. Appl. 2014, 41, 208-213.
- Le, B.V.; Bang, J.; Lee, S. Hierarchical Emotion Classification using Genetic Algorithms. In Proceedings of the Fourth Symposium on Information and Communication Technology, New York, NY, USA, 5 December 2013.
- Cristianini, N.; Shawe-Taylor, J. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods; Cambridge University Press: New York, NY, USA, 2000.
- Naver Maps API. Available online: http://developer.naver.com/wiki/pages/MapAPI (accessed on 3 October 2016).
- Banos, O.; Damas, M.; Pomares, H.; Rojas, F.; Delgado-Marquez, B.; Valenzuela, O. Human activity recognition based on a sensor weighting hierarchical classifier. Soft Comput. 2013, 17, 333-343.
- Banos, O.; Galvez, J.M.; Damas, M.; Guillen, A.; Herrera, L.J.; Pomares, H.; Rojas, I.; Villalonga, C.; Hong, C.; Lee, S. Multiwindow Fusion for Wearable Activity Recognition. In International Work-Conference on Artificial Neural Networks; Rojas, I., Joya, G., Catala, A., Eds.; Springer International Publishing: Palma de Mallorca, Spain, 2015; Volume 9095, pp. 290-297.
- W3C OWL Working Group. OWL 2 Web Ontology Language: Document Overview (Second Edition);
- Carnielli, W.A. Systematization of finite many-valued logics through the method of tableaux. J. Symb. Log. 1987, 52, 473-493.
- Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; Witten, I.H. The WEKA data mining software: an update. ACM SIGKDD Explor. Newsl. 2009, 11, 10-18.
- Apache Jena. Available online: https://jena.apache.org/ (accessed on 3 October 2016).
- Brickley, D.; Guha, R.V. RDF Schema 1.1; W3C Recommendation. Available online: https://www.w3.org/ TR/rdf-schema/ (accessed on 25 February 2014).
- Harris, S.; Seaborne, A. SPARQL 1.1 (SPARQL Query Language for RDF);
- Sirin, E.; Parsia, B.; Grau, B.C.; Kalyanpur, A.; Katz, Y. Pellet: A practical OWL-DL reasoner. J. Web Semant. 2007, 5, 51-53.
- Richardson, L.; Ruby, S. RESTful Web Services; O'Reilly Media, Inc.: Sebastopol, CA, USA, 2008.
- Samsung Galaxy S5. Available online: http://www.samsung.com/us/search/searchMain?Dy=1&Nty=1& Ntt=samsung+galaxy+s5&3Acq_csrf_token=undefined (accessed on 3 October 2016).
- LG G Watch R. Available online: http://www.lg.com/us/smart-watches/lg-W110-g-watch-r (accessed on 8 November 2015).
- Bulling, A.; Blanke, U.; Schiele, B. A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors. ACM Comput. Surv. 2014, 46, 1-33.