Eliciting pairwise preferences in recommender systems
2018, Proceedings of the 12th ACM Conference on Recommender Systems
https://doi.org/10.1145/3240323.3240364Abstract
Preference data in the form of ratings or likes for items are widely used in many Recommender Systems. However, previous research has shown that even item comparisons, which generate pairwise preference data, can be used to model user preferences. Moreover, pairwise preferences can be e ectively combined with ratings to compute recommendations. In such hybrid approaches, the Recommender System requires to elicit both types of preference data from the user. In this work, we aim at identifying how and when to elicit pairwise preferences, i.e., when this form of user preference data is more meaningful for the user to express and more bene cial for the system. We conducted an online A/B test and compared a rating-only based system variant with another variant that allows the user to enter both types of preferences. Our results demonstrate that pairwise preferences are valuable and useful, especially when the user is focusing on a speci c type of items. By incorporating pairwise preferences, the system can generate be er recommendations than a state of the art rating-only based solution. Additionally, our results indicate that there seems to be a dependency between the user's personality, the perceived system usability and the satisfaction for the preference elicitation procedure, which varies if only ratings or a combination of ratings and pairwise preferences are elicited.
References (41)
- Dan Ariely. 2009. Predictably irrational. HarperCollins New York.
- Linas Baltrunas, Bernd Ludwig, Stefan Peer, and Francesco Ricci. 2012. Context relevance assessment and exploitation in mobile recommender systems. Personal and Ubiquitous Computing 16, 5 (2012), 507-526.
- Aaron Bangor, Philip T Kortum, and James T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human-Computer Interaction 24, 6 (2008), 574-594.
- Henry Blanco and Francesco Ricci. 2013. Acquiring user pro les from implicit feedback in a conversational recommender system. In Proceedings of the 7th ACM conference on Recommender systems. ACM, 307-310.
- Laura Blédaité and Francesco Ricci. 2015. Pairwise preferences elicitation and exploitation for conversational collaborative ltering. In Proceedings of the 26th ACM Conference on Hypertext & Social Media. ACM, 231-236.
- Ma hias Braunhofer, Mehdi Elahi, and Francesco Ricci. 2014. STS: A Context- Aware Mobile Recommender System for Places of Interest.. In UMAP Workshops.
- Ma hias Braunhofer, Mehdi Elahi, and Francesco Ricci. 2014. Usability assess- ment of a context-aware and personality-based mobile recommender system. In International conference on electronic commerce and web technologies. Springer, 77-88.
- Ma hias Braunhofer, Mehdi Elahi, and Francesco Ricci. 2015. User personality and the new user problem in a context-aware point of interest recommender system. In Information and Communication Technologies in Tourism 2015. Springer, 537-549.
- Ma hias Braunhofer, Mehdi Elahi, Francesco Ricci, and omas Schievenin. 2013. Context-aware points of interest suggestion with dynamic weather data management. In Information and communication technologies in tourism 2014. Springer, 87-100.
- Derek Bridge and Francesco Ricci. 2007. Supporting product selection with query editing recommendations. In Proceedings of the 2007 ACM conference on Recommender systems. ACM, 65-72.
- Armelle Brun, Ahmad Hamad, Olivier Bu et, and Anne Boyer. 2010. Towards preference relations in recommender systems. In Workshop on Preference Learning, European Conference on Machine Learning and Principle and Practice of Knowledge Discovery in Databases (ECML-PKDD 2010). Citeseer.
- Ben Cartere e, Paul N Benne , David Maxwell Chickering, and Susan T Du- mais. 2008. Here or there: Preference judgments for relevance. In Advances in Information Retrieval 2008. Springer, 16-27.
- Maunendra Sankar Desarkar and Sudeshna Sarkar. 2012. Rating prediction using preference relations based matrix factorization.. In UMAP Workshops. Citeseer.
- Maunendra Sankar Desarkar, Roopam Saxena, and Sudeshna Sarkar. 2012. Pref- erence relation based matrix factorization for recommender systems. In Inter- national conference on user modeling, adaptation, and personalization. Springer, 63-75.
- Mehdi Elahi, Francesco Ricci, and Neil Rubens. 2016. A survey of active learning in collaborative ltering recommender systems. Computer Science Review 20 (2016), 29-50.
- David F Gleich and Lek-heng Lim. 2011. Rank aggregation via nuclear norm minimization. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 60-68.
- Samuel D Gosling, Peter J Rentfrow, and William B Swann. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in personality 37, 6 (2003), 504-528.
- Mark P Graus and Martijn C Willemsen. 2015. Improving the User Experience during Cold Start through Choice-Based Preference Elicitation. In Proceedings of the 9th ACM Conference on Recommender Systems. ACM, 273-276.
- Sandra Heldsinger and Stephen Humphry. 2010. Using the method of pairwise comparison to obtain reliable teacher assessments. e Australian Educational Researcher 37, 2 (2010), 1-19.
- Rong Hu and Pearl Pu. 2009. A comparative user study on rating vs. personality quiz based preference elicitation methods. In Proceedings of the 14th international conference on Intelligent user interfaces. ACM, 367-372.
- Ian Jones and Lara Alcock. 2014. Peer assessment without assessment criteria. Studies in Higher Education 39, 10 (2014), 1774-1787.
- Nicolas Jones, Armelle Brun, and Anne Boyer. 2011. Comparisons instead of ratings: Towards more stable preferences. In Web Intelligence and Intelligent Agent Technology (WI-IAT), 2011 IEEE/WIC/ACM International Conference on, Vol. 1. IEEE, 451-456.
- Saikishore Kalloori. 2017. Pairwise Preferences and Recommender Systems. In Proceedings of the 22nd International Conference on Intelligent User Interfaces Companion. ACM, 169-172.
- Saikishore Kalloori and Francesco Ricci. 2017. Improving Cold Start Recommenda- tion by Mapping Feature-Based Preferences to Item Comparisons. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. ACM, 289-293.
- Saikishore Kalloori, Francesco Ricci, and Marko Tkalcic. 2016. Pairwise prefer- ences based matrix factorization and nearest neighbor recommendation tech- niques. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 143-146.
- Bart P Knijnenburg, Martijn C Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 441-504.
- Michal Kosinski, Yoram Bachrach, Pushmeet Kohli, David Stillwell, and ore Graepel. 2014. Manifestations of user personality in website choice and behaviour on online social networks. Machine learning 95, 3 (2014), 357-380.
- uy Ngoc Nguyen and Francesco Ricci. 2017. A Chat-Based Group Recom- mender System for Tourism. In Information and Communication Technologies in Tourism 2017. Springer, 17-30.
- Tien T Nguyen, Daniel Kluver, Ting-Yu Wang, Pik-Mai Hui, Michael D Ekstrand, Martijn C Willemsen, and John Riedl. 2013. Rating support interfaces to improve user experience and recommender accuracy. In Proceedings of the 7th ACM conference on Recommender systems. ACM, 149-156.
- Syavash Nobarany, Louise Oram, Vasanth Kumar Rajendran, Chi-Hsiang Chen, Joanna McGrenere, and Tamara Munzner. 2012. e design space of opinion measurement interfaces: exploring recall support for rating and ranking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2035-2044.
- Francesco Ricci, Lior Rokach, and Bracha Shapira. 2015. Recommender Systems: Introduction and Challenges. In Recommender Systems Handbook. Springer.
- Silvia Rossi, Francesco Barile, Sergio Di Martino, and Davide Improta. 2017. A comparison of two preference elicitation approaches for museum recommenda- tions. Concurrency and Computation: Practice and Experience 29, 11 (2017).
- Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan. 2015. Active learning in recommender systems. In Recommender systems handbook. Springer, 809-846.
- omas L Saaty. 2008. Decision making with the analytic hierarchy process. International journal of services sciences 1, 1 (2008), 83-98.
- Je Sauro. 2011. Measuring usability with the system usability scale (SUS), h p://www.measuringusability.com/sus.php. (2011).
- Je Sauro and James R Lewis. 2016. antifying the user experience: Practical statistics for user research. Morgan Kaufmann.
- Michael Scholz. 2008. From consumer preferences towards buying decisions. 21st Bled eConference eCollaboration: Overcoming Boundaries rough Multi-Channel Interaction (2008), 223-235.
- Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008. Intro- duction to information retrieval. Vol. 39. Cambridge University Press.
- Loren Terveen Shuo Chang, F. Maxwell Harper. 2015. Using Groups of Items for Preference Elicitation in Recommender Systems. In CSCW '15 Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing (2015-03-11). ACM, 1258-1269. h p://dl.acm.org/citation.cfm?id= 2675210
- Paul Slovic and Sarah Lichtenstein. 1983. Preference reversals: A broader per- spective. e American Economic Review 73, 4 (1983), 596-605.
- Joshua Wilt and William Revelle. 2009. Extraversion. (2009).