Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Conclusions
References

On the Application Multi-Armed and Contextual Bandits

Profile image of Djallel BouneffoufDjallel Bouneffouf

2020

Abstract

The multi-armed bandit field is currently experiencing a renaissance, as novel problem settings and algorithms motivated by various practical applications are being introduced, building on top of the classical bandit problem. This article aims to provide a comprehensive review of top recent developments in multiple real-life applications of the multi-armed bandit. Specifically, we introduce a taxonomy of common MAB-based applications and summarize the state-of-the-art for each of those domains. Furthermore, we identify important current trends and provide new perspectives pertaining to the future of this burgeoning field.

Related papers

Multi-facet Contextual Bandits
Yikun Ban

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Contextual multi-armed bandit has shown to be an effective tool in recommender systems. In this paper, we study a novel problem of multi-facet bandits involving a group of bandits, each characterizing the users' needs from one unique aspect. In each round, for the given user, we need to select one arm from each bandit, such that the combination of all arms maximizes the final reward. This problem can find immediate applications in E-commerce, healthcare, etc. To address this problem, we propose a novel algorithm, named MuFasa, which utilizes an assembled neural network to jointly learn the underlying reward functions of multiple bandits. It estimates an Upper Confidence Bound (UCB) linked with the expected reward to balance between exploitation and exploration. Under mild assumptions, we provide the regret analysis of Mu-Fasa. It can achieve the near-optimal O ((+ 1) √) regret bound where is the number of bandits and is the number of played rounds. Furthermore, we conduct extensive experiments to show that MuFasa outperforms strong baselines on real-world data sets.

View PDFchevron_right
Multi-armed bandit problems with dependent arms
Sandeep Pandey

2007

We provide a framework to exploit dependencies among arms in multi-armed bandit problems, when the dependencies are in the form of a generative model on clusters of arms. We find an optimal MDP-based policy for the discounted reward case, and also give an approximation of it with formal error guarantee. We discuss lower bounds on regret in the undiscounted reward scenario, and propose a general two-level bandit policy for it. We propose three different instantiations of our general policy and provide theoretical justifications of how the regret of the instantiated policies depend on the characteristics of the clusters. Finally, we empirically demonstrate the efficacy of our policies on large-scale realworld and synthetic data, and show that they significantly outperform classical policies designed for bandits with independent arms.

View PDFchevron_right
Local Clustering in Contextual Multi-Armed Bandits
Yikun Ban

2021

We study identifying user clusters in contextual multi-armed bandits (MAB). Contextual MAB is an effective tool for many real applications, such as content recommendation and online advertisement. In practice, user dependency plays an essential role in the user’s actions, and thus the rewards. Clustering similar users can improve the quality of reward estimation, which in turn leads to more effective content recommendation and targeted advertising. Different from traditional clustering settings, we cluster users based on the unknown bandit parameters, which will be estimated incrementally. In particular, we define the problem of cluster detection in contextual MAB, and propose a bandit algorithm, LOCB, embedded with local clustering procedure. And, we provide theoretical analysis about LOCB in terms of the correctness and efficiency of clustering and its regret bound. Finally, we evaluate the proposed algorithm from various aspects, which outperforms state-of-the-art baselines.

View PDFchevron_right
Combinatorial Multi-Armed Bandits with Filtered Feedback
Kevin Glazebrook

arXiv (Cornell University), 2017

Motivated by problems in search and detection we present a solution to a Combinatorial Multi-Armed Bandit (CMAB) problem with both heavy-tailed reward distributions and a new class of feedback, filtered semibandit feedback. In a CMAB problem an agent pulls a combination of arms from a set {1, ..., k} in each round, generating random outcomes from probability distributions associated with these arms and receiving an overall reward. Under semibandit feedback it is assumed that the random outcomes generated are all observed. Filtered semibandit feedback allows the outcomes that are observed to be sampled from a second distribution conditioned on the initial random outcomes. This feedback mechanism is valuable as it allows CMAB methods to be applied to sequential search and detection problems where combinatorial actions are made, but the true rewards (number of objects of interest appearing in the round) are not observed, rather a filtered reward (the number of objects the searcher successfully finds, which must by definition be less than the number that appear). We present an upper confidence bound type algorithm, Robust-F-CUCB, and associated regret bound of order O(ln(n)) to balance exploration and exploitation in the face of both filtering of reward and heavy tailed reward distributions.

View PDFchevron_right
Dynamic Clustering of Contextual Multi-Armed Bandits
Trọng Toàn Nguyễn

Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, 2014

With the prevalence of the Web and social media, users increasingly express their preferences online. In learning these preferences, recommender systems need to balance the trade-off between exploitation, by providing users with more of the "same", and exploration, by providing users with something "new" so as to expand the systems' knowledge. Multiarmed bandit (MAB) is a framework to balance this tradeoff. Most of the previous work in MAB either models a single bandit for the whole population, or one bandit for each user. We propose an algorithm to divide the population of users into multiple clusters, and to customize the bandits to each cluster. This clustering is dynamic, i.e., users can switch from one cluster to another, as their preferences change. We evaluate the proposed algorithm on two real-life datasets.

View PDFchevron_right
Designing multi-objective multi-armed bandits algorithms: A study
Madalina Drugan

The 2013 International Joint Conference on Neural Networks (IJCNN), 2013

We propose an algorithmic framework for multiobjective multi-armed bandits with multiple rewards. Different partial order relationships from multi-objective optimization can be considered for a set of reward vectors, such as scalarization functions and Pareto search. A scalarization function transforms the multi-objective environment into a single objective environment and are a popular choice in multi-objective reinforcement learning. Scalarization techniques can be straightforwardly implemented into the current multi-armed bandit framework, but the efficiency of these algorithms depends very much on their type, linear or non-linear (e.g. Chebyshev), and their parameters. Using Pareto dominance order relationship allows to explore the multi-objective environment directly, however this can result in large sets of Pareto optimal solutions. In this paper we propose and evaluate the performance of multi-objective MABs using three regret metric criteria. The standard UCB1 is extended to scalarized multi-objective UCB1 and we propose a Pareto UCB1 algorithm. Both algorithms are proven to have a logarithmic upper bound for their expected regret. We also introduce a variant of the scalarized multi-objective UCB1 that removes online inefficient scalarizations in order to improve the algorithm's efficiency. These algorithms are experimentally compared on multi-objective Bernoulli distributions, Pareto UCB1 being the algorithm with the best empirical performance.

View PDFchevron_right
Maximal Objectives in the Multi-armed Bandit with Applications
EREN ÖZBAY

arXiv (Cornell University), 2020

In several applications of the stochastic multi-armed bandit problem, the traditional objective of maximizing the expected total reward can be inappropriate. In this paper, motivated by certain operational concerns in online platforms, we consider a new objective in the classical setup. Given K arms, instead of maximizing the expected total reward from T pulls (the traditional "sum" objective), we consider the vector of total rewards earned from each of the K arms at the end of T pulls and aim to maximize the expected highest total reward across arms (the "max" objective). For this objective, we show that any policy must incur an instance-dependent asymptotic regret of Ω(log T ) (with a higher instance-dependent constant compared to the traditional objective) and a worst-case regret of Ω(K 1/3 T 2/3 ). We then design an adaptive explorethen-commit policy featuring exploration based on appropriately tuned confidence bounds on the mean reward and an adaptive stopping criterion, which adapts to the problem difficulty and achieves these bounds (up to logarithmic factors). We then generalize our algorithmic insights to the problem of maximizing the expected value of the average total reward of the top m arms with the highest total rewards. Our numerical experiments demonstrate the efficacy of our policies compared to several natural alternatives in practical parameter regimes. We discuss applications of these new objectives to the problem of grooming an adequate supply of value-providing market participants (workers/sellers/service providers) in online platforms.

View PDFchevron_right
Best-of-Both-Worlds Algorithms for Linear Contextual Bandits
Fabio Vitale

arXiv (Cornell University), 2023

We study best-of-both-worlds algorithms for K-armed linear contextual bandits. Our algorithms deliver near-optimal regret bounds in both the adversarial and stochastic regimes, without prior knowledge about the environment. In the stochastic regime, we achieve the polylogarithmic rate (dK) 2 polyln(dKT) ∆min

View PDFchevron_right
Multi-Objective X-Armed Bandits
K. Van Moffaert, Peter Vrancx

Many of the standard optimization algorithms focus on optimizing a single, scalar feedback signal. However, real-life optimization problems often require a simultaneous optimization of more than one objective. In this paper, we propose a multi-objective extension to the standard X -armed bandit problem. As the feedback signal is now vector-valued, the goal of the agent is to sample actions in the Pareto dominating area of the objective space. Therefore, we propose the multi-objective Hierarchical Optimistic Optimization strategy that discretizes the continuous action space in relation to the Pareto optimal solutions obtained in the multi-objective objective space. We experimentally validate the approach on two well-known multi-objective test functions and a simulation of a real life application, the filling phase of a wet clutch. We demonstrate that the strategy allows to identify the Pareto front after just a few epochs and to sample accordingly. After learning, several multi-objective quality indicators indicate that the set of sampled solutions by the algorithm very closely approximates the Pareto front.

View PDFchevron_right
PAC-Bayesian Analysis of Contextual Bandits
François Laviolette, Yevgeny Seldin

We derive an instantaneous (per-round) data-dependent regret bound for stochas-tic multiarmed bandits with side information (also known as contextual bandits). The scaling of our regret bound with the number of states (contexts) N goes as p NI ⇢t (S; A), where I ⇢t (S; A) is the mutual information between states and ac-tions (the side information) used by the algorithm at round t. If the algorithm uses all the side information, the regret bound scales as p N ln K, where K is the number of actions (arms). However, if the side information I ⇢t (S; A) is not fully used, the regret bound is significantly tighter. In the extreme case, when I ⇢t (S; A) = 0, the dependence on the number of states reduces from linear to logarithmic. Our analysis allows to provide the algorithm large amount of side information, let the algorithm to decide which side information is relevant for the task, and penalize the algorithm only for the side information that it is using de facto. We also present an alg...

View PDFchevron_right

References (123)

  1. A. Durand, C. Achilleos, D. Iacovides, K. Strati, G. D. Mitsis, and J. Pineau, "Contextual bandits for adapting treatment in a mouse model of de novo carcinogenesis," in Machine Learning for Healthcare Conference, pp. 67-82, 2018.
  2. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Hybrid- ε-greedy for mobile context-aware recommender system," in Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 468-479, Springer, Berlin, Heidelberg, 2012.
  3. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Follow- ing the user's interests in mobile context-aware recommender systems: The hybrid-e-greedy algorithm," in 2012 26th Inter- national Conference on Advanced Information Networking and Applications Workshops, pp. 657-662, IEEE, 2012.
  4. R. Allesiardo, R. Féraud, and D. Bouneffouf, "A neural net- works committee for the contextual bandit problem," in Interna- tional Conference on Neural Information Processing, pp. 374- 381, Springer, Cham, 2014.
  5. D. Bouneffouf, "Situation-aware approach to improve context- based recommender system," arXiv preprint arXiv:1303.0481, 2013.
  6. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Consid- ering the high level critical situations in con-text-aware recom- mender systems," in 2nd International Workshop on Information Management for Mobile Applications, p. 26, 2012.
  7. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Ex- ploration/exploitation trade-off in mobile context-aware recom- mender systems," in Australasian Joint Conference on Artificial Intelligence, pp. 591-601, Springer, Berlin, Heidelberg, 2012.
  8. D. Bouneffouf, "Applying machine learning techniques to improve user acceptance on ubiquitous environement," arXiv preprint arXiv:1301.4351, 2013.
  9. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganarski, "Risk- aware recommender systems," in International Conference on Neural Information Processing, pp. 57-65, Springer, Berlin, Heidelberg, 2013.
  10. D. Bouneffouf, "Role of temporal inference in the recognition of textual inference," arXiv preprint arXiv:1302.5645, 2013.
  11. D. Bouneffouf, DRARS, a dynamic risk-aware recommender system. PhD thesis, 2013.
  12. D. Bouneffouf, "Improving adaptation of ubiquitous recomman- der systems by using reinforcement learning and collaborative filtering," arXiv preprint arXiv:1303.2308, 2013.
  13. D. Bouneffouf, R. Laroche, T. Urvoy, R. Féraud, and R. Alle- siardo, "Contextual bandit for active learning: Active thompson sampling," in International Conference on Neural Information Processing, pp. 405-412, Springer, Cham, 2014.
  14. D. Bouneffouf, "Towards user profile modelling in recom- mender system," arXiv preprint arXiv:1305.1114, 2013.
  15. D. Bouneffouf, Role de l'inference temporelle dans la recon- naissance de l'inference textuelle. PhD thesis, Université des Sciences et de la Technologie, 2008.
  16. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Contex- tual bandits for context-based information retrieval," in Interna- tional Conference on Neural Information Processing, pp. 35-42, Springer, Berlin, Heidelberg, 2013.
  17. D. Bouneffouf, "The impact of situation clustering in contextual-bandit algorithm for context-aware recommender systems," arXiv preprint arXiv:1304.3845, 2013.
  18. D. Bouneffouf, "Recommandation mobile, sensible au contexte de contenus\'evolutifs: Contextuel-e-greedy," arXiv preprint arXiv:1402.1986, 2014.
  19. D. Bouneffouf and I. Birol, "Sampling with minimum sum of squared similarities for nystrom-based large scale spectral clustering," in Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
  20. D. Bouneffouf, "Hybrid q-learning applied to ubiquitous rec- ommender system," arXiv preprint arXiv:1303.2651, 2013.
  21. D. Bouneffouf, "Mobile recommender systems methods: An overview," arXiv preprint arXiv:1305.1745, 2013.
  22. D. Bouneffouf, "Exponentiated gradient exploration for active learning," Computers, vol. 5, no. 1, p. 1, 2016.
  23. D. Bouneffouf, "Evolution of the user's content: An overview of the state of the art," arXiv preprint arXiv:1305.1787, 2013.
  24. D. Bouneffouf, "" l'apprentissage automatique", une étape importante dans l'adaptation des systèmes d'information à l'utilisateur," 2013.
  25. D. Bouneffouf, "Context-based information retrieval in risky environment," arXiv preprint arXiv:1409.7729, 2014.
  26. D. Bouneffouf, "\'etude des dimensions sp\'ecifiques du con- texte dans un syst\eme de filtrage d'informations," arXiv preprint arXiv:1405.6287, 2014.
  27. D. Bouneffouf, "Freshness-aware thompson sampling," in In- ternational Conference on Neural Information Processing, pp. 373-380, Springer, Cham, 2014.
  28. R. Allesiardo, R. Féraud, and D. Bouneffouf, "Prise de décision contextuelle en bande organisée: Quand les bandits font un brainstorming," 2014.
  29. D. Bouneffouf, "Temporal logic and its applications in natural language processing," 2011.
  30. D. Bouneffouf, "R-ucb: a contextual bandit algorithm for risk- aware recommender systems," arXiv preprint arXiv:1408.2195, 2014.
  31. D. Bouneffouf, "Proposition d'une technique de gestion de projet dans les startups," arXiv preprint arXiv:1303.2317, 2013.
  32. D. Bouneffouf, "Learning and inference engine applied to ubiquitous recommender system,"
  33. D. Bouneffouf, "La logique temporelle et ses applications dans le traitement du langage naturel," 2013.
  34. D. Bouneffouf, "Ant clustering to improve situation-aware rec- ommender systems," 2014.
  35. D. Bouneffouf, "Contextual bandit algorithm for risk-aware recommender systems," in 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 4667-4674, IEEE, 2016.
  36. D. Bouneffouf and R. Féraud, "Multi-armed bandit problem with known trend," Neurocomputing, vol. 205, pp. 16-21, 2016.
  37. D. Bouneffouf, "Exponentiated gradient linucb for contextual multi-armed bandits," arXiv preprint arXiv:1305.2415, 2013.
  38. D. Bouneffouf and I. Birol, "Theoretical analysis of the min- imum sum of squared similarities sampling for nyström-based spectral clustering," in 2016 International Joint Conference on Neural Networks (IJCNN), pp. 3856-3862, IEEE, 2016.
  39. D. Bouneffouf, "Temporal logic in natural language processing," 2013.
  40. D. Bouneffouf, "Optimizing an utility function for explo- ration/exploitation trade-off in context-aware recommender sys- tem," arXiv preprint arXiv:1303.0485, 2013.
  41. D. Bouneffouf and I. Birol, "Ensemble minimum sum of squared similarities sampling for nyström-based spectral clus- tering," in 2016 International Joint Conference on Neural Net- works (IJCNN), pp. 3851-3855, IEEE, 2016.
  42. D. Bouneffouf, I. Rish, G. A. Cecchi, and R. Féraud, "Context attentive bandits: contextual bandit with restricted context," in IJCAI 2017, 2017.
  43. D. Bouneffouf, I. Rish, and G. A. Cecchi, "Bandit models of human behavior: Reward processing in mental disorders," in International Conference on Artificial General Intelligence, pp. 237-248, Springer, Cham, 2017.
  44. D. Bouneffouf, "Drars: un système de recommandation dy- namique sensible au risque,"
  45. B. Lin, G. Cecchi, D. Bouneffouf, and I. Rish, "Adaptive representation selection in contextual bandit with unlabeled history," 2018.
  46. D. Bouneffouf, "Nystrom sampling depends on the eigenspec- trum shape of the data," 2018.
  47. A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi, "Using contextual bandits with behavioral constraints for constrained online movie recommendation.," in IJCAI, pp. 5802-5804, 2018.
  48. M. Riemer, T. Klinger, D. Bouneffouf, and M. Franceschini, "Scalable recollections for continual lifelong learning," in Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1352-1359, 2019.
  49. A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi, "Incorporating behavioral constraints in online ai systems," in AAAI, 2019.
  50. D. Bouneffouf, "Eigenspectrum shape based nyström sampling," in 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1-6, IEEE, 2018.
  51. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "Follow- ing the user's interests in mobile context-aware,"
  52. M. Riemer, M. Franceschini, D. Bouneffouf, and T. Klinger, "Generative knowledge distillation for general purpose function compression," in NIPS 2017 Workshop on Teaching Machines, Robots, and Humans, vol. 5, p. 30, 2017.
  53. B. Lin, D. Bouneffouf, G. A. Cecchi, and I. Rish, "Contex- tual bandit with adaptive feature extraction," in 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 937-944, IEEE, 2018.
  54. A. Choromanska, B. Cowen, S. Kumaravel, R. Luss, M. Rigotti, I. Rish, P. Diachille, V. Gurev, B. Kingsbury, R. Tejwani, et al., "Beyond backprop: Online alternating minimization with auxiliary variables," in International Conference on Machine Learning, pp. 1193-1202, 2019.
  55. D. Bouneffouf and I. Rish, "A survey on practical applications of multi-armed and contextual bandits," in The IEEE World Congress on Computational Intelligence (IEEE WCCI), 2020.
  56. B. Djallel, A. Bouzeghoub, and A. L. Ganarski, "Risk-aware recommender systems,"
  57. S. Upadhyay, M. Agarwal, D. Bounneffouf, and Y. Khazaeni, "A bandit approach to posterior dialog orchestration under a budget," NIPS 2018.
  58. S. Liu, P. Ram, D. Bouneffouf, G. Bramble, A. R. Conn, H. Samulowitz, and A. G. Gray, "Automated machine learning via admm," CoRR, abs/1905.00424, 2019.
  59. D. Bouneffouf, S. Parthasarathy, H. Samulowitz, and M. Wistub, "Optimal exploitation of clustering and history information in multi-armed bandit," arXiv preprint arXiv:1906.03979, 2019.
  60. R. Noothigattu, D. Bouneffouf, N. Mattei, R. Chandra, P. Madan, K. Varshney, M. Campbell, M. Singh, and F. Rossi, "Interpretable multi-objective reinforcement learning through policy orchestration," arXiv preprint arXiv:1809.08343, 2018.
  61. M. Yurochkin, S. Upadhyay, D. Bouneffouf, M. Agarwal, and Y. Khazaeni, "Online semi-supervised learning with bandit feedback," 2019.
  62. B. Lin, G. Cecchi, D. Bouneffouf, J. Reinen, and I. Rish, "Re- inforcement learning models of human behavior: Reward pro- cessing in mental disorders," arXiv preprint arXiv:1906.11286, 2019.
  63. R. Noothigattu, D. Bouneffouf, N. Mattei, R. Chandra, P. Madan, K. R. Varshney, M. Campbell, M. Singh, and F. Rossi, "Teaching ai agents ethical values using reinforcement learning and policy orchestration," IBM Journal of Research and Devel- opment, vol. 63, no. 4/5, pp. 2-1, 2019.
  64. C. Aggarwal, D. Bouneffouf, H. Samulowitz, B. Buesser, T. Hoang, U. Khurana, S. Liu, T. Pedapati, P. Ram, A. Rawat, et al., "How can ai automate end-to-end data science?," arXiv preprint arXiv:1910.14436, 2019.
  65. A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi, "Using multi-armed bandits to learn ethical priorities for online ai systems," IBM Journal of Research and Development, vol. 63, no. 4/5, pp. 1-1, 2019.
  66. S. Mehta, F. Rossi, K. Varshney, A. Balakrishnan, D. Boun- effouf, N. Mattei, R. Noothigattu, R. Chandra, P. Madan, M. Campbell, et al., "Ai ethics," 2019.
  67. S. Liu, P. Ram, D. Vijaykeerthy, D. Bouneffouf, G. Bramble, H. Samulowitz, D. Wang, A. Conn, and A. G. Gray, "An admm based framework for automl pipeline configuration.," in AAAI, pp. 4892-4899, 2020.
  68. S. Sharma, Y. Zhang, J. M. Ríos Aliaga, D. Bouneffouf, V. Muthusamy, and K. R. Varshney, "Data augmentation for discrimination prevention and bias disambiguation," in Proceed- ings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 358-364, 2020.
  69. A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi, "Constrained decision-making and explanation of a recommen- dation," Jan. 16 2020. US Patent App. 16/050,176.
  70. B. Lin, G. A. Cecchi, D. Bouneffouf, J. Reinen, and I. Rish, "A story of two streams: Reinforcement learning models from human behavior and neuropsychiatry.," in AAMAS, pp. 744-752, 2020.
  71. K. Varshney, M. Campbell, M. Singh, and F. Rossi, "Teaching ai agents ethical values using reinforcement learning and policy orchestration,"
  72. D. Bouneffouf and E. Claeys, "Hyper-parameter tuning for the contextual bandit," arXiv preprint arXiv:2005.02209, 2020.
  73. B. Lin, G. Cecchi, D. Bouneffouf, J. Reinen, and I. Rish, "Uni- fied models of human behavioral agents in bandits, contextual bandits and rl," arXiv preprint arXiv:2005.04544, 2020.
  74. D. Bouneffouf and E. Claeys, "Learning exploration for con- textual bandit," in AutoML@ ICML 2019: 6th ICML Workshop on Automated Machine Learning, 2016.
  75. B. Lin, D. Bouneffouf, and G. Cecchi, "Online learning in iterated prisoner's dilemma to mimic human behavior," arXiv preprint arXiv:2006.06580, 2020.
  76. P. Ram, S. Liu, D. Vijaykeerthi, D. Wang, D. Bounef- fouf, G. Bramble, H. Samulowitz, and A. G. Gray, "Solv- ing constrained cash problems with admm," arXiv preprint arXiv:2006.09635, 2020.
  77. D. Bouneffouf, "Online learning with corrupted context: Cor- rupted contextual bandits," arXiv preprint arXiv:2006.15194, 2020.
  78. D. Bouneffouf, S. Upadhyay, and Y. Khazaeni, "Contextual bandit with missing rewards," arXiv preprint arXiv:2007.06368, 2020.
  79. D. Bouneffouf, "Location-aware approach to improve context- based recommender system," arXiv preprint arXiv:1303.0481, 2013.
  80. K. Toutanova and H. Wu, "Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers)," in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2014.
  81. C. S. Leung, Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part III. Springer, 2012.
  82. A. T. B. Jin, Neural Information Processing: 21st International Conference, ICONIP 2014, Kuching, Malaysia, November 3-6, 2014: Proceedings. Springer, 2014.
  83. D. Bouneffouf, "Computing the dirichlet-multinomial log- likelihood function," arXiv preprint arXiv:2007.11967, 2020.
  84. D. Bouneffouf, "Spectral clustering using eigenspectrum shape based nystrom sampling," arXiv preprint arXiv:2007.11416, 2020.
  85. D. Bouneffouf, S. Upadhyay, and Y. Khazaeni, "Online learning from less data: Contextual bandit with missing rewards,"
  86. A. Gupta, Y. Ong, B. Da, L. Feng, and S. Handoko, "2016 ieee congress on evolutionary computation (cec)," 2016.
  87. D. Bouneffouf, A. Bouzeghoub, and A. L. Ganc ¸arski, "contextual-bandit algorithm for context-aware recommender system," in International conference on neural information processing, pp. 324-331, Springer, Berlin, Heidelberg, 2012.
  88. D. Bouneffouf, I. Rish, and G. A. Cecchi, "Bandit models of human behavior,"
  89. D. Bouneffouf, C. Aggarwal, H. Samulowitz, B. Buesser, T. Hoang, U. Khurana, S. Liu, T. Pedapati, P. Ram, A. Rawat, et al., "Survey on automated end-to-end data science?,"
  90. D. Bouneffouf, "Spectral clustering using eigenspectrum shape based sampling,"
  91. D. Bouneffouf and E. Claeys, "Online hyperparameter tuning for contextual bandits," 2018.
  92. D. Bouneffouf, "Toward computing the dirichlet-multinomial log-likelihood function,"
  93. K. Ding, J. Li, and H. Liu, "Interactive anomaly detection on attributed networks," in Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM '19, (New York, NY, USA), pp. 357-365, ACM, 2019.
  94. T. L. Lai and H. Robbins, "Asymptotically efficient adaptive allocation rules," Advances in Applied Mathematics, vol. 6, no. 1, pp. 4-22, 1985.
  95. P. Auer, N. Cesa-Bianchi, and P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Machine Learning, vol. 47, no. 2-3, pp. 235-256, 2002.
  96. D. Bouneffouf and R. Féraud, "Multi-armed bandit problem with known trend," Neurocomputing, vol. 205, pp. 16-21, 2016.
  97. S. Agrawal and N. Goyal, "Analysis of thompson sampling for the multi-armed bandit problem," in COLT 2012 -The 25th Annual Conference on Learning Theory, June 25-27, 2012, Edinburgh, Scotland, pp. 39.1-39.26, 2012.
  98. J. Langford and T. Zhang, "The epoch-greedy algorithm for multi-armed bandits with side information," in Advances in neural information processing systems, pp. 817-824, 2008.
  99. S. Agrawal and N. Goyal, "Thompson sampling for contextual bandits with linear payoffs," in ICML (3), pp. 127-135, 2013.
  100. L. Li, W. Chu, J. Langford, and R. E. Schapire, "A contextual- bandit approach to personalized news article recommendation," CoRR, 2010.
  101. R. Allesiardo, R. Féraud, and D. Bouneffouf, "A neural net- works committee for the contextual bandit problem," in Neu- ral Information Processing -21st International Conference, ICONIP 2014, Kuching, Malaysia, November 3-6, 2014. Pro- ceedings, Part I, pp. 374-381, 2014.
  102. H. Bastani and M. Bayati, "Online decision-making with high- dimensional covariates," Available at SSRN 2661896, 2015.
  103. W. Shen, J. Wang, Y.-G. Jiang, and H. Zha, "Portfolio choices with orthogonal bandit learning," in Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
  104. X. Huo and F. Fu, "Risk-aware multi-armed bandit problem with application to portfolio selection," Royal Society open science, vol. 4, no. 11, p. 171377, 2017.
  105. K. Misra, E. M. Schwartz, and J. Abernethy, "Dynamic online pricing with incomplete information using multi-armed bandit experiments," 2018.
  106. J. Mueller, V. Syrgkanis, and M. Taddy, "Low-rank bandit methods for high-dimensional dynamic pricing," arXiv preprint arXiv:1801.10242, 2018.
  107. Q. Zhou, X. Zhang, J. Xu, and B. Liang, "Large-scale bandit approaches for recommender systems," in International Confer- ence on Neural Information Processing, pp. 811-821, Springer, 2017.
  108. S. Vaswani, B. Kveton, Z. Wen, M. Ghavamzadeh, L. V. Lak- shmanan, and M. Schmidt, "Model-independent online learning for influence maximization," in Proceedings of the 34th Interna- tional Conference on Machine Learning-Volume 70, pp. 3530- 3539, JMLR. org, 2017.
  109. Z. Wen, B. Kveton, M. Valko, and S. Vaswani, "Online influence maximization under independent cascade model with semi- bandit feedback," in Advances in neural information processing systems, pp. 3022-3032, 2017.
  110. D. E. Losada, J. Parapar, and A. Barreiro, "Multi-armed ban- dits for adjudicating documents in pooling-based evaluation of information retrieval systems," Information Processing & Management, vol. 53, no. 5, pp. 1005-1025, 2017.
  111. B. Liu, T. Yu, I. Lane, and O. J. Mengshoel, "Customized nonlinear bandits for online response selection in neural con- versation models," in AAAI, 2018, pp. 5245-5252, 2018.
  112. T. Silander et al., "Contextual memory bandit for pro-active dialog engagement," 2018.
  113. S. Upadhyay, M. Agarwal, D. Bounneffouf, and Y. Khazaeni, "A bandit approach to posterior dialog orchestration under a budget," 2018.
  114. D. J. Soemers, T. Brys, K. Driessens, M. H. Winands, and A. Nowé, "Adapting to concept drift in credit card transaction data streams using contextual bandits and decision trees," in AAAI, 2018.
  115. S. Boldrini, L. De Nardis, G. Caso, M. Le, J. Fiorina, and M.- G. Di Benedetto, "mumab: A multi-armed bandit model for wireless network selection," Algorithms, vol. 11, no. 2, p. 13, 2018.
  116. R. Kerkouche, R. Alami, R. Féraud, N. Varsier, and P. Maillé, "Node-based optimization of lora transmissions with multi- armed bandit algorithms," in ICT 2018, Saint Malo, France, June 26-28, 2018, pp. 521-526, 2018.
  117. M. Gagliolo and J. Schmidhuber, "Algorithm selection as a bandit problem with unbounded losses," in Learning and In- telligent Optimization, 4th International Conference, LION 4, Venice, Italy, January 18-22, 2010. Selected Papers, pp. 82-96, 2010.
  118. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Tal- walkar, "Hyperband: A novel bandit-based approach to hyperpa- rameter optimization," arXiv preprint arXiv:1603.06560, 2016.
  119. J. Wang, P. Zhao, S. C. Hoi, and R. Jin, "Online feature selection and its applications," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 3, pp. 698-710, 2014.
  120. D. Bouneffouf, I. Rish, G. A. Cecchi, and R. Féraud, "Context attentive bandits: Contextual bandit with restricted context," in IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 1468-1475, 2017.
  121. R. Ganti and A. G. Gray, "Building bridges: Viewing active learning from the multi-armed bandit lens," arXiv preprint arXiv:1309.6830, 2013.
  122. J. Sublime and S. Lefebvre, "Collaborative clustering through constrained networks using bandit optimization," in 2018 Inter- national Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13, 2018, pp. 1-8, 2018.
  123. R. Laroche and R. Féraud, "Algorithm selection of off-policy reinforcement learning algorithm," CoRR, vol. abs/1701.08810, 2017.

Related papers

Learning techniques in multi-armed bandits
Nikhil Gupta

Multi-armed bandit problem is a classic example of the exploration vs. exploitation dilemma in which a collection of one-armed bandits, each with unknown but fixed reward probability, is given. The key idea is to develop a strategy, which results in the arm with the highest reward probability to be played such that the total reward obtained is maximized. Although seemingly a simplistic problem, solution strategies are important because of their wide applicability in a myriad of areas such as adaptive routing, resource allocation, clinical trials, and more recently in the area of online recommendation of news articles, advertisements, coupons, etc. to name a few. In this dissertation, we present different types of Bayesian Inference based bandit algorithms for Two and Multiple Armed Bandits which use Order Statistics to select the next arm to play. The Bayesian strategies, also known in literature as "Thompson Method" are shown to function well for a whole range of values, including very small values, outperforming UCB and other commonly used strategies. Empirical analysis results show a significant improvement in both synthetic and real

View PDFchevron_right
The multi-armed bandit, with constraints
Eugene Feinberg

Annals of Operations Research, 2012

The early sections of this paper present an analysis of a Markov decision model that is known as the multi-armed bandit under the assumption that the utility function of the decision maker is either linear or exponential. The analysis includes efficient procedures for computing the expected utility associated with the use of a priority policy and for identifying a priority policy that is optimal. The methodology in these sections is novel, building on the use of elementary row operations. In the later sections of this paper, the analysis is adapted to accommodate constraints that link the bandits. It was demonstrated in [12, 10] that, given each multi-state, it is optimal to play any Markov chain (bandit) whose current state has the largest index (lowest label). Following [12, 10], the multi-armed bandit problem has stimulated research in control theory, economics, probability, and operations research. A sampling of noteworthy papers includes Bergemann and Välimäkim [2], Bertsimas and Niño-Mora [4], El Karoui and Karatzas [8], Katehakis and Veinott [15], Schlag [17], Sonin [18], Tsisiklis [19], Variaya, Walrand and Buyukkoc [20], Weber [22], and Whittle [24],. Books on the subject (that list many references) include Berry and Fristedt [3], Gittins [11], Gittins, Glazebrook and Weber [13]. The last and most recent of these books provides a status report on the multi-armed bandit that is almost up-to-date. An implication of the analysis in [12, 10] is that the largest of all of the indices equals the maximum over all states of the ratio r(i)/(1 − c), where r(i) denotes the expectation of the reward that is earned if state i's bandit is played once while state i is observed and where c is the discount factor. In 1994, Tsitsiklis [19]

View PDFchevron_right
Evaluation of performance: multi-armed bandit vs. contextual bandit
Ranojoy Chatterjee

Master of ScienceDepartment of Computer ScienceWilliam H. HsuThis work compares two methods, the multi-armed bandit (MAB) and contextual multi-armed bandit (CMAB), for action recommendation in a sequential decision making domain. It empirically evaluates their effectiveness on a customer relationship management task. The goal of this project is to experiment using [epsilon]-greedy and random selection strategies to characterize the exploration vs. exploitation tradeoff , which manifests when trying to increase or maximize profit while gaining new information regarding the process. The first method under observation, the multi-armed bandit (MAB), is simpler to compute and scales better to larger amounts of data; it has a wide range of applicability, including website optimization, clinical trials, adaptive routing, and stock trading. The contextual multi-armed bandit (CMAB) is an advanced version of the multi-armed bandit which takes into consideration the user’s past usage patterns,...

View PDFchevron_right
Multi-armed bandits with simple arms
Robert W Keener

Advances in Applied Mathematics, 1986

An exact solution to certain multi-armed bandit problems with independent and simple arms is presented. An arm is simple if the observations associated with the arm have one of two distributions conditional on the value of an unknown dichotomous parameter. This solution is obtained relating Gittins indices for the arms to ladder variables for associated random walks. 0 1986 Academic PRSS. IX

View PDFchevron_right
Linear Programming for Finite State Multi-Armed Bandit Problems
Michael N Katehakis

Mathematics of Operations Research, 1986

... Soc. Ser. B. 42 143-149. .(1982). Optimization over Time. Vol. 1. John Wiley, New York. Varaiya, P., Walrand, J. and Buyukkoc, C. (1984). Extensions of the Multi-Armed Bandit Problem. Electronic Research Laboratory, University of California, Berkeley, Technical Report, 41 pp. ...

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved