Academia.eduAcademia.edu

Figure 5 – uploaded by Rajat sen

See full PDFdownloadDownload figure
Table 2: Win/Draw/Loss statistics among algorithms for the 6 datasets. When the difference in results between two algorithms is not significant according to the statistical significance formula in (Bietti et al., 2018) then it is deemed to be a draw.

Table 2 Win/Draw/Loss statistics among algorithms for the 6 datasets. When the difference in results between two algorithms is not significant according to the statistical significance formula in (Bietti et al., 2018) then it is deemed to be a draw.

Related Figures (13)

Our proposed algorithm for top-k arm selection in general contextual bandits in a non-extreme setting is provided as Algorithm 1. It is a natural extension of the Inverse Gap Weighting (IGW) sampling scheme (Abe and Long, 1999; Foster and Rakhlin, 2020; Simchi-Levi and Xu, 2020). In Section 3.3 we will show that this algorithm with r = 1 has good regret guarantees for the top- k; problem even though the action space is combinatorial, thanks to the linearity of the regret objective in terms of rewards of individual arms in the subset. Note that a naive extension of IGW by treating each action in A* as a separate arm would require a computation of o((#)) per time step and a similar regret scaling. In contrast, Algorithm 1 only requires O(A) computation for the sampling per time step. where a, = argmax,¢ 4 /(z,a), y is a scaling factor. 3.2. IGW for top-/ Contextual Bandits
Our proposed algorithm for top-k arm selection in general contextual bandits in a non-extreme setting is provided as Algorithm 1. It is a natural extension of the Inverse Gap Weighting (IGW) sampling scheme (Abe and Long, 1999; Foster and Rakhlin, 2020; Simchi-Levi and Xu, 2020). In Section 3.3 we will show that this algorithm with r = 1 has good regret guarantees for the top- k; problem even though the action space is combinatorial, thanks to the linearity of the regret objective in terms of rewards of individual arms in the subset. Note that a naive extension of IGW by treating each action in A* as a separate arm would require a computation of o((#)) per time step and a similar regret scaling. In contrast, Algorithm 1 only requires O(A) computation for the sampling per time step. where a, = argmax,¢ 4 /(z,a), y is a scaling factor. 3.2. IGW for top-/ Contextual Bandits
Theorem 1. Algorithm I under Assumptions I and 3, when run with parameters In the next theorem we bound the regret under e€-realizability. In this section we show that our algorithm has favorable regret guarantees. Our regret guar- antees are only derived for the case when Algorithm 1 is run with r = 1. However, we will see that other values of r also work well in practice in Section 5. For ease of exposition we assume F is finite; our results can be extended to infinite function classes with standard techniques (see e.g. (Simchi-Levi and Xu, 2020)). We first present the bounds under exact realizability.! Theorem 2. Algorithm I under Assumptions 2 and 3, when run with parameters
Theorem 1. Algorithm I under Assumptions I and 3, when run with parameters In the next theorem we bound the regret under e€-realizability. In this section we show that our algorithm has favorable regret guarantees. Our regret guar- antees are only derived for the case when Algorithm 1 is run with r = 1. However, we will see that other values of r also work well in practice in Section 5. For ease of exposition we assume F is finite; our results can be extended to infinite function classes with standard techniques (see e.g. (Simchi-Levi and Xu, 2020)). We first present the bounds under exact realizability.! Theorem 2. Algorithm I under Assumptions 2 and 3, when run with parameters
Figure 1: Left: an illustration of the hierarchical decomposition for A, where each gray dot indicates an arm. Middle: the adaptive search for a given context x. The yellow nodes are further explored as they are close to x while the blue nodes are not as they are far from x. The set of effective arms for x consists of the blue nodes and the singleton arms in the yellow leaf nodes. Right: For a fixed x, the corresponding row shows the x-dependent hierarchical arm space decomposition. As x varies, the decomposition also changes. Each blue block stands for a non-singleton effective arm, valid for a contiguous block of contexts. Each gray block contains the singleton effective arms, valid again for a contiguous block of contexts.
Figure 1: Left: an illustration of the hierarchical decomposition for A, where each gray dot indicates an arm. Middle: the adaptive search for a given context x. The yellow nodes are further explored as they are close to x while the blue nodes are not as they are far from x. The set of effective arms for x consists of the blue nodes and the singleton arms in the yellow leaf nodes. Right: For a fixed x, the corresponding row shows the x-dependent hierarchical arm space decomposition. As x varies, the decomposition also changes. Each blue block stands for a non-singleton effective arm, valid for a contiguous block of contexts. Each gray block contains the singleton effective arms, valid again for a contiguous block of contexts.
To ensure the consistency condition (6) in the general case, one requires the expected reward function r(z,a) to be nearly constant over each effective arm a,j; and work with a function class that is constant over each a, ;. The following definition formalizes this. Each internal node e;,; is assumed to be associated with a routing function g;,;(x) mapping ¥ — [0,1] and C),; is used to denote the immediate children of node ep,;. Based on these routing functions and an integer parameter b, we define a beam search in Algorithm 2 for any context c € X as an input. During its execution, this beam search keeps at each level h only the top 6 nodes that return the highest gp,;(z) values. The output of the beam search, denoted also by Az, is the union of a set of nodes denoted as J, and a set of singleton arms denoted as S,. The tree structure ensures that there are at most bm singleton arms in S, and at most (p—1)b(H — 1) nodes n I,. Therefore, |A,| < (p — 1)b(H — 1) + bm = O(log A), implying that A, satisfies (5). Though che cardinality |A,| can vary slightly depending on the context x, in what follows we make the simplifying assumption that |.A,| is equal to a constant Z = O(log A) independent of x and denote Ae = {Get yj 2:05 ez bs
To ensure the consistency condition (6) in the general case, one requires the expected reward function r(z,a) to be nearly constant over each effective arm a,j; and work with a function class that is constant over each a, ;. The following definition formalizes this. Each internal node e;,; is assumed to be associated with a routing function g;,;(x) mapping ¥ — [0,1] and C),; is used to denote the immediate children of node ep,;. Based on these routing functions and an integer parameter b, we define a beam search in Algorithm 2 for any context c € X as an input. During its execution, this beam search keeps at each level h only the top 6 nodes that return the highest gp,;(z) values. The output of the beam search, denoted also by Az, is the union of a set of nodes denoted as J, and a set of singleton arms denoted as S,. The tree structure ensures that there are at most bm singleton arms in S, and at most (p—1)b(H — 1) nodes n I,. Therefore, |A,| < (p — 1)b(H — 1) + bm = O(log A), implying that A, satisfies (5). Though che cardinality |A,| can vary slightly depending on the context x, in what follows we make the simplifying assumption that |.A,| is equal to a constant Z = O(log A) independent of x and denote Ae = {Get yj 2:05 ez bs
Figure 2: Left: A (7, 9, b)-constant predictor function f(a, a) in the 1D motivating example with V C [0,1] and A c [0, 1]. Within each blue block, f(x, a) is constant in a but varies with x. Right: the function f after the reduction.
Figure 2: Left: A (7, 9, b)-constant predictor function f(a, a) in the 1D motivating example with V C [0,1] and A c [0, 1]. Within each blue block, f(x, a) is constant in a but varies with x. Right: the function f after the reduction.
(b) Inference Time per context on amazon-3m
(b) Inference Time per context on amazon-3m
Figure 3: In (a) we compare the different sampling strategies on a realizable setting with & = 50 and r = 25, derived from the eurlex-4k dataset. In (b) we compare the avg. inference times per context vs different beam sizes on the amazon-3m dataset. Note that for this dataset b = 290, 000 will include all arms in the beam in our setting and is order wise equivalent to no hierarchy. This comparison is done for inference in a setting with k = 5,r = 3. Note that for larger datasets in Table 1 our implementation with b = 10, 30 remains efficient for real-time inference as the time-complexity scales only with the beams-size and the height of the tree. We plot the progressive mean rewards collected by each algorithm as a function of time in two of our 6 datasets in (c)-(d) where the algorithms are implemented under our eXtreme reduction framework. In our experiments in (c)-(d) we have k = 5 and r = 3. The beam size is 10 except for IGW-topk (b=400) in (c), which serves as a proxy for Algorithm | without the extreme reduction, as b = 400 includes all the arms in this dataset.
Figure 3: In (a) we compare the different sampling strategies on a realizable setting with & = 50 and r = 25, derived from the eurlex-4k dataset. In (b) we compare the avg. inference times per context vs different beam sizes on the amazon-3m dataset. Note that for this dataset b = 290, 000 will include all arms in the beam in our setting and is order wise equivalent to no hierarchy. This comparison is done for inference in a setting with k = 5,r = 3. Note that for larger datasets in Table 1 our implementation with b = 10, 30 remains efficient for real-time inference as the time-complexity scales only with the beams-size and the height of the tree. We plot the progressive mean rewards collected by each algorithm as a function of time in two of our 6 datasets in (c)-(d) where the algorithms are implemented under our eXtreme reduction framework. In our experiments in (c)-(d) we have k = 5 and r = 3. The beam size is 10 except for IGW-topk (b=400) in (c), which serves as a proxy for Algorithm | without the extreme reduction, as b = 400 includes all the arms in this dataset.
We use 6 XMC datasets for our experiments. Table 1 provides some basic properties of each dataset. We can see that the number of arms in the largest dataset is as large as 2.8MM. The column Initialization Size denotes the size of the held out set used to intialize our algorithms. Note that for the datasets eurlex-4k and wikil0-31k we bootstrap the original training dataset to a larger size by sampling with replacement, as the original number of samples are too small to show noticeable effects. Table 1: Properties of eXtreme Datasets
We use 6 XMC datasets for our experiments. Table 1 provides some basic properties of each dataset. We can see that the number of arms in the largest dataset is as large as 2.8MM. The column Initialization Size denotes the size of the held out set used to intialize our algorithms. Note that for the datasets eurlex-4k and wikil0-31k we bootstrap the original training dataset to a larger size by sampling with replacement, as the original number of samples are too small to show noticeable effects. Table 1: Properties of eXtreme Datasets
Figure 4: We plot the progressive mean rewards collected by each algorithm as a function of time. All algorithms are implemented under our eXtreme reduction framework. The initialization held out set for each dataset is used to train the hierarchy and the routing functions. Then the regressors for all nodes are trained on collected data at the beginning of each epoch. In all our experiments we have k = 5. In Algorithm 3 we set the number of explore slots r = 3. The common legend for all the plots is provided in (d). The beam-size used is b = 10.
Figure 4: We plot the progressive mean rewards collected by each algorithm as a function of time. All algorithms are implemented under our eXtreme reduction framework. The initialization held out set for each dataset is used to train the hierarchy and the routing functions. Then the regressors for all nodes are trained on collected data at the beginning of each epoch. In all our experiments we have k = 5. In Algorithm 3 we set the number of explore slots r = 3. The common legend for all the plots is provided in (d). The beam-size used is b = 10.
Related topics:
MathematicsComputer Science
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved