The software world is always atwitter with predictions on the next big piece of technology. And a lot of chatter focuses on what venture capitalists express interest in. As an investor, how do you pick a good company to invest in? Do you notice quirky names like “Kaggle” and “Meebo,” require deep technical abilities, or value a charismatic sales pitch?

This author personally believes we’re not thinking as big as we should be when it comes to innovation in software engineering and computer science, and that as a society we should value big pushes forward much more than we do. But making safe investments is almost always at odds with innovation. And so every venture capitalist faces the following question. When do you focus investment in those companies that have proven to succeed, and when do you explore new options for growth? A successful venture capitalist must strike a fine balance between this kind of exploration and exploitation. Explore too much and you won’t make enough profit to sustain yourself. Narrow your view too much and you will miss out on opportunities whose return surpasses any of your current prospects.

In life and in business there is no correct answer on what to do, partly because we just don’t have a good understanding of how the world works (or markets, or people, or the weather). In mathematics, however, we can meticulously craft settings that have solid answers. In this post we’ll describe one such scenario, the so-called *multi-armed bandit *problem, and a simple algorithm called UCB1 which performs close to optimally. Then, in a future post, we’ll analyze the algorithm on some real world data.

As usual, all of the code used in the making of this post are available for download on this blog’s Github page.

## Multi-Armed Bandits

The multi-armed bandit scenario is simple to describe, and it boils the exploration-exploitation tradeoff down to its purest form.

Suppose you have a set of actions labeled by the integers . We call these *actions* in the abstract, but in our minds they’re slot machines. We can then play a game where, in each round, we choose an action (a slot machine to play), and we observe the resulting payout. Over many rounds, we might explore the machines by trying some at random. Assuming the machines are not identical, we naturally play machines that seem to pay off well more frequently to try to maximize our total winnings.

This is the most general description of the game we could possibly give, and every bandit learning problem has these two components: actions and rewards. But in order to get to a concrete problem that we can reason about, we need to specify more details. Bandit learning is a large tree of variations and this is the point at which the field ramifies. We presently care about two of the main branches.

**How are the rewards produced? **There are many ways that the rewards could work. One nice option is to have the rewards for action be drawn from a fixed distribution (a different reward distribution for each action), and have the draws be independent across rounds and across actions. This is called the *stochastic *setting and it’s what we’ll use in this post. Just to pique the reader’s interest, here’s the alternative: instead of having the rewards be chosen randomly, have them be *adversarial.* That is, imagine a casino owner *knows your algorithm* and your internal beliefs about which machines are best at any given time. He then fixes the payoffs of the slot machines in advance of each round to screw you up! This sounds dismal, because the casino owner could just make all the machines pay nothing every round. But actually we can design good algorithms for this case, but “good” will mean something different than absolute winnings. And so we must ask:

**How do we measure success? **In both the stochastic and the adversarial setting, we’re going to have a hard time coming up with any theorems about the performance of an algorithm if we care about how much absolute reward is produced. There’s nothing to stop the distributions from having terrible expected payouts, and nothing to stop the casino owner from intentionally giving us no payout. Indeed, the problem lies in our measurement of success. A better measurement, which we can apply to both the stochastic and adversarial settings, is the notion of *regret*. We’ll give the definition for the stochastic case, and investigate the adversarial case in a future post.

**Definition: **Given a player algorithm and a set of actions , the *cumulative regret *of in rounds is the difference between the expected reward of the best action (the action with the highest expected payout) and the expected reward of for the first rounds.

We’ll add some more notation shortly to rephrase this definition in symbols, but the idea is clear: we’re competing against the best action. Had we known it ahead of time, we would have just played it every single round. Our notion of success is not in how well we do absolutely, but in how well we do *relative to what is feasible*.

## Notation

Let’s go ahead and draw up some notation. As before the actions are labeled by integers . The *reward* of action is a -valued random variable distributed according to an unknown distribution and possessing an unknown expected value . The game progresses in rounds so that in each round we have different random variables for the reward of action in round (in particular, and are identically distributed). The are independent as both and vary, although when varies the distribution changes.

So if we were to play action 2 over and over for rounds, then the total payoff would be the random variable . But by independence across rounds and the linearity of expectation, the expected payoff is just . So we can describe the *best action* as the action with the highest expected payoff. Define

We call the action which achieves the maximum .

A *policy* is a randomized algorithm which picks an action in each round based on the history of chosen actions and observed rewards so far. Define to be the action played by in round and to be the number of times we’ve played action in rounds . These are both random variables. Then the cumulative payoff for the algorithm over the first rounds, denoted , is just

and its expected value is simply

.

Here the expectation is taken over all random choices made by the policy and over the distributions of rewards, and indeed both of these can affect how many times a machine is played.

Now the *cumulative regret* of a policy after the first steps, denoted can be written as

And the goal of the policy designer for this bandit problem is to minimize the expected cumulative regret, which by linearity of expectation is

.

Before we continue, we should note that there are theorems concerning *lower bounds* for expected cumulative regret. Specifically, for this problem it is known that no algorithm can guarantee an expected cumulative regret better than . It is also known that there are algorithms that guarantee no worse than expected regret. The algorithm we’ll see in the next section, however, only guarantees . We present it on this blog because of its simplicity and ubiquity in the field.

## The UCB1 Algorithm

The policy we examine is called UCB1, and it can be summed up by the principle of **optimism in the face of uncertainty**. That is, despite our lack of knowledge in what actions are best we will construct an optimistic guess as to how good the expected payoff of each action is, and pick the action with the highest guess. If our guess is wrong, then our optimistic guess will quickly decrease and we’ll be compelled to switch to a different action. But if we pick well, we’ll be able to exploit that action and incur little regret. In this way we balance exploration and exploitation.

The formalism is a bit more detailed than this, because we’ll need to ensure that we don’t rule out good actions that fare poorly early on. Our “optimism” comes in the form of an *upper confidence bound* (hence the acronym UCB). Specifically, we want to know with high probability that the true expected payoff of an action is less than our prescribed upper bound. One general (distribution independent) way to do that is to use the Chernoff-Hoeffding inequality.

As a reminder, suppose are independent random variables whose values lie in and whose expected values are . Call and . Then the Chernoff-Hoeffding inequality gives an exponential upper bound on the probability that the value of deviates from its mean. Specifically,

For us, the will be the payoff variables for a single action in the rounds for which we choose action . Then the variable is just the empirical average payoff for action over all the times we’ve tried it. Moreover, is our one-sided upper bound (and as a lower bound, sometimes). We can then solve this equation for to find an upper bound big enough to be confident that we’re within of the true mean.

Indeed, if we call the number of times we played action thus far, then in the equation above, and using we get that , which converges to zero very quickly as the number of rounds played grows. We’ll see this pop up again in the algorithm’s analysis below. But before that note two things. First, assuming we don’t play an action , its upper bound grows in the number of rounds. This means that we never permanently rule out an action no matter how poorly it performs. If we get extremely unlucky with the optimal action, we will eventually be convinced to try it again. Second, the probability that our upper bound is wrong decreases in the number of rounds independently of how many times we’ve played the action. That is because our upper bound is getting bigger for actions we haven’t played; any round in which we play an action , it must be that , although the empirical mean will likely change.

With these two facts in mind, we can formally state the algorithm and intuitively understand why it should work.

**UCB1:**

Play each of the actions once, giving initial values for empirical mean payoffs of each action .

For each round :

Let represent the number of times action was played so far.

Play the action maximizing .

Observe the reward and update the empirical mean for the chosen action.

And that’s it. Note that we’re being super stateful here: the empirical means change over time, and we’ll leave this update implicit throughout the rest of our discussion (sorry, functional programmers, but the notation is horrendous otherwise).

Before we implement and test this algorithm, let’s go ahead and prove that it achieves nearly optimal regret. The reader uninterested in mathematical details should skip the proof, but the discussion of the theorem itself is important. If one wants to use this algorithm in real life, one needs to understand the guarantees it provides in order to adequately quantify the risk involved in using it.

**Theorem: **Suppose that UCB1 is run on the bandit game with actions, each of whose reward distribution has values in [0,1]. Then its expected cumulative regret after rounds is at most .

Actually, we’ll prove a more specific theorem. Let be the difference , where is the expected payoff of the best action, and let be the minimal nonzero . That is, represents how suboptimal an action is and is the suboptimality of the second best action. These constants are called *problem-dependent constants. *The theorem we’ll actually prove is:

**Theorem: **Suppose UCB1 is run as above. Then its expected cumulative regret is at most

Okay, this looks like one nasty puppy, but it’s actually not that bad. The first term of the sum signifies that we expect to play any suboptimal machine about a logarithmic number of times, roughly scaled by how hard it is to distinguish from the optimal machine. That is, if is small we will require more tries to know that action is suboptimal, and hence we will incur more regret. The second term represents a small constant number (the part) that caps the number of times we’ll play suboptimal machines in excess of the first term due to unlikely events occurring. So the first term is like our expected losses, and the second is our risk.

But note that this is a *worst-case bound* on the regret. We’re not saying we will achieve this much regret, or anywhere near it, but that UCB1 simply cannot do worse than this. Our hope is that in practice UCB1 performs much better.

Before we prove the theorem, let’s see how derive the bound mentioned above. This will require familiarity with multivariable calculus, but such things must be endured like ripping off a band-aid. First consider the regret as a function (excluding of course ), and let’s look at the worst case bound by maximizing it. In particular, we’re just finding the problem with the parameters which screw our bound as badly as possible, The gradient of the regret function is given by

and it’s zero if and only if for each , . However this is a *minimum* of the regret bound (the Hessian is diagonal and all its eigenvalues are positive). Plugging in the (which are all the same) gives a total bound of . If we look at the only possible endpoint (the ), then we get a local maximum of . But this isn’t the we promised, what gives? Well, this upper bound grows arbitrarily large as the go to zero. But at the same time, if all the are small, then we shouldn’t be incurring much regret because we’ll be picking actions that are close to optimal!

Indeed, if we assume for simplicity that all the are the same, then another trivial regret bound is (why?). The true regret is hence the minimum of this regret bound and the UCB1 regret bound: as the UCB1 bound degrades we will eventually switch to the simpler bound. That will be a non-differentiable switch (and hence a critical point) and it occurs at . Hence the regret bound at the switch is , as desired.

## Proving the Worst-Case Regret Bound

*Proof. *The proof works by finding a bound on , the expected number of times UCB chooses an action up to round . Using the notation, the regret is then just , and bounding the ‘s will bound the regret.

Recall the notation for our upper bound and let’s loosen it a bit to so that we’re allowed to “pretend” a action has been played times. Recall further that the random variable has as its value the index of the machine chosen. We denote by the indicator random variable for the event . And remember that we use an asterisk to denote a quantity associated with the optimal action (e.g., is the empirical mean of the optimal action).

Indeed for any action , the only way we know how to write down is as

The 1 is from the initialization where we play each action once, and the sum is the trivial thing where just count the number of rounds in which we pick action . Now we’re just going to pull some number of plays out of that summation, keep it variable, and try to optimize over it. Since we might play the action fewer than times overall, this requires an inequality.

These indicator functions should be read as sentences: we’re just saying that we’re picking action in round and we’ve already played at least times. Now we’re going to focus on the inside of the summation, and come up with an event that happens at least as frequently as this one to get an upper bound. Specifically, saying that we’ve picked action in round means that the upper bound for action exceeds the upper bound for every other action. In particular, this means its upper bound exceeds the upper bound of the best action (and might coincide with the best action, but that’s fine). In notation this event is

Denote the upper bound for action in round by . Since this event must occur every time we pick action (though not necessarily vice versa), we have

We’ll do this process again but with a slightly more complicated event. If the upper bound of action exceeds that of the optimal machine, it is also the case that the maximum upper bound for action we’ve seen after the first trials exceeds the minimum upper bound we’ve seen on the optimal machine (ever). But on round we don’t know how many times we’ve played the optimal machine, nor do we even know how many times we’ve played machine (except that it’s more than ). So we try all possibilities and look at minima and maxima. This is a pretty crude approximation, but it will allow us to write things in a nicer form.

Denote by the random variable for the empirical mean after playing action a total of times, and the corresponding quantity for the optimal machine. Realizing everything in notation, the above argument proves that

Indeed, at each for which the max is greater than the min, there will be at least one pair for which the values of the quantities inside the max/min will satisfy the inequality. And so, even worse, we can just count the number of pairs for which it happens. That is, we can expand the event above into the double sum which is at least as large:

We can make one other odd inequality by increasing the sum to go from to . This will become clear later, but it means we can replace with and thus have

Now that we’ve slogged through this mess of inequalities, we can actually get to the heart of the argument. Suppose that this event actually happens, that . Then what can we say? Well, consider the following three events:

(1)

(2)

(3)

In words, (1) is the event that the empirical mean of the optimal action is less than the lower confidence bound. By our Chernoff bound argument earlier, this happens with probability . Likewise, (2) is the event that the empirical mean payoff of action is larger than the upper confidence bound, which also occurs with probability . We will see momentarily that (3) is impossible for a well-chosen (which is why we left it variable), but in any case the claim is that one of these three events must occur. For if they are all false, we have

and

But putting these two inequalities together gives us precisely that (3) is true:

This proves the claim.

By the union bound, the probability that at least one of these events happens is plus whatever the probability of (3) being true is. But as we said, we’ll pick to make (3) always false. Indeed depends on which action is being played, and if then , and by the definition of we have

.

Now we can finally piece everything together. The expected value of an event is just its probability of occurring, and so

The second line is the Chernoff bound we argued above, the third and fourth lines are relatively obvious algebraic manipulations, and the last equality uses the classic solution to Basel’s problem. Plugging this upper bound in to the regret formula we gave in the first paragraph of the proof establishes the bound and proves the theorem.

## Implementation and an Experiment

The algorithm is about as simple to write in code as it is in pseudocode. The confidence bound is trivial to implement (though note we index from zero):

def upperBound(step, numPlays): return math.sqrt(2 * math.log(step + 1) / numPlays)

And the full algorithm is quite short as well. We define a function ub1, which accepts as input the number of actions and a function reward which accepts as input the index of the action and the time step, and draws from the appropriate reward distribution. Then implementing ub1 is simply a matter of keeping track of empirical averages and an argmax. We implement the function as a Python generator, so one can observe the steps of the algorithm and keep track of the confidence bounds and the cumulative regret.

def ucb1(numActions, reward): payoffSums = [0] * numActions numPlays = [1] * numActions ucbs = [0] * numActions # initialize empirical sums for t in range(numActions): payoffSums[t] = reward(t,t) yield t, payoffSums[t], ucbs t = numActions while True: ucbs = [payoffSums[i] / numPlays[i] + upperBound(t, numPlays[i]) for i in range(numActions)] action = max(range(numActions), key=lambda i: ucbs[i]) theReward = reward(action, t) numPlays[action] += 1 payoffSums[action] += theReward yield action, theReward, ucbs t = t + 1

The heart of the algorithm is the second part, where we compute the upper confidence bounds and pick the action maximizing its bound.

We tested this algorithm on synthetic data. There were ten actions and a million rounds, and the reward distributions for each action were uniform from , biased by for some . The regret and theoretical regret bound are given in the graph below.

Note that both curves are logarithmic, and that the actual regret is quite a lot smaller than the theoretical regret. The code used to produce the example and image are available on this blog’s Github page.

## Next Time

One interesting assumption that UCB1 makes in order to do its magic is that the payoffs are stochastic and independent across rounds. Next time we’ll look at an algorithm that assumes the payoffs are instead *adversarial*, as we described earlier. Surprisingly, in the adversarial case we can do about as well as the stochastic case. Then, we’ll experiment with the two algorithms on a real-world application.

Until then!

Lucid explanation. Thanks

LikeLike

Hi Jeremy,

Regarding $X_{t,i}$ I should add since you assume they have the same mean it appears to me that you are assuming [implicitly] that they are from the same distribution which from my point of view justifies replaceing it with $X_i$. Am I wrong?

LikeLike

I think it’s notationally more appropriate to have separate random variables, each drawn from the same distribution, because there is a difference between and as random variables. For example, one is affected by statements like the Central Limit Theorem.

LikeLike

I agree with you that these two are different but even in that case what you need is to declare that there are stationary process; What seemed confusing for me was the fact that you assumed the “independently” but not “identically” of i.i.d. random variables above, right?

Btw, thanks for these great articles.

LikeLike

I hope the identical part was clear from the prose.

LikeLike

I agree with you that these two are different but even in that case what you need is to declare that there are stationary process; What seemed confusing for me was the fact that you assumed the “independently” but not “identically” of i.i.d. random variables above, right?

Btw, thanks for these great articles.

LikeLike

I see. I am just so skeptic 🙂 . At least now I am sure the claim was exactly about i.i.d.s

LikeLike

i think you have a code error in your python code. in ucb1 function, you never actually applied part 2 algorithms because it stops at the initialize empirical sums step.

LikeLike

The python function is a generator, so you have to iterate through the generator in order for it to run all the steps. For example, you might run:

for roundinfo in ucb1(…):

print(roundinfo)

LikeLike

This is a really well-written explanation of UCB1, Jeremy. Thanks!

Just had one question though:

How do you know when to stop the experiment? Is the computed upper bound value ( Math.sqrt(2*logT/n_j) ) analogous to the confidence interval? So if I had three arms after T plays with, say, upper bound values of: 0.09, 0.08, and 0.05, what does that actually mean?

LikeLike

It depends on your goal. In practice, often the goal is to serve ads to users, which never ends. If you’re trying to identify the single best arm, then you have to add additional assumptions to find it because, even though the process will converge in the limit to a distribution with overwhelming weight on the best arm, the convergence rate will depend on the difference of the payoff of the best arm and the second best arm, but you can use the last line of the main proof (along with a union bound over all the arms) to bound the probability that you pick a suboptimal arm. Then set that less than your threshold and solve for T.

LikeLike

Why “then another trivial regret bound is \Delta T” ?

LikeLike

In this case you never pick an optimal action (and all the penalties for suboptimal actions are the same). So you incur the most regret in every of the T rounds.

LikeLike

May I know, is UCB a good algorithm for adversarial settings, where an expert changes the reward in each iteration?

LikeLike

In practice, perhaps. In theory, no. See EXP3 and it’s variations for that. e.g., https://jeremykun.com/2013/11/08/adversarial-bandits-and-the-exp3-algorithm/

LikeLike

Technical point: I don’t think you can reduce to all the $\Delta_i$ being the same and then compare to the function $\Delta T$ without some further argument. You’ll have to maximize a combined function $\min(R(\Delta_1,\ldots),T \max \Delta_i)$, or something like this.

LikeLike

Hello brother, congrats by your explanation! Is great, but I had some problems on running: appears this log error, when I run your code

for roundinfo in ucb1(100, 1):

print(roundinfo)

Appears that on log:

Traceback (most recent call last):

File “/home/ubuntu/workspace/ex50/bin/mab.py”, line 35, in

for roundinfo in ucb1(100, 1):

File “/home/ubuntu/workspace/ex50/bin/mab.py”, line 13, in ucb1

payoffSums[t] = reward(t,t)

TypeError: ‘int’ object is not callable

Can you help me please? Thankful!

LikeLike

Yes. The

`reward`

input to`ucb1`

is a function that accepts two inputs: an integer representing which action was chosen, and the current time step. It produces as output the reward for that action in that round (the rewards may change over time). You can see an example input where the reward of each action is a biased random number here: https://github.com/j2kun/ucb1/blob/master/ucb1.py#L50LikeLike

For the part “Theorem: Suppose UCB1 is run as above. Then its expected cumulative regret”. For the second term with summation, should it be added from 1 to T instead of 1 to K?

LikeLike