Annie Duke: Quitting, Bill Gates, and poker

When we look at a lot of the way that we think about quitting from a cultural standpoint, we think that quitters are losers, are weak-willed or lack character. That's on the cultural side. But then there's a lot of cognitive biases that we know about individually, things like sunk, cost bias, endowment effect, which we can go into a whole bunch of things like that. Where, when you actually look across those biases, they all result in a failure to stop things. I think it's very multi-pronged, which I think is part of what makes it such a hard problem to address.

What we'd ideally like to do as humans is when we're approaching something that we're trying to decide whether we're going to continue. We would only continue that thing if we were willing to start it fresh today. If we applied that to, for example, a stock that we bought, let's say that there's a stock trading at 40 and we've never owned it. And we look at it and we do our analysis on it and we decide that it's not a buy that we ought not buy it at 40. We'd like to then, if we bought it at 50 and it's now trading at 40, be able to sell it. Because when it's trading at 40, regardless of whether we owned it at 50. What we're really saying is that I wouldn't buy it today, so if I wouldn't buy it today, I should sell it. Because holding is the same as buying, right?

So the problem with the sunk cost fallacy is that we don't do that. And we have that feeling that I think everybody can intuit of. If I sell it, I won't be able to get my money back.

If I walk away now, I'll have wasted everything that I put into it. So, as an example, if you're in a job that there's no way that you would take today, you won't walk away. Because you're afraid you'll have wasted everything that you put into it, all the training all the time and that kind of thing. But again, if you wouldn't take it today, you ought not to continue forward. And really, what it comes down to is that we have this irrationality. Where we think of waste as a retrospective problem, what I will have wasted, as opposed to a prospective problem. Which is, is the next dollar, the next unit of effort or attention that I put into this thing going forward, going to be a positive expected value?

Am I going to actually win to it? And we have to start getting our eyes forward on these things.

Or it could be I can buy this stock and I'm winning to the decision, but not as much as I would win if I put my money in a different stock. Okay, so that's how we want to think about expected value, so expected value only makes sense if you related to Nikolai, what you value or what your goals are right. So something that might be positive expected value for me might be negative expected value for you, and vice versa, so we can think about.

For me, I wouldn't want to start climbing Mount Everest because like the discomfort, and I'd be very cold and it wouldn't really be my thing. But for you, you may be somebody who really wants to stand on the top of the world and do something that is different than almost any human being has been able to accomplish.

Like, if it's still worth it, you shouldn't stop, but failure is also failing to stop something that is no longer worthwhile. So I think, for example, if you continue to climb Everest when you're in the middle of a blizzard, I would consider that a failure. So some people don't want to turn around there because they're going to process that as a failure to have reached the summit. But I think that once you start climbing under conditions, when you're clearly negative expectancy, then that's failure.

If we take a very simple example, like the stop, right, so we buy a stock at 50, we sell it at 40. We feel like a failure because we lost $10, but what the real failure is would be to continue to hold it. When you could put those resources, that $40 into something that is actually going to win, that's actual failure.

And you actually want to buy more, even though you wouldn't start it today. So how do we actually overcome that problem? Because obviously, if we're rationalizing away or ignoring the signals that tell us we should stop, how do we solve the problem? And it turns out there's a relatively simple trick that actually really drastically increases the chance that you'll stop when the time is right. And that is, to think about what those signals that you might encounter are in the future, and then commit before you start to actions if you see those signals.

So I'll give you a very simple example. Let's just imagine that it's 2021. And you have bought Bitcoin. And your stated reason, your thesis for buying bitcoin is that you believe will be a good hedge against the market going down. OK, so that's why you are buying it?

If the market goes down, you believe that Bitcoin will be uncorrelated with that, and so it will act as a good hedge against sort of market chaos. So now I'm going to say to you, Nikolai, all right, so let's say that you buy it. How much of a correlation would you have to see over what time period between the market going down and Bitcoin going down as well? So we can now write that down.

So if you see a strong enough correlation and you determine what that is over a certain amount of time, so it's durable. That you'll actually start to take risk off and sell bitcoin, so if we do that in advance, you're much more likely to take risk off than if you just say, well, my thesis implies that. So obviously, I'm going to pay attention when bitcoin is going down along with the market. And what that helps you to avoid is rationalization. No, I still believe it.

And now Bitcoin is cheaper, so I'll buy more, or the other thing that will happen in order to avoid selling is thesis creep. Well, yeah, OK, it is correlated, but actually, I own Bitcoin as a philosophical move, right? Or something like that. So there's a very simple example of a kill criteria that actually mountaineers use.

So if you take Everest on summit day, they leave Camp 4 at about midnight. And the reason is that they want to get to this part of the mountain called the Southeast Ridge, which is really dangerous. You have to go up single file. They want to get there in daylight, on the way up the mountain, but they also want to get there in time that they can come back down and traverse it again in daylight.

All right, so this is kind of a ballet, right? Like they've got to sort of deal with this dance so they leave at midnight. But what that also means is, in order to get back down the Southeast Ridge, you must have reached the summit by 1 p..m. And if you reach the summit past 1 p..m, the probability that you're coming down that Southeast ridge in darkness is too high.

And the Southeast Ridge, again, is very, very dangerous. You don't want to traverse it in darkness. So they actually set on that day a turnaround time of 1 p..m. No matter where you are on the mountain, you have to turn around at 1 p..m. So this is a very good, simple example of a kill criteria that gets people to turn around when they otherwise might not.

And I actually don't know what the analytics are, whether that was correct for them to do on those tries, but if it was mathematically correct, it was mathematically correct. It doesn't really matter. But none of it worked out, so I think. Nate Silver posted that it was going to set fourth down decisions back a decade just because it didn't work out.

So one of the famous examples of that actually also comes from American football in the 2015 Super Bowl.

The Seattle Seahawks were on the one yard line of the New England Patriots, so they have to move the ball one yard. They're four points down, meaning they have to score a touchdown in order to go ahead. And there's only 26 seconds left in the whole game and Pete Carroll decides to throw a pass play in this situation.

And the expected play is for them to run the ball, hand it off to the running back, try to get across the goal line. But he chooses a pass play, the pass is intercepted. And people to this day think it was one of the worst calls in Super Bowl history. I'm not going to go into the details of the math, but it's not even close.

The choice that he made is the much better choice, like by a lot.

What's the probability of an incomplete pass, what's the probability of the run scoring? So and so forth? Because we have base rates for all that stuff and we can figure out like which, on average, is going to get you the best results. But what happens is that the outcome so overshadows the result of the decision, so overshadows our ability to think that through.

That none of that really matters. When we're evaluating the decision, and we can see that with the Pete Carroll decision, it's intercepted. He loses the game, everybody says it's the worst decision in Super Bowl history.

But we can do the thought experiment really easily. Well, what if it had been caught for the game winning touchdown? Obviously everybody would have said it was an amazing decision. And as you know, in investing, it's like, if you invest in one stock and it loses, it doesn't mean it was a bad decision. If you invest in a well diversified portfolio over some durable period of time, where you continually are performing below beta, now I can say something about your decision making, right?

But I have to have enough data points to be able to do that, and what we do instead is we take one data point. We look at the outcome and say that was a mistake.

So basically, what I try to get people to do when I'm working with clients is I take. I say, Look, you have some model that you're using for making these decisions. I would like to take that model and make it explicit, and then make sure that you're judging your decisions based on that model. This is for big decisions, so let me give you an example. So, one of the favorite things that venture investors, early stage venture investors, like to say, and I'm sure you've heard this, is when you ask them how they make their decisions. They say, I know a good founder when I see one.

Right, okay, so what I would say to somebody in that situation is, I can't disagree with you. You may very well know a good founder when you see one. I don't have the data to be able to disagree with you.

But what do you mean by good founder?

So now I'm going to ask them to be explicit by what they mean by good founder. What is your model of a good founder, right? Is it someone who's gone really deep on the problem? is it someone who is single-minded? is it someone who, whatever it is, right?

So we're going to write down what are the qualities of somebody, a founder, that makes you put them in the category of good founder. And then I might go beyond that and say, when you think about the investment broadly, what are the things that you're thinking about? Right? So they might tell me, well, I want it to be in a good market with no headwinds and growing. TAM. I want the product to be something that is going to solve a problem for people, whatever. So we're now going to create a decision rubric, which basically is an explicit representation of what your model of a good investment is.

So all I'm doing is taking your belief, I'm not disagreeing with what you think, I'm taking how you model these problems, and I'm making it explicit now. What that's going to do is as you make decisions about investments, it's going to reduce the noise in your investments. It's also going to reduce the bias, because it's going to be hard for you to downplay bad signals or overplay good signals.

And then the other thing it's going to do is it's going to help you to up-level your team very quickly. Because, let's say, Nikolai, that you are actually amazing. Now we've created the model of how you view investments, it's now explicit. We can share that out with the other people on your team, who now are going to learn a lot faster how to make great investments.

Not only that, we become a data collection machine, because every investment we're actually now writing down explicitly, what is Nikolai's point of view on this investment? So now, when we get a good or bad result, we can now go back and not fall into the resulting problem. Because we can go back and look at what you were thinking at the time.

I'm simplifying how strong the product is, how strong the founder is. I'm going to have you make some forecast of whatever, like, you could make a forecast of. How many widgets do you believe this company is going to produce in the next six months? Whatever. So it's whatever you care about in an investment, so you're going to fill that out. I don't want to get your opinion in front of other people.

So I'm going to give you that rubric independently of the next person, and the next person and the next person. And this is probably the single biggest thing you can do in order to improve decision making. Because when you talk in groups, it makes your decisions in general worse. And what I mean by that is not that you discuss other people's opinions in groups, it's that you do the opinion discovery in groups.

Because, remember, decisions have two parts, right? There are facts, but then there are also your subjective judgments. How are you modeling those facts? So you could look at facts about a particular market and I could look at facts about a particular market.

And we could have very different judgments of that market, even so. So, where we're going to be really helped as a team is to understand that you're really bullish on that market. I'm really bearish on that market, even though we're looking at the same set of facts. And for me to truly understand what your rationale is for why you're bullish, for you, to truly understand what my rationale is for why I'm bearish for us. Not to try to convince each other of the other person's point of view, but to, with curiosity, understand why the person is modeling the decision that way. Because all of the increase in accuracy is going to happen in those gaps.

Now, in order to do that, though, I have to know the gap exists. And the problem is if we're sort of eliciting those opinions in a group setting, you're going to talk about how strongly you feel that this is a great market. And me over here who came into the room being bearish, I am going to be much less likely to express that opinion.

Who have the right knowledge set to be able to actually give me good advice here? And then here's the key. Because, remember this idea of I don't want to contaminate anybody, right? I want to know what their true point of view is. When I elicit their advice, I don't tell them what I think. So this is something that I learned in poker.

So obviously, if I'm coming to ask you about a hand of poker that I played, I'm asking you about something that has already occurred. So I know what my opinion was because I executed it, my opinion was I should raise or fold, or call or whatever. And I also know how the hand turned out, right, so I know both of those things.

So if I come and I tell you the whole hand and then ask to rewind to ask you advice, I'm not going to get good advice. For a couple of reasons. One is the resulting problem. If you know I lost the hand, you're going to tend to interpret my decisions negatively. If you know I won the hand, you're going to tend to interpret them positively. And the other thing is, if you already know what I thought I should do, that's going to now infect your opinion.

So the way that I would describe a hand to you is, Hey, Nicolai, I have a question about a hand. I played. This person was in front of me, they were in first position, they raised. I would tell you things about the way that person played, that you needed.

Were they aggressive, were they not aggressive, were they tight, were they loose, these kinds of things, right?

I would tell you how many chips they had, and then I would say, So, now I'm in second position, I look down and I have Ace Queen.

And then that's it. I say, What would you do?

So let's say that we're playing chess and I lose at chess to you. This is a game that has very little luck. I can't do anything but sort of say I've got to look back and examine my decisions. Because what's the explanation for why I lost, right?

And if I won, I should rightly be able to say I should look back, because I think I probably made decisions well. But once we add volatility in, it becomes really hard to close a good feedback loop. Because what I would see from players all the time, and I certainly had this urge myself, is that when they lost, it was always luck's fault, and when they won, it was always due to their skill. So, one of the reasons why when I was talking to other people to get their advice, I never told them how it turned out, was because I wanted to make sure I was getting their true opinion.

Because I wanted to know, win or lose, did I play the hand? Well? That's what I was really obsessed with. But it's really hard because it comes to this thing. And this is, I think, the biggest lesson that I learned from poker.

And I think that Daniel Kahneman, Nobel laureate, I think he also thinks this is the biggest urge for humans. That really creates a lot of decision error is that we always want to have a positive self-narrative of our life. And there's a catch-22 in there. Which is making great decisions over time and accepting your mistakes over time will in the end create the most positive self-narrative. But human beings aren't that way. Human beings want to feel good about themselves now, not some future version of you.

You've never even met now, so what happens when you lose a hand at poker? You feel bad, right? Oh, but you feel bad. Maybe I made mistakes.

Maybe I'm a bad player. You're not advancing that positive self-image in that moment, and so what do we do? We say, Oh, I got bad luck.

But we can think about a two-by-two matrix, right? Where you've got on one side the decision quality, good decision, bad decision, and then you've got the outcome quality, good outcome, bad outcome, right? And if you've got good good, that would be an earned reward. If you've got good bad, that would be bad luck. If you've got bad good, that would be dumb luck.

And if you've got bad, bad, that would be what you deserve, right? All right. So we can think about when we're thinking for ourselves, the way that we interpret the world when luck is. An available. Explanation is. Everything that we're doing is trying to get back to this idea that we made a good decision.

So that means that we interpret the world either as earned reward or as bad luck. This is something called self-serving bias. And boy, if you ever want to see that in play, just go into a casino, because that's all you're going to hear.

So, just to give you an idea, if I sat down in a $3 $6 game, I might risk $200.

So Bill Gates would come in, and so this was way back in the day. Like the Bellagio Casino wasn't even open yet. It was at the mirage and I was playing. I remember I was playing in a game that was kind of low stakes for me.

I think I was playing like an 80 160 game.

And that's just the betting limits is the maximum bet is $160. Well, this person taps me on the shoulder and says, Do you mind if I sit down behind you and watch you play? And I'm about to turn around and say no? Like, you're a stranger?

Why would I let you watch me play? And I look up and I see it's Bill Gates? So I was like, Sure, sit down and watch me play. So it's interesting. He sat down behind me and watched me play.

And he mostly asked me about bluffing.

That is, maybe, maybe, and remember this. I'm guessing at the expectancy because I can't see the other player's cards. But if I'm in a situation where playing straightforward is going to be a problem. Because people are going to start to be able to read me. And then they're not going to pay off my good hands and that's going to reduce my earn. Now I want to bring bluffing into the equation, and the bluff itself can be 50-50 or slightly less than 50-50. I can actually be slightly negative expectancy on the bluff itself because it will earn me money later. So, in other words, Nikolai, if I'm playing with you and you see that every time that I bet big, I have a good hand. At some point, you're only going to call me when you know you have a better one.

So that's going to be bad, so I'm not going to get paid because you're only going to call me when I'm beat, so that's bad.

But if you see that, sometimes when I bluff, I have an 8 and a 3 and nothing particularly good happening. And you see that from me now, when I do have a really good hand. You're going to not know, and you're going to have to start what's called paying me off, you're going to have to start calling me with weaker hands. So that's why that bluff is really important now. I have played in games with players who are what we would call very tight. Meaning that they'll only ever bet when they know they have the best hand in those types of games. Bluffs on their own are very effective.

In other words, I'm not even using the bluff for signaling value, I'm using it just because I can push you around.

And then I can also play against players who are really bad, who pay you off all the time, and against those players. I would never bluff because you're already doing what I'm trying to get the bluff to do, so I would never bluff. And then in the in-between, I'll bluff depending on where I'm sitting in that spectrum.

So that was really the discussion that I had with Bill Gates, It was a fun discussion. I've never seen him again since then, but he sat behind me for like an hour.

And the individual man might be more risk-averse, but I think that that's generally true, and it's one of the reasons why. If I'm thinking about a trading desk, and how do you compose that desk? It's really good to have a mix.

So that's where we're going to learn all of the information. So we have equally informed people who have a different subjective judgment about the facts on the ground.

Wow, now, I know, right? And it would always, from year to year, toggle between 3 and 5, never above 5, not really, ever below 3%.

But this was long before poker was on television, now poker gets on television. And there's people like me and Vanessa Selbst and Liv Boree and Maria Ho. And a variety of women who are representing women and showing that women can do this professionally and being very successful. And in my case, I had four children and I would talk very openly about how poker was actually a pretty good career. If you have kids, because you're organizing your day, where I was basically a stay-at-home mom.

And then when my kids would go to bed, I would go play poker, right? And so it actually worked out pretty well in terms of my lifestyle. But anyway, we're now repping right, we're representing women in poker. And at the time, I had the intuition.

And this is going back to that thing about intuition, how good is intuition, why you want to make it explicit? I had the intuition that. It was more that people didn't know it was something that you could do, right. It was more that people didn't know. It was something that was totally fine for a woman to do.

And so now it's been since really 2002 2003 that women have been featured playing poker. There's been lots of success among women in poker. And last year, in the 2023 main event of the World Series of Poker, the percentage of the field that was women was slightly over 3. And so I look back to that first year I played, which I think was 1994 to 2023, and all that happened in the interim. And yet there aren't more women entering that event. So my answer to you is, I don't know, right?

I have this intuition that if you get to people early and you educate them. If you start thinking about, how do you get women more comfortable in these very risky, not risky in the long run, but very risk-heavy, volatile decision-making environments? That, naturally, you'll get women to come into those fields. It's a problem that I would love to solve, but at least in poker, I didn't see it solved, so the answer is, I don't know.

But I also took classes with John Barron, who's a giant in the field of judgment and decision-making. So first, language acquisition is a problem of learning under uncertainty.

So, right at the end of graduate school, I ended up pivoting for a variety of reasons, one of which was some health reasons. And that's when I started playing poker. And for eight years, I just dove into this problem, which, as we've discussed, and we only got the tip of the iceberg. Poker is just this amazing laboratory to think about decision-making under uncertainty. What goes wrong? How can you make it go right?

And then, in 2002, I got asked by a hedge fund to speak to their traders. About how poker might inform risk, and what I talked about was how poker and cognitive science inform this idea of risk. Attitudes and relationship to the path that you've gotten onto your decision. So what I talked about was if you've been losing, how does that affect your risk attitude? If you've been winning, how does that affect your risk attitude? So it was that moment in 2002 that I really realized that there was this very interesting conversation to be had between cognitive science and poker, where the cognitive science could really inform my thinking about poker and make me a better poker player, and the poker could really inform my thinking about cognitive science and make me a better cognitive scientist.

What I kind of realized was that from the time I really was in college, working as an assistant, actually a research assistant to a cognitive scientist. What I've been really obsessed with is this idea of uncertainty and what that does to the quality of your decisions. And I think because I played poker, I've been obsessed with how do you mitigate those errors, right? It's not enough for me just to identify an error. I want to know how you mitigate it. Because that's really what I was obsessed with as a practitioner.

So what I'm obsessed with is, how do you model that fact? in other words, how do you take something that's true and figure out what the truth is? And that takes a checklist of understanding how to model this fact. So if we take this fact of 58 of the people who died of Covid were vaccinated. We have to go through a series of questions, the most important of which is what percentage of the population is vaccinated. Because otherwise, what will happen is, you'll say, oh, vaccinations don't work because more people died who were vaccinated than were unvaccinated.

But once you know that over 80 of the population in that month were vaccinated, now all of a sudden that number looks very different to you. And then you could go beyond that, and you could age match. You could say, well, let's compare 70-year-olds to 70-year-olds, and so on, and so forth. And then it turns out vaccinations in that month were five times better. But somebody looking at that, nobody's trying to lie to you, right?

They're trying to give you a data point that's going to help you decide. And if you don't understand how to model that, how can you come up with what the truth of the matter is? And I've been super obsessed with that topic recently.

Luck you have no control over, you can't control the circumstances of your birth, for example. And we know that people are born into different circumstances that change sort of the distribution of outcomes that are available to them. And you can't do anything about that. So, first of all, embrace the luck. It is what it is, right.

I can't do anything about it, but what you can do something about is the quality of your decision. And over time, if you're making better decisions than you otherwise would have, those gains will accrue over time and it's going to change. That's the thing that can change the trajectory of your life. So it's this idea of, while you can't control luck, you are an agent of your own decisions.

And if you can become totally focused on, how do I improve the quality of your decisions? That's the thing that can change your life. Because better decisions lead to better outcomes for individuals, and in fact, that also leads to better outcome for society, because society is a collection of individuals. So really start focusing on your decisions and become an agent in your own decisions.