Hindsight bias and overconfidence

4th October 2011 by Mindful Money

Why did the chicken cross the road?

Too abstract?  How about, why did gold’s price drop?

We think we understand why the past happened as it did, when all we actually know is what happened

Humans evolved to like simple solutions – we don’t have the mental capacity to hold lots of independent data (that’s why we have statistics, to summarise) and we have many mental shortcuts that helped us survive for over a million years.  Those shortcuts are useful in everyday life (who wants to evaluate the merits of each item of clothing each day before dressing or to decide which arm to put into a jacket first?)  But because they make life so simple, we forget that we’re using them and that they haven’t (and won’t) disappear.

One of those shortcuts is to assume that we understand life, particularly the past (hence the expression, “we have 20:20 vision in hindsight”).

Past events seem far more inevitable than they were – Obama’s election, the US property crash etc.  Before, nobody knew, afterwards everybody said it was obvious and can point to the moment when it became inevitable and the predictor that told them it would happen.  But each person points to a slightly different moment and a different predictor.

And if you examine the “estimate of likelihood” of events, people’s memories are strangely distorted.

It’s a feature called hindsight bias.

That phenomenon is complex too, but there are three factors usually associated with it, plus something that makes it all even more complicated.

One factor is availability – if you have it available in memory, it must be important.  So after the book and film of Jaws came out, people thought shark attacks were much more likely than they were.  Media coverage of unusual events, like shark attacks, means we distort what is likely.  And we remember that a gold price crash was really likely, because the media is full of the gold price crashing – so we must have expected it, mustn’t we?

Another factor is Representitiveness – the degree to which your pet theory or what you remember happening resembles (represents) the available data.  We make massive approximations, ignore the base rates of actual occurrences (like shark attacks versus common illnesses) and get our statistics wrong.

The third factor is that we then look at what happened and assume that what we saw fitted our theory and “selectively forget” the data that don’t fit.

It makes things simple, and tidy – and frequently misleads us into believing we’ve understood why what happened did so

The thing that makes it even more complex is chaos theory.  I won’t even attempt to explain that – just give two examples.

In the Black Swan, Taleb points out that you can predict the size and shape of the pool of water formed by the melting of an ice sculpture.  But given the pool of water, you can’t recreate the ice sculpture.  We know where we are, but we can’t necessarily know the route by which we got there.

In the Discworld books (and if you don’t read Terry Pratchett, you should) there is much talk of “the trousers of time”.  We never know precisely which leg we’ve gone down, and aren’t even conscious of which leg we’re in or the existence of the other legs.

Why did the chicken cross the road?  Maybe it was to get to the other side, maybe it wasn’t conscious of the concept of a road, maybe it was following the food, joining it’s friends, exploring – nobody knows, probably not even the chicken.

But we all think we know, and me writing this won’t alter the conviction that most people have that they know exactly why events in a chaotic, non-linear, complex system like a market happened.

Sign up for our free email newsletter here.

2 thoughts on “Hindsight bias and overconfidence”

  1. Anonymous says:

    Sorry I’m so late and know no body will pay any attention anyway, but I really liked this article and should be given to all those young scientists about to embark on a career of research.  In my day we used to subscribe to a saying:  “There is nothing like a few facts to get in the way of a good theory.”
    However, that said, I do disagree with: ” We know where we are, but we can’t necessarily know the route by which we got there.”   It is true that it may be difficult or impossible to retrace steps.

    1. Kim Stephenson says:

      Thanks, particularly for the “give it to young scientists” point.  Both theory and fact are important, but that gets simplified even more with many people valuing experience and ignoring theory and science and others valuing only theory and ignoring facts (particularly ones that prove the theory wrong – Popper would turn in his grave!) 
      One thing, do you mean that you disagree, it looks as if you’re confirming what I said with your comment that it may be difficult or impossible to retrace steps, did you perhaps miss the second part of that sentence?

Leave a Reply

Your email address will not be published. Required fields are marked *