Why results can overshadow good and bad decision making

Results, results, results. That’s what counts – of course, it makes sense. But results can be the result of fluke and chance. How do we distinguish between bad decision making that has a fluke positive outcome, and good decision making that has a fluke bad outcome? It’s hard because, as I show, our brains are wired to measure outcomes and not processes.

"Don't make decisions based on results” may seem like really bad business advice, particularly from an article on decision making. But that is essentially what I am saying – I am not saying that results are bad, they are critical, but the point is to avoid outcome bias. Why so? Good decisions can have bad outcomes, and bad decisions can have good outcomes? Am I confusing you even more? Let me give an example:

It is the dying moments of the legendary Rugby World Cup Game between Japan, massive underdogs, and South Africa, a global force in rugby, on 19^th September 2015. Japan had matched South Africa the whole match and 5 minutes towards the end it was even at 29-29. This in itself was a shock – Japan had held the great South Africa to a draw, this would have been a huge upset.

But South Africa got and kicked a penalty to lead by three points with the clock ticking down. Just as the clock was ticking out with seconds to go Japan received a penalty. They could now take the safe option and kick for goal giving them a draw - their best ever performance against one of the so-called “tier one” rugby nations, and a valuable drawing point. Or they could strategically kick to the corner and attempt from a line-out (a throw in, for you non rugby fans) to score a try which would give them victory but also a loss if they failed, which is more than likely. So, what do you do? Take the points and have the best ever result for your nation, or take the much lower chance for victory, greater glory, but also a greater chance of a loss?

They kicked for the corner. Their coach Eddie Jones, now coaching England, groaned. And then the remarkable happened, leading to the biggest upset in Rugby World Cup History, and this match going down in the history books as one of the greatest of all time. Japan scored a try and won the match – and the day became known as the Brighton Miracle (made into a film in Japan in 2019).

So, the question is, was this decision to kick to the corner, a good decision or bad decision?

In the light of the result, the outcome, it was a good decision. But hang on a second just a few days later this very same decision was now in the hands of England, playing Wales. At stake was progressing through to the next round or exiting in England’s home World Cup. It is the dying minutes of the match. England have been awarded a penalty and trail Wales by three points. Three points will give them a draw and the potential to progress through to the next round. Loss will be an almost certain exit from the tournament. Victory will be an almost certain progression to the next round. The decision is almost identical to that of Japan: in the same dying seconds, kick for goal for a draw and safety, or kick to corner for glory? England kicked to the corner. Wales were prepared for this, disrupted the attempt of England to drive for the line. The ball went dead. England lost and ended up exiting the world cup in shame on home soil.

Two almost identical decisions made in the heat of the moment with a lot at stake. Two different outcomes. The first was seen as a belief in the ability of the team and desire to win. The second as a bad strategic choice. But that is only in light of the outcome. If England had won, it would have been a good decision. If japan had lost they may be mourning the chance they had to equal a big rugby nation. This is outcome bias – judging the decision on the outcome not the decision quality itself. One that is prevalent in sports but not less so in the business world, and our everyday lives.

The term outcome bias was probably first proposed by Jonathon Baron and John Hershey in 1988 in a key paper titled “Outcome Bias in Decision Evaluation”. In this paper they outlined a series of experiments they conducted on decision making. These involved different decision-making contexts such as in a medical context (heart bypass surgery), or in gambling. The description of the basis for the decision was given including probabilities of different outcomes - but different groups were presented with different outcomes.

The quality of the decisions was then rated on a scale such as:

3 = clearly correct, and the opposite decision would be inexcusable

2 = correct, all things considered

1 = correct, but the opposite would be reasonable too

0 = the decision and its opposite are equally good

-1 = incorrect, but not unreasonable

-2 = incorrect, all things considered

-3 = incorrect and inexcusable.

What is interesting in these experiments is that participants were asked to focus on the decision-making process itself, not the outcome, and to rate the quality of the decision. Despite this focus on the decision making itself there was consistent outcome bias in the rating of decisions. This means that those that had a bad outcome were consistently rated as worse decisions, and those that had positive outcomes were rated as good decisions.

This bias is based on a logical interpretation of events and natural learning mechanisms – try something and it has a positive outcome = good, try again. Try something and it has a negative outcome = bad, don’t try again, or try something different. This is the basic evolutionary roots of outcome bias and “in the wild” is a pretty good, if very rough and ready, rule to follow.

So, as a primitive simplistic learning mechanism, it would make sense. The point here is that outcomes particularly in a complex world and in the complex world of business have many potential influencing factors. Some of these are random events, luck, chance, or unknown, and unknowable influences. Making good decisions means making the decisions with best chances of success more often. In the world of business with multiple decisions being made, multiple times during the day, the more of these that are good decisions the higher the chances of success over time.

There are also a few other interesting aspects to consider in decision-making and outcome bias scenarios. The above-mentioned experiments also showed that experts, in the above examples, physicians specifically, are judged more harshly on outcome bias. There is an assumption that as an expert they should somehow know better. Secondly information that becomes present later is also applied at source – so unconsciously we update the decision-making process even though we logically should know that the decision maker did not have that information available at the time. Thirdly there seem to be also a “a belief in luck or clairvoyance as a consistent trait” as Baron and Hershey note. Some people expect or consider this ability to be lucky or have this clairvoyance as a personality trait.

The results of Baron and Hershey’s experiments, and many others, show that on average there is an outcome bias even when specifically asked to focus on decision quality - but when we look at the data, we can also see that not everyone has outcome bias. Some people, Baron and Hershey noted, show clear and rational evaluation of the probabilities and quality of decisions. But even so, on average, there is a persistent outcome bias.

So, to summarise so far:

• There is a consistent and persistent outcome bias (decisions are based on their outcome and not the decision-making process)

• This is a consistent effect even when asked to focus on quality of decision making

• Experts are judged more harshly on this

• Facts that become known after the decision are applied to the decision-making quality

• Clairvoyance is considered a trait by many

The question that you will now want answered is how can we overcome this and how can we make better decisions in life and in business? This outcome bias may hinder better decision making but it will also hinder career progression amongst other factors.

• Make decisions with as much information as possible and apply good statistical analysis (more of this in later articles).

• Consider how the statistics applies to your own personal situation (for example, many statistics on personal health are from large population averages).

• When making a decision consider all outcomes and your ability to deal with this. So, consider a negative outcome and how sure you will still be that the decision you made was a good and justifiable decision.

• Build awareness of outcome bias – when making the decision and when justifying decisions.

A paper by Zhang in 2016 also came up with some interesting experimental data. In this experiment, study participants were asked to judge decisions from a person with fair intentions with a bad outcome, and a person with selfish intentions with a positive outcome.

• Counterintuitively judging decisions separately and not in comparison enabled better evaluation

• Third-party judgments were more balanced

• And raising the salience, importance and emotionality of the intention, also made for better evaluation

My examples at the start show how much outcome bias there is in sports – there are also multiple other decision-making biases particularly prevalent in sports.

However, in the background many of these professional teams have extremely good understanding of the risks, biases, and the role of luck and chance in outcomes. In the article on average high performers , I spoke about the case of Shane Battier who seemed to be singularly average, or below average, statistically. But the article I quoted in the New York Magazine, looks in more detail at the statistical analysis that is happening in the background in the NBA. And you then find that these are very refined. Similarly, Liverpool football club in the UK has an exceptionally good statistics department (a novelty at the time in football). A New York Times ran an article on their statistics shortly after coming an agonising second in the Premier league after only losing 1 game all season. But the next season would give them their first Premier league title in 30 years.

Footballers are as much today not just footballers but are fed all sorts of statistics to improve their performance which may be unperceived by the crowd. A defender may be standing off an attacker and forcing them on to their weak foot. This looks undramatic but on average decreases the chances of the attacker getting into a better position. The average football punter may be complaining: “get in there and tackle him”. Increasingly the sports world and fans are disconnected with the decisions made on and off the field.

Sports teams do this because they know on average this yields bigger benefits. This is certainly something business leaders and businesses can learn. Learn about how probabilities can increase chances of success and how to avoid outcome bias. If you can do this, you can make better decisions on average and in any sizeable business. In my mind, decision-making ability and awareness of outcome bias (we will focus on more decision biases in future articles) is up there in terms of skills for leaders. But beware of the leader that mitigates their own outcome bias but exaggerates those of others – they are the most dangerous.

---

References

Outcome Bias

Agrawal, N., and Maheswaran, D. (2005). Motivated reasoning in outcome-bias effects. J. Consum. Res. 31. doi:10.1086/426614.

Bachmann, K. (2018). Can advisors eliminate the outcome bias in judgements and outcome-based emotions? Rev. Behav. Financ. 10. doi:10.1108/RBF-11-2016-0072.

Brownback, A., and Kuhn, M. A. (2019). Understanding outcome bias. Games Econ. Behav. 117. doi:10.1016/j.geb.2019.07.003.

Gino, F., Moore, D. A., and Bazerman, M. H. (2011). No Harm, No Foul: The Outcome Bias in Ethical Judgments. SSRN Electron. J. doi:10.2139/ssrn.1099464.

Hugh, T. B., and Dekker, S. W. A. (2009). Hindsight bias and outcome bias in the social construction of medical negligence: a review. J. Law Med. 16.

Kausel, E. E., Ventura, S., and Rodríguez, A. (2019). Outcome bias in subjective ratings of performance: Evidence from the (football) field. J. Econ. Psychol. 75. doi:10.1016/j.joep.2018.12.006.

König-Kersting, C., Pollmann, M., Potters, J., and Trautmann, S. T. (2021). Good decision vs. good results: Outcome bias in the evaluation of financial agents. Theory Decis. 90. doi:10.1007/s11238-020-09773-1.

Lefgren, L., Platt, B., and Price, J. (2015). Sticking with what (Barely) worked: A test of outcome bias. Manage. Sci. 61. doi:10.1287/mnsc.2014.1966.

Lewis, J., and Simmons, J. P. (2019). Prospective Outcome Bias: Incurring (Unnecessary) Costs to Achieve Outcomes That Are Already Likely. J. Exp. Psychol. Gen. doi:10.1037/xge0000686.

Murata, A., Nakamura, T., Matsushita, Y., and Moriwaka, M. (2015). Outcome Bias in Decision Making on Punishment or Reward. Procedia Manuf. 3. doi:10.1016/j.promfg.2015.07.914.

Savani, K., and King, D. (2015). Perceiving outcomes as determined by external forces: The role of event construal in attenuating the outcome bias. Organ. Behav. Hum. Decis. Process. 130. doi:10.1016/j.obhdp.2015.05.002.

Seta, C. E., Seta, J. J., Petrocelli, J. V., and McCormick, M. (2015). Even better than the real thing: Alternative outcome bias affects decision judgements and decision regret. Think. Reason. 21. doi:10.1080/13546783.2015.1034779.

Sezer, O., Zhang, T., Gino, F., and Bazerman, M. H. (2016). Overcoming the outcome bias: Making intentions matter. Organ. Behav. Hum. Decis. Process. 137. doi:10.1016/j.obhdp.2016.07.001.

Strohmaier, N., Pluut, H., van den Bos, K., Adriaanse, J., and Vriesendorp, R. (2021). Hindsight bias and outcome bias in judging directors’ liability and the role of free will beliefs. J. Appl. Soc. Psychol. 51. doi:10.1111/jasp.12722.

Baron and Hershey’s Paper

https://www.sas.upenn.edu/~baron/papers/outcomebias.pdf

Statistics in Sport

Liverpool: https://www.nytimes.com/2019/05/22/magazine/soccer-data-liverpool.html

The Average Performers Who Enable High Performance: https://andyhab.medium.com/the-average-performers-who-enable-high-performance-c5e25fb7084?sk=5561fc514d9031477e1ca2c4509b4274

New York Times Magazine: https://www.nytimes.com/2009/02/15/magazine/15Battier-t.html