Regression To The Mean

Why Perfection Rarely Lasts

Whenever correlation is imperfect, and thus causation is unproven, over time the best score will get worse and the worst will get better


Towards Data Science

Regression to the mean shows that following an extreme random event, the next random event is likely to be less extreme.

The concept was first presented by Sir Francis Galton and the more comprehensive statement of regression to the mean is that:

In any series of events with complex phenomena that are dependent on many variables, and where chance is involved, extreme outcomes tend to be followed by more moderate ones.

It is not a mathematical law but a statistical tendency.



    "Regression to the mean is a common statistical phenomenon that can mislead us when we observe the world.

    Learning to recognize when regression to the mean is at play can help us avoid misinterpreting data and seeing patterns that don’t exist." [Shane Parrish]






Regression To The Mean - The Imperfect Correlation Of Factors/Causes and Chance

To be able to understand regression to the mean, we first need to understand correlation.

Correlation is a relationship, thus action A relates to outcome B.

The strength or weakness of the correlation of shared factors [or causes] will show the level of dependency of the outcome [or score] on those factors.

If the correlation is strong there will be little to no regression to the mean, whereas as a weak or non-existent correlation between shared factors will likely show that there is a regression.

Examples:

[1] Strong correlation of factors - TEMPERATURE

The only factor determining temperature – velocity of molecules — is shared by all scales, hence each degree in Celsius will have exactly one corresponding value in Fahrenheit.

So the temperature in Celsius and Fahrenheit will have a correlation coefficient of 1 and a plotting of temperature score will be a straight line.

[2] Weak to non-existent correlation of factors - BOTTLED WATER CONSUMPTION  and DEATHS FROM AIR CRASHES

If we looked at all countries in the world and plotted death rates from air crashes per country, in any specific year , against per capita consumption of bottled water, the plotting of scores for each event would show no pattern at all. Thus the correlation coefficient is 0, i.e. there is no correlation.



    Whenever the correlation between two outcomes or scores is weak to non existent, there will be regression to the mean.






Regression To The Mean - The Dangers Of False Attribution Of Cause & Effect

We need to be especially careful of regression to the mean when we are trying to establish causality between two factors.

As we have already established:

Correlation is a relationship, thus action A relates to outcome B.

Whereas:

# Causation can be defined as action A causes outcome B.

# Causation results from a perfect correlation between action A and outcome B.



  • Correlation and causation are often confused because the human mind likes to find patterns even when they do not exist.
  • We can not assume causation just because we see an apparent connection or link between A and B, such as we see that B follows A or we see A and B simultaneously.
  • We can only show a causation when we know how it causes B and why it causes B.
  • We need to be able to separate action A from all other variable factors
  • We must also be able to show how Action A can be replicated under the same conditions to achieve outcome B, AND be able to support this with supporting empirical evidence [thus following scientific method].




This is something that many people, social media, the general media, and sometimes even trained scientists, fail to understand or recognise.



    Whenever correlation is imperfect, and thus a causation is unproven, the best score or outcome will always appear to get worse and the worst will appear to get better over time, regardless of any additional treatment.





Regression To The Mean - Examples

Can you recall a wonderful evening out? The weather was fabulous, your table was great and restaurant was excellent, the food and wine were superb and you were with great people.

Then you tried to repeat the experience and somehow it wasn't as good and you were disappointed. Why was this?  It was because your perfect evening was due to a random series of chance events that all fell in place on the wonderful evening.

Whenever you try to repeat the perfect experience, chances are that at least one thing won't be perfect the next time.




Regression to the mean is prevalent in sport and can explain the “manager of the month curse” in football.

This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck.

When the luck runs out, the “curse” strikes.




Regression to the mean can be harmless, but it becomes a problem when it is misinterpreted:

"For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance."






Countering The Dangers Of False Attribution Of Cause & Effect

Here are 4 suggestions for identifying and countering false attribution of cause and effect and understanding the regression to the mean


  • Gather more data
  • Identify the variables
  • Wait and watch
  • Rinse and repeat






Return to: Mental Models






Chinese (Traditional)EnglishFrenchGermanItalianRussianSpanishVietnamese


Custom Site Search


The Balanced Toolkit




3 Keys Solutions

Think -> Action = Change


Get new posts by email:

LATEST ARTICLES

  1. The Power Pause - Regain Your Balance And Manifest Your Intentions

    Aug 02, 21 01:00 AM

    "The Three Minute Power Pause" is a very simple but powerful technique developed and taught by John Harricharan. It relieves stress and anxiety, helps you regain your balance and manifest your intenti…

    Read More

  2. Julia Galef - Applied Rationality

    Jul 30, 21 01:00 AM

    “Do you have the scout mindset and yearn to see the world as clearly as you possibly can?” Julia Galef is a writer and public speaker on the topics of rationality, science, technology, and design. App…

    Read More

  3. Stanford Prison Experiment - A Masterclass In Self Deception

    Jul 29, 21 04:14 AM

    "The military told me they used the SPE tapes to train people not to behave like our guards. I used to quote that as a good outcome of the research. Now I know they also used it to train interrogators…

    Read More

  4. Philip Zimbardo - How Good People Are Transformed into Perpetrators Of Evil

    Jul 28, 21 03:00 AM

    Philip Zimbardo has contributed greatly to our understanding of the situationist perspective as to why people can change and choose to exercise bad/evil behaviour in response to the pressures and dema…

    Read More

  5. Roxanne - I Will Not Share You With So Many Other Men

    Jul 28, 21 12:00 AM

    The Toxicity Of Addiction To Instagram Self-Validation Roxanne...
    You don't have to put on the red light
    Those days are over
    No more posting selfies on Instagram tonight
    You don't have to sell your bo…

    Read More

  6. How To Win Without Succeeding - Avoid Losing!

    Jul 23, 21 10:31 AM

    We Celebrate Success And Ignore Failure. The personal development and self-help world is steeped in case studies and analysis of success. Numerous studies have been undertaken on successful people to…

    Read More

  7. Hanlons Razor - Never Attribute To Malice That Which Can be Explained By Stupidity

    Jul 20, 21 12:00 AM

    Hanlons Razor - will check your innate tendency to quickly jump to the wrong conclusion and assume something is the result of an intentional malicious action. Applying Hanlon’s Razor will check your…

    Read More

  8. Keeping Things Simple - The Art of Knowing What To Overlook

    Jul 16, 21 12:00 AM

    Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius—and a lot of courage—to move in the opposite direction. The phrase "keeping things simple" is in…

    Read More

  9. Satisficing - When Near Enough Is Good Enough

    Jul 13, 21 12:00 AM

    Satisficing is a heuristic - a practical shortcut - for making decisions that aims for a satisfactory or adequate result, rather than the optimal solution. The opposite decision-making process to sat…

    Read More

  10. Skin In The Game - Invested With Shared Risk And Reward

    Jul 09, 21 12:00 AM

    In business, competent action is required as well as shared risk and reward. Only take advice from people who are already invested in the situation they are advising you about, and where they have som…

    Read More

  11. Black Swans - Impossible To Predict, Rare and Devastating

    Jul 05, 21 12:00 AM

    Black Swans are events that are difficult to predict, occur very occasionally and have potentially severe consequences. It is extremely foolish to just ignore the potential for black swans to occur. T…

    Read More

  12. Nassim Taleb - Focus On The Consequences And Not On The Probability

    Jun 30, 21 12:00 AM

    An investigation of opacity, luck, uncertainty, probability, human error, risk, and decision making when we don’t understand the world. Our world is full of chance and the unknown and the broader canv…

    Read More

  13. Inflection Points - The Tipping Point

    Jun 25, 21 12:00 AM

    The critical point in a situation, process, or system beyond which a significant and often unstoppable effect or change takes place Inflection points occur when there is build up of forces, a critical…

    Read More

  14. Root Cause Analysis

    Jun 22, 21 12:00 AM

    Establishing The Issue That Lies Beneath A Presenting Problem Often the true underlying root issue is not immediately apparent and is only uncovered with persistent investigation. The traditional appl…

    Read More

  15. "The Balanced Toolkit" Approach To Life

    Jun 17, 21 12:00 AM

    Introducing - The Balanced Toolkit The purpose of this site is to show you how to cope in tough times, and to provide you with the tools to do this successfully. The way we do this is by adopting a ba…

    Read More




Support This Site