Regression To The Mean

Why Perfection Rarely Lasts

Whenever correlation is imperfect, and thus causation is unproven, over time the best score will get worse and the worst will get better


Towards Data Science

Regression to the mean shows that following an extreme random event, the next random event is likely to be less extreme.

The concept was first presented by Sir Francis Galton and the more comprehensive statement of regression to the mean is that:

In any series of events with complex phenomena that are dependent on many variables, and where chance is involved, extreme outcomes tend to be followed by more moderate ones.

It is not a mathematical law but a statistical tendency.



    Regression to the mean is a common statistical phenomenon that can mislead us when we observe the world.

    Learning to recognize when regression to the mean is at play can help us avoid misinterpreting data and seeing patterns that don’t exist.

    [Shane Parrish]






Regression To The Mean - The Imperfect Correlation Of Factors/Causes and Chance

To be able to understand regression to the mean, we first need to understand correlation.

Correlation is a relationship, thus action A relates to outcome B.

The strength or weakness of the correlation of shared factors [or causes] will show the level of dependency of the outcome [or score] on those factors.

If the correlation is strong there will be little to no regression to the mean, whereas as a weak or non-existent correlation between shared factors will likely show that there is a regression.

Examples:

[1] Strong correlation of factors - TEMPERATURE

The only factor determining temperature – velocity of molecules — is shared by all scales, hence each degree in Celsius will have exactly one corresponding value in Fahrenheit.

So the temperature in Celsius and Fahrenheit will have a correlation coefficient of 1 and a plotting of temperature score will be a straight line.

[2] Weak to non-existent correlation of factors - BOTTLED WATER CONSUMPTION  and DEATHS FROM AIR CRASHES

If we looked at all countries in the world and plotted death rates from air crashes per country, in any specific year , against per capita consumption of bottled water, the plotting of scores for each event would show no pattern at all. Thus the correlation coefficient is 0, i.e. there is no correlation.



    Whenever the correlation between two outcomes or scores is weak to non existent, there will be regression to the mean.






Regression To The Mean - The Dangers Of False Attribution Of Cause & Effect

We need to be especially careful of regression to the mean when we are trying to establish causality between two factors.

As we have already established:

Correlation is a relationship, thus action A relates to outcome B.

Whereas:

# Causation can be defined as action A causes outcome B.

# Causation results from a perfect correlation between action A and outcome B.



  • Correlation and causation are often confused because the human mind likes to find patterns even when they do not exist.
  • We can not assume causation just because we see an apparent connection or link between A and B, such as we see that B follows A or we see A and B simultaneously.
  • We can only show a causation when we know how it causes B and why it causes B.
  • We need to be able to separate action A from all other variable factors
  • We must also be able to show how Action A can be replicated under the same conditions to achieve outcome B, AND be able to support this with supporting empirical evidence [thus following scientific method].




This is something that many people, social media, the general media, and sometimes even trained scientists, fail to understand or recognise.



    Whenever correlation is imperfect, and thus a causation is unproven, the best score or outcome will always appear to get worse and the worst will appear to get better over time, regardless of any additional treatment.





Regression To The Mean - Examples

Can you recall a wonderful evening out? The weather was fabulous, your table was great and restaurant was excellent, the food and wine were superb and you were with great people.

Then you tried to repeat the experience and somehow it wasn't as good and you were disappointed. Why was this?  It was because your perfect evening was due to a random series of chance events that all fell in place on the wonderful evening.

Whenever you try to repeat the perfect experience, chances are that at least one thing won't be perfect the next time.




Regression to the mean is prevalent in sport and can explain the “manager of the month curse” in football.

This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck.

When the luck runs out, the “curse” strikes.




Regression to the mean can be harmless, but it becomes a problem when it is misinterpreted:

"For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance."






Countering The Dangers Of False Attribution Of Cause & Effect

Here are 4 suggestions for identifying and countering false attribution of cause and effect and understanding the regression to the mean


  • Gather more data
  • Identify the variables
  • Wait and watch
  • Rinse and repeat






Return to: Mental Models



Zen-Tools.Net



English Chinese (Traditional) Russian French German Italian Spanish Vietnamese


LATEST ARTICLES

  1. Your Relationship With Failure Is Critical To Your Success

    Just Because You Have A Spectacular Failure Doesn’t Mean The Game Is Over. The thought that you could fail can really slow you down. But the problem is not the failure, the problem is your relationshi…

    Read More

  2. Lost, Stuck & Going Round In Circles?

    Are You Lost In Transition? To be in an alien North African city, late in the evening, in the dark, completely lost in transition from the main market square back to the obscure location of our Riad…

    Read More

  3. Seeing Is Believing Or Is It?

    Do You Believe What You See Or Do You See What You Believe? Like the characters in "The Matrix" we are fooled by the illusion that what we see is how things really are. Why? Because this is the mind's…

    Read More

  4. What Difference Did You Make?

    What Impact Will You Leave In The Lives Of Others? If you knew you were going to die tonight and you were given a few minutes to reflect before that happened, what would you say was the meaning of yo…

    Read More

  5. Compassion Is Selfish And You Need It

    If You Want Others To Be Happy, Practice Compassion. If You Want To Be Happy, Practice Compassion. Compassion is derived from the Latin words "com pati" meaning to suffer with. It has been defined as…

    Read More

  6. The Placebo Effect - 2 Key Takeaways

    The Power Of Expectation Of An Outcome Regardless Of The Lack Of Any Substance To That Expectation. Traditionally, the placebo effect occurs when false, or dummy, treatments of inert substances [that…

    Read More

  7. The Red Pill

    This Is What Will Happen After You Take It. "This is your last chance," rebel leader Morpheus tells Neo, holding a coloured pill in each hand. “You take the blue pill, the story ends, you wake up in y…

    Read More

  8. Transformational Life Hacks

    This Has To Be One Of The Best Ways For Living The Best Life You Can Possibly Live. An online search on life hacks will invariably provide you with multiple links to ways of increasing your productivi…

    Read More

  9. This Too Shall Pass

    The Blessing And The Curse Of Impermanence. There is a story of a king who sought wisdom. Eventually he heard of a man living in a remote part of the kingdom who had a reputation for great wisdom. The…

    Read More

  10. The Stories We Tell Ourselves

    The Stories That We Tell Ourselves Are A Matter Of Choice. The stories we tell ourselves form the backcloth and the tapestry of our lives. These narratives shape and define our sense of self and how a…

    Read More

  11. Who Do You Trust?

    Trust Is A Choice And It Creates Your Reality. Nothing in life is permanent and nothing lasts forever. Every relationship ends in tears. Somebody leaves or somebody dies. This is how things are. This…

    Read More

  12. How Positive Asymmetry Can Transform Your Life

    Finding This Hidden Value Is The Key To A Huge Upside. Most of us spend our lives working to linear rules where there is an equal relationship between effort and result. Work hard you get a result. Wo…

    Read More

  13. 5 Tips For Developing A Growth Mindset

    How To Be There When Preparation Meets Opportunity. The growth mindset holds the core belief that your intelligence and personality is something you can develop and that you can expand and grow. The p…

    Read More

  14. Lost For Words? Experience Truth

    If you can't think of the right thing to say, say nothing. The concepts we employ, the categorizations we apply and the words we choose and use to articulate a direct experience put us in a double bi…

    Read More

  15. Belief - A Terrible Tyrant Or An Empowering Servant

    To Believe Or Not To Believe - That Is The Question. Nothing has inflicted more suffering on humanity than its beliefs and dogmas. The religious and cultural beliefs that sanction and allow or impose…

    Read More

  16. How To Avoid Herd Mentality

    How To Stay Far From The Madding Crowd. Herd mentality describes how we make decisions based on what everyone else is thinking and doing. It is also known as a mob or crowd mentality. We follow the he…

    Read More

  17. Are You Trying To Teach A Cat To Bark?

    We Suffer Every Time We Believe A Thought That Argues With What Is. When your mind is perfectly clear, what is, is what you want. If you want reality to be different than it is, you might as well try…

    Read More

  18. The Value Of Slack

    Slack Is The Capacity To Effect Change And The Resource To Deal With The Unexpected. Many organizations are obsessed with efficiency and try to ensure that every resource is fully utilized to its full…

    Read More

  19. 6 Tips To Help You Stop Trying Too Hard

    Less and less do you need to force things, until you arrive at non-action. We are brought up to believe that we get what we want in life as a result of what we do, by the actions that we take, that ha…

    Read More

  20. Your Point Of Focus

    Is It Helping You Or Hindering You? Your point of focus is the central point of your attention. Your point of focus is a binary choice and the result of that choice is critical to your happiness, your…

    Read More

  21. Your Secret For Success

    To accomplish any success in life, you must identify accurately what it is that you hunger for, thirst for and long for. The purpose of this article is to look at the foundation of the secret for succ…

    Read More

  22. Can You Stop Thinking?

    Discover The Joy Of A Quiet Mind Can you stop thinking? According to Zen Buddhist priest Shunmyo Masuno the art of not thinking about anything is not easy to achieve. Even practising monks find it dif…

    Read More

  23. How To Build Self Confidence

    What Is Holding You Back From Pursuing Your Dreams? Your self confidence is based on ability. Your self confidence is all about your level of trust, faith and belief in your abilities [your power] and…

    Read More

  24. How To Overcome Fear And Perform Powerfully

    Learning how to overcome fear and perform powerfully is a skill that we all need to acquire as fear of failure affects most of us at some point in our lives. Whilst it is natural to feel a level of fe…

    Read More

  25. Dance Into The Light And Stop Chasing Your Tail

    You are creating hell on earth when you are too subsumed within your underlying and deeply embedded negative energetic states. To counter your dissatisfactions and misery you initially go unconscious…

    Read More

  26. Change Your Language To Change How You Feel

    How you mentally characterize a situation has a profound impact on how you respond to it emotionally. There is a direct causal link between how you think about a situation, the words you use to articu…

    Read More

  27. How To Live In A Participatory Universe

    What good is a universe without somebody around to look at it? What Conclusions Can We Draw About Living In A Participatory Universe? The inescapable conclusion is that the world and the universe (wit…

    Read More

  28. 4 Keys To Working With The Energy Of Life

    The journey home to the energy of life does not consist in arriving at a new destination it is a coming home to one’s self. [1] How are you feeling, what is your dominant emotion right now? [2] Where…

    Read More

  29. How To Deal With The Challenges Of Playing The Long Game

    You have to be willing to suffer today in order to reap enormous rewards tomorrow. The long game is so hard to play because it involves taking a series of very small steps, that people rarely see, per…

    Read More

  30. If You Don't Take Care Of You, Who Will?

    How To Build Self Esteem In my early life and until my early twenties I suffered from a deep inferiority complex and low self esteem. Forty years later and, said in sincerity and with humility, I have…

    Read More





The Balanced Life



The Balanced Toolkit



3 Keys Solutions


Get new posts by email:



Support This Site