Regression To The Mean

Why Perfection Rarely Lasts

Whenever correlation is imperfect, and thus causation is unproven, over time the best score will get worse and the worst will get better


Regression To The Mean. Why Perfection Rarely Lasts. Graphic.


Regression to the mean shows that following an extreme random event, the next random event is likely to be less extreme.

The concept was first presented by Sir Francis Galton and the more comprehensive statement of regression to the mean is that:

In any series of events with complex phenomena that are dependent on many variables, and where chance is involved, extreme outcomes tend to be followed by more moderate ones.

It is not a mathematical law but a statistical tendency.



    Regression to the mean is a common statistical phenomenon that can mislead us when we observe the world.

    Learning to recognize when regression to the mean is at play can help us avoid misinterpreting data and seeing patterns that don’t exist.

    [Shane Parrish]






Regression To The Mean - The Imperfect Correlation Of Factors/Causes and Chance

To be able to understand regression to the mean, we first need to understand correlation.

Correlation is a relationship, thus action A relates to outcome B.

The strength or weakness of the correlation of shared factors [or causes] will show the level of dependency of the outcome [or score] on those factors.

If the correlation is strong there will be little to no regression to the mean, whereas as a weak or non-existent correlation between shared factors will likely show that there is a regression.

Examples:

[1] Strong correlation of factors - TEMPERATURE

The only factor determining temperature – velocity of molecules — is shared by all scales, hence each degree in Celsius will have exactly one corresponding value in Fahrenheit.

So the temperature in Celsius and Fahrenheit will have a correlation coefficient of 1 and a plotting of temperature score will be a straight line.

[2] Weak to non-existent correlation of factors - BOTTLED WATER CONSUMPTION  and DEATHS FROM AIR CRASHES

If we looked at all countries in the world and plotted death rates from air crashes per country, in any specific year , against per capita consumption of bottled water, the plotting of scores for each event would show no pattern at all. Thus the correlation coefficient is 0, i.e. there is no correlation.



    Whenever the correlation between two outcomes or scores is weak to non existent, there will be regression to the mean.






Regression To The Mean - The Dangers Of False Attribution Of Cause & Effect

We need to be especially careful of regression to the mean when we are trying to establish causality between two factors.

As we have already established:

Correlation is a relationship, thus action A relates to outcome B.

Whereas:

# Causation can be defined as action A causes outcome B.

# Causation results from a perfect correlation between action A and outcome B.



  • Correlation and causation are often confused because the human mind likes to find patterns even when they do not exist.
  • We can not assume causation just because we see an apparent connection or link between A and B, such as we see that B follows A or we see A and B simultaneously.
  • We can only show a causation when we know how it causes B and why it causes B.
  • We need to be able to separate action A from all other variable factors
  • We must also be able to show how Action A can be replicated under the same conditions to achieve outcome B, AND be able to support this with supporting empirical evidence [thus following scientific method].




This is something that many people, social media, the general media, and sometimes even trained scientists, fail to understand or recognise.



    Whenever correlation is imperfect, and thus a causation is unproven, the best score or outcome will always appear to get worse and the worst will appear to get better over time, regardless of any additional treatment.





Regression To The Mean - Examples

Can you recall a wonderful evening out? The weather was fabulous, your table was great and restaurant was excellent, the food and wine were superb and you were with great people.

Then you tried to repeat the experience and somehow it wasn't as good and you were disappointed. Why was this?  It was because your perfect evening was due to a random series of chance events that all fell in place on the wonderful evening.

Whenever you try to repeat the perfect experience, chances are that at least one thing won't be perfect the next time.




Regression to the mean is prevalent in sport and can explain the “manager of the month curse” in football.

This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck.

When the luck runs out, the “curse” strikes.




Regression to the mean can be harmless, but it becomes a problem when it is misinterpreted:

"For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance."






Countering The Dangers Of False Attribution Of Cause & Effect

Here are 4 suggestions for identifying and countering false attribution of cause and effect and understanding the regression to the mean


  • Gather more data
  • Identify the variables
  • Wait and watch
  • Rinse and repeat






Return to: Mental Models



English Chinese (Traditional) Russian French German Italian Spanish Vietnamese


LATEST ARTICLES

  1. How To Influence Without Authority - 6 Key Tips

    The secret to how to influence without authority is that you get what you really want by giving other people what they really want. We live in an interconnected world and knowing how to influence with…

    Read More

  2. Change Questions To Change Your Outcomes

    Asking The Right Questions Is Critical For A Successful Change. Every time we initiate a significant change - whether in our personal life or in an organisation - we will most likely over-estimate our…

    Read More

  3. Group Culture - The Invisible Software That Rules Your Life

    Group culture is: "How we do things round here". We like to see ourselves as free agents making our own choices and living authentically but the reality is that The Matrix has many layers and we are u…

    Read More

  4. Why Getting From A to B Is Not Aways A Straight Line

    In circumstances of significant change, the progress from A to B will not be in a straight line. We run our lives largely on auto-pilot. In most circumstances your experience of getting from A to B is…

    Read More

  5. The Art Of Persuasion Planning For Success - Here's How To Do It!

    To be successful in the art of persuasion you must ensure that certain things happen. To be successful in the art of persuasion you must establish a framework of what has to happen to get you to that…

    Read More

  6. The Art Of Persuasion Advanced Communication Skills - Gaining Buyin

    Create The Environment Where They Want To Buyin to Your Proposal In order to build the win-win you have to uncover what it is that the other person really wants or needs, and to do that you have to as…

    Read More

  7. The Art Of Persuasion The One Fundamental Principle - Create A Win-Win

    The art of persuasion is based on the simple idea that you get what you want by enabling the other party to get what they want. Being a nice friendly person with good inter-personal skills may be a go…

    Read More

  8. Communication Persuasion And Change - Key Skills To Survive & Succeed

    It's not the strongest that survive, nor the most intelligent, but those who are most responsive to change, the most persuasive, and the best communicators. We are living in an age of unprecedented ch…

    Read More

  9. The Eisenhower Box - What Is Important Is Seldom Urgent

    What Is Important Is Seldom Urgent And What Is Urgent Is Seldom Important. The Eisenhower Box is a time management and decision-making model devised by President Dwight Eisenhower to help him prioriti…

    Read More

  10. Zen Enlightenment [Satori] - The Stink Of Zen

    Lost In Our Delusions About Enlightenment. There is something in human nature - a desire to glamorise, sanctify, objectify and idolise – that elevates people who have offered deep insights to the huma…

    Read More

  11. 5 Zen Mindsets For Mastery - In Any Area Of Your Life

    The Wisdom Of A Person Who Masters In Any Art Is Reflected In Their Every Attitude. The state and quality of your mind has a very large bearing on the quality of your performance in any area of life t…

    Read More

  12. Dealing With The Toxicity Of Online Dating - 6 Key Tips From A Clinical Psychologist

    Toxicity Is The Price Tag Of Accessibility. In the early days of online dating, users were vetted and had to go through a registration process and agree to comply with a code of conduct designed to en…

    Read More

  13. Why Understanding Ergodicity Is Critical To Your Long Term Survival

    How Not To Be Fooled By Randomness. Ergodicity is an ugly word from the world of mathematics. It is an umbrella term for two sets of conditions of probability and outcome. These two conditions form th…

    Read More

  14. Dealing With Imposter Syndrome - Ego Is The Enemy

    How You Frame A Situation Has A Profound Impact On How You Respond To It Emotionally. Imposter syndrome is a psycho-emotional experience of a fear of being found out as incompetent despite ongoing evi…

    Read More

  15. The Challenges Of The Road Less Traveled

    Issues You'll Face When Playing The Long Game. The challenges of the road less traveled is loosely based around the phrase popularised by M.Scott Peck with his book "The Road Less Traveled". This arti…

    Read More

  16. How To Benefit From the Unseen Margins - 5 Key Tips For Success

    These Unseen Margins Can Have A Very Dramatic Impact On Your Life. To understand how to benefit from the unseen margins we need to start by understanding what they are and where we find them. In this…

    Read More

  17. The Art Of Thinking Clearly - How To Do More Than Just Survive And Reproduce

    3 Key Tips The art of thinking clearly starts with the sobering realisation that our brains are designed to achieve two things: Survival and Reproduction! 98% of our thinking is unconscious, automatic…

    Read More

  18. Algorithms to Live By - 5 Useful Rules Of Thumb

    5 Useful Heuristics From Algorithms To Live By The thesis of the book "Algorithms to Live By" is that algorithms developed for computers can be used by people in everyday life in a wide range of situa…

    Read More

  19. Finding Signal In The Noise - How To Avoid The Noise Bottleneck

    The Art Of Being Wise Is The Art Of Knowing What To Overlook. We are blessed and cursed to live in the digital age. We have access to more information than we can possibly handle yet we struggle to fi…

    Read More

  20. The Checklist Manifesto - Your Personal Safety Net

    A Checklist Is A Safety Net That Encourages Better Results And Prevents Avoidable Mistakes. The volume and complexity of what we know has exceeded our individual ability to deliver its benefits correc…

    Read More

  21. How Not To Be Stupid - 4 Key Tips

    How To Avoid The 7 Causes Of Everyday Stupidity. We are all capable of everyday stupidity as we undertake routine tasks in our business and working lives and also in our personal lives. This is not ab…

    Read More

  22. Beginners Mind And The Voice Of Experience

    Only The Experts Survived Evolution. "In the beginner's mind there are many possibilities, but in the expert's there are few." These are the famous words of Shunryu Suzuki in "Zen Mind, Beginner's Min…

    Read More

  23. The Art Of Being Alone Is A Skill

    Let's Make Today A Good Day. You may not have chosen the condition of being alone, and it may have been imposed upon you by circumstances beyond your control, but your response to the situation is wit…

    Read More

  24. Intuition & Anxiety - Are There Angels Or Devils Calling Here?

    How To Tell The Difference Between Intuition and Anxiety. How do you know whether the voice of your intuition is real or just the product of your inner anxiety? We all struggle with these inner voices…

    Read More

  25. Like A Prayer - Life Is A Mystery

    It Isn't The Process Of Prayer That’s The Problem, It’s The Way It’s Framed. Regardless of what we feel about Madonna or her song the topic of prayer often arouses strong reactions. Usually, it is som…

    Read More

  26. Fear Of Missing Out - "I'll Have What She's Having!"

    We Follow The Herd - We Mimic Other People's Choices. Fear of missing out - or FOMO as it is popularly referred to - is the feeling that everyone else in your peer group is having much more fun than y…

    Read More

  27. Free - Self Improvement Resources

    Exercising Balance and Discernment. I have just updated these self improvement resources with a number of additional sources of material that are practical and can help you change your life. Check it…

    Read More

  28. Your Higher Self - Your Hardwired Portal To The Universe

    This Is The You That Is Beyond Your Thinking Mind. This is the big you, the transcendent you, the you that is often referred to as your higher consciousness or higher self. This is the you that acts a…

    Read More

  29. The Power Of Gratitude - It's Good For You!

    Gratitude And Attitude Are Not Challenges, They Are Choices. The power of gratitude quite simply is that it is good for you! Many of us were raised by parents who instilled in us the social niceties o…

    Read More

  30. Living With Your Thoughts - 4 Tools To Help You

    Develop A Good Working Relationship With Your Mind. The first step in living with your thoughts is to understand and develop a good working relationship with all areas of your mind. The key to develop…

    Read More



Get new posts by email:









Zen-Tools.Net





Support This Site