Regression To The Mean

Why Perfection Rarely Lasts

Whenever correlation is imperfect, and thus causation is unproven, over time the best score will get worse and the worst will get better


Regression To The Mean. Why Perfection Rarely Lasts. Graphic.


Regression to the mean shows that following an extreme random event, the next random event is likely to be less extreme.

The concept was first presented by Sir Francis Galton and the more comprehensive statement of regression to the mean is that:

In any series of events with complex phenomena that are dependent on many variables, and where chance is involved, extreme outcomes tend to be followed by more moderate ones.

It is not a mathematical law but a statistical tendency.



    Regression to the mean is a common statistical phenomenon that can mislead us when we observe the world.

    Learning to recognize when regression to the mean is at play can help us avoid misinterpreting data and seeing patterns that don’t exist.

    [Shane Parrish]






Regression To The Mean - The Imperfect Correlation Of Factors/Causes and Chance

To be able to understand regression to the mean, we first need to understand correlation.

Correlation is a relationship, thus action A relates to outcome B.

The strength or weakness of the correlation of shared factors [or causes] will show the level of dependency of the outcome [or score] on those factors.

If the correlation is strong there will be little to no regression to the mean, whereas as a weak or non-existent correlation between shared factors will likely show that there is a regression.

Examples:

[1] Strong correlation of factors - TEMPERATURE

The only factor determining temperature – velocity of molecules — is shared by all scales, hence each degree in Celsius will have exactly one corresponding value in Fahrenheit.

So the temperature in Celsius and Fahrenheit will have a correlation coefficient of 1 and a plotting of temperature score will be a straight line.

[2] Weak to non-existent correlation of factors - BOTTLED WATER CONSUMPTION  and DEATHS FROM AIR CRASHES

If we looked at all countries in the world and plotted death rates from air crashes per country, in any specific year , against per capita consumption of bottled water, the plotting of scores for each event would show no pattern at all. Thus the correlation coefficient is 0, i.e. there is no correlation.



    Whenever the correlation between two outcomes or scores is weak to non existent, there will be regression to the mean.






Regression To The Mean - The Dangers Of False Attribution Of Cause & Effect

We need to be especially careful of regression to the mean when we are trying to establish causality between two factors.

As we have already established:

Correlation is a relationship, thus action A relates to outcome B.

Whereas:

# Causation can be defined as action A causes outcome B.

# Causation results from a perfect correlation between action A and outcome B.



  • Correlation and causation are often confused because the human mind likes to find patterns even when they do not exist.
  • We can not assume causation just because we see an apparent connection or link between A and B, such as we see that B follows A or we see A and B simultaneously.
  • We can only show a causation when we know how it causes B and why it causes B.
  • We need to be able to separate action A from all other variable factors
  • We must also be able to show how Action A can be replicated under the same conditions to achieve outcome B, AND be able to support this with supporting empirical evidence [thus following scientific method].




This is something that many people, social media, the general media, and sometimes even trained scientists, fail to understand or recognise.



    Whenever correlation is imperfect, and thus a causation is unproven, the best score or outcome will always appear to get worse and the worst will appear to get better over time, regardless of any additional treatment.





Regression To The Mean - Examples

Can you recall a wonderful evening out? The weather was fabulous, your table was great and restaurant was excellent, the food and wine were superb and you were with great people.

Then you tried to repeat the experience and somehow it wasn't as good and you were disappointed. Why was this?  It was because your perfect evening was due to a random series of chance events that all fell in place on the wonderful evening.

Whenever you try to repeat the perfect experience, chances are that at least one thing won't be perfect the next time.




Regression to the mean is prevalent in sport and can explain the “manager of the month curse” in football.

This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck.

When the luck runs out, the “curse” strikes.




Regression to the mean can be harmless, but it becomes a problem when it is misinterpreted:

"For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance."






Countering The Dangers Of False Attribution Of Cause & Effect

Here are 4 suggestions for identifying and countering false attribution of cause and effect and understanding the regression to the mean


  • Gather more data
  • Identify the variables
  • Wait and watch
  • Rinse and repeat






Return to: Mental Models



English Chinese (Traditional) Russian French German Italian Spanish Vietnamese



LATEST ARTICLES

  1. Coram Deo - Living In Consciousness

    In you there is a dimension of consciousness far deeper than thought. It is the very essence of who you are. Coram Deo is about living in consciousness. It is a Latin phrase which literally means “to…

    Read More

  2. The Power Of Patience - Why You Need The World's Toughest Quality

    Nothing in the world can take the place of patience. Patience and persistence are omnipotent. In everyday life, patience is often overshadowed by the desire for immediate results. We live in an era of…

    Read More

  3. Demonizing The Other and Personal Acts Of Compassion

    What Does Demonizing The Other Mean? Demonizing the other refers to the act of portraying a group of people or an individual as inherently evil, threatening, or inferior. It often serves to justify di…

    Read More

  4. Why You Should Embrace Anomalies - The Incredible Value Of Disconfirming Evidence

    Is Your Desire To Be Right Greater Than Your Desire To Have Been Right? An anomaly is a deviation from what is expected or commonly regarded as the norm. It often appears as an unexpected observation…

    Read More

  5. Amazing Grace - The Majesty And The Mercy of Freedom From Your Pain

    "I once was lost, but now I am found, was blind, but now I see." The hymn and popular song "Amazing Grace" was written 250 years ago by John Newton, a former slave trader who in 1748 nearly died in a…

    Read More

  6. The Transformative Power Of Acceptance

    Experience The Power Of Acceptance. This website contains about 500,000 words. You could read every single word and it wouldn't make any real difference to you. You might become better informed, but t…

    Read More

  7. Inversion - The Power Of Opposite Thinking

    How To Avoid The Unwanted Outcome. The power of opposite thinking, also known as inversion, lies in its ability to stimulate creativity, enhance problem-solving, and provide a fresh perspective on cha…

    Read More

  8. Are You Aligned With Reality? Or Do You See What You Believe?

    We tend to see that which aligns with what we believe, and to act upon that rather than acting on reality. Being aligned with reality starts with a clear and accurate understanding of the world. It me…

    Read More

  9. The Law Of Response and Outcome

    A New Approach To A New Life At time of writing we are entering a new year which is traditionally a time of making resolutions to change our behaviour and improve the quality of our lives. And yet pow…

    Read More

  10. Clear Thinking - Turning Ordinary Moments Into Extraordinary Results

    There are two ways to handle the world - try to predict, or try to prepare. "Clear Thinking" by Shane Parrish, published in Oct 2023, is a laudable testament to the art of cogent thinking, and will be…

    Read More

  11. Self Dialogue - Working With Your Many Selves

    Self Dialogue Is About Working With ALL Levels Of Your Mind. The key to effective self dialogue is to have tools, techniques and resources that work with all levels of your mind. This simple self-faci…

    Read More

  12. The Balanced Brain - The Ultimate Route To Personal Transformation

    How To Experience The Benefits Of A Balanced Brain. Underpinning all of the belief systems and practices that offer routes to personal change, transformation and spiritual growth is the balanced brain…

    Read More

  13. How To Make Better Decisions - By Avoiding The Narrative Trap

    To Understand The Truth We Have To See The Whole Picture. One of the best ways to make better decisions is to have a deeper understanding of the many things that might stop that happening. To understa…

    Read More

  14. The Greatest Love - The Most Important Relationship You Will Ever Have

    Yet Most Will Never Know It Everyone of us has a place, in our hearts there's a space, that is home to the greatest love of all. This love transcends everything we think we know about the world of for…

    Read More

  15. Everything Is Connected And Why You Don't Feel It

    ...And Why It Matters As human beings we are skating on very thin ice with our sense of self and certainty about "how things are" and what we like to think of as reality:

    - What if everything we think…

    Read More

  16. Who Is In Charge Of Your Brain?

    How Not To Be Stupid. Who is in charge of your brain? This is not a silly questions. It matters because the outcomes that you experience in your life are determined by how you respond to the events th…

    Read More

  17. How To Be A Winner On A Very Large Scale

    The Incredible Benefits Of Selective Attention. This is not a typical article about how to be a winner. We are not going to talk about goal setting, the importance of habits, the power of focus and al…

    Read More

  18. The Metagame Approach To Life

    How To Achieve Your Biggest Objectives. The metagame approach to life is all about winning and achieving your biggest objectives by: - Understanding the bigger picture
    - Being better by doing things d…

    Read More

  19. Shantideva - The Way Of The Bodhisattva

    Walking The Path Of Compassion. Shantideva the 8th century Indian Buddhist sage is famous for his treatise "The Way of the Bodhisattva" delivered as an extended teaching to the monks of Nalanda monast…

    Read More

  20. Reframing History - Deconstruction And Discussion Not Destruction

    History is always about context, not imposing our own moral values on the past. For those of us fortunate enough to live within western democracies, we are living in an age where a vociferous and into…

    Read More

  21. Tao Te Ching - Connecting To Your True Source Of Power.

    How To Be Lived By The Tao. The Tao Te Ching is one of those books that many people read, few understand, and even fewer put into practice. The only way to know the Tao is to experience it, and it is…

    Read More

  22. How Things Really Are - The Inbuilt Design Flaws

    Chaos, Disorder And Decay Is The Natural Order Of Things. Nobody has the perfect life. We all struggle and strive to attain health, wealth and personal happiness. Yet these three big areas: our health…

    Read More

  23. Intuition Or Anxiety - Are There Angels Or Devils Crawling Here?

    How To Tell The Difference Between Intuition and Anxiety. How do you know whether the voice of your intuition is real or just the product of your inner anxiety? Several months ago I was having a drink…

    Read More

  24. What Is Truth - How To Tell A Partial Truth From The Whole Truth?

    How the truth and nothing but the truth is often not the whole truth. My great aunty Flo broke her arm and died. It is true that she broke her arm in 1923. It is also true that she died in 1949. But t…

    Read More

  25. Duality And Life Beyond Your Thinking Mind

    Duality and life beyond your thinking mind focuses on the limitations of time, foreground and background, duality and "stuckness". The first aspect of duality and life beyond your thinking mind focuse…

    Read More

  26. The Conscious Mind Is Limited - Be Aware And Be Prepared

    Being aware is the first stage of being prepared. The conscious mind is limited in so many ways. There are some who would argue that there is no such thing as conscious thought and that it is represen…

    Read More

  27. Your Inner Map Of Reality - Here's Why You Think The Way You Do

    The big picture of how your inner map of reality creates your feelings, thoughts, and behaviours. Your inner map of reality is based on the filters of your own ethnic, national, social, family and rel…

    Read More

  28. The Failure Of Cancel Culture - Suppression Not Engagement

    Why we need to wear our beliefs lightly and develop negative capability. Throughout history people have campaigned to fight beliefs, ideologies, and injustices that they perceived to be oppressive, di…

    Read More

  29. 4 Big Reasons Why We Get Stuck In Our Attempts At Personal Change

    Most People Spend Their Entire Life Imprisoned Within The Confines Of Their Own Thoughts. This first of the 4 big reasons why we get stuck is, in my view, the most important. The "self-help industry…

    Read More

  30. How Do I Change And Why Is It So Hard?

    We Would Rather Die Than Change, And We Usually Do In my experience, the vast majority of people who say they want to change don’t change. Most people reading this won’t change because they don’t real…

    Read More



Get new posts by email:









Zen-Tools.Net





Support This Site