Regression To The Mean
Why Perfection Rarely Lasts
Whenever correlation is imperfect, and thus causation is unproven, over time the best score will get worse and the worst will get better
Regression to the mean shows that following an extreme random event, the next random event is likely to be less extreme.
The concept was first presented by Sir Francis Galton and the more comprehensive statement of regression to the mean is that:
In any series of events with complex phenomena that are dependent on many variables, and where chance is involved, extreme outcomes tend to be followed by more moderate ones.
It is not a mathematical law but a statistical tendency.
Regression To The Mean - The Imperfect Correlation Of Factors/Causes and Chance
To be able to understand regression to the mean, we first need to understand correlation.
Correlation is a relationship, thus action A relates to outcome B.
The strength or weakness of the correlation of shared factors [or causes] will show the level of dependency of the outcome [or score] on those factors.
If the correlation is strong there will be little to no regression to the mean, whereas as a weak or non-existent correlation between shared factors will likely show that there is a regression.
Examples:
[1] Strong correlation of factors - TEMPERATURE
The only factor determining temperature – velocity of molecules — is shared by all scales, hence each degree in Celsius will have exactly one corresponding value in Fahrenheit.
So the temperature in Celsius and Fahrenheit will have a correlation coefficient of 1 and a plotting of temperature score will be a straight line.
[2] Weak to non-existent correlation of factors - BOTTLED WATER CONSUMPTION and DEATHS FROM AIR CRASHES
If we looked at all
countries in the world and plotted death rates from air crashes per country, in any specific year ,
against per capita consumption of bottled water, the plotting of scores for each event would show no
pattern at all. Thus the correlation coefficient is 0, i.e. there is no correlation.
We need to be especially careful of regression to the mean when we are trying to establish causality between two factors. As we have already established: Correlation is a relationship, thus action A relates to outcome B. Whereas: # Causation can be defined as action A causes outcome B. # Causation results from a perfect correlation between action A and outcome B.
This is something that many people, social media, the general media, and
sometimes even trained scientists, fail to understand or recognise.
Regression To The Mean - The Dangers Of False Attribution Of Cause & Effect
Regression To The Mean - Examples
Can you recall a wonderful evening out? The weather was fabulous, your table was great and restaurant was excellent, the food and wine were superb and you were with great people.
Then you tried to repeat the experience and somehow it wasn't as good and you were disappointed. Why was this? It was because your perfect evening was due to a random series of chance events that all fell in place on the wonderful evening.
Whenever you try to repeat the perfect experience, chances are that at least one thing won't be perfect the next time.
Regression to the mean is prevalent in sport and can explain the “manager of the month curse” in football.
This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck.
When the luck runs out, the “curse” strikes.
Regression to the mean can be harmless, but it becomes a problem when it is misinterpreted:
"For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.
You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.
Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance."
Countering The Dangers Of False Attribution Of Cause & Effect
Here are 4 suggestions for identifying and countering false attribution of cause and effect and understanding the regression to the mean
- Gather more data
- Identify the variables
- Wait and watch
- Rinse and repeat
Return to: Mental Models
LATEST ARTICLES
The Challenges Of The Road Less Traveled
Issues You'll Face When Playing The Long Game. The challenges of the road less traveled is loosely based around the phrase popularised by M.Scott Peck with his book "The Road Less Traveled". This arti…How To Benefit From the Unseen Margins - 5 Key Tips For Success
These Unseen Margins Can Have A Very Dramatic Impact On Your Life. To understand how to benefit from the unseen margins we need to start by understanding what they are and where we find them. In this…The Art Of Thinking Clearly - How To Do More Than Just Survive And Reproduce
3 Key Tips The art of thinking clearly starts with the sobering realisation that our brains are designed to achieve two things: Survival and Reproduction! 98% of our thinking is unconscious, automatic…Algorithms to Live By - 5 Useful Rules Of Thumb
5 Useful Heuristics From Algorithms To Live By The thesis of the book "Algorithms to Live By" is that algorithms developed for computers can be used by people in everyday life in a wide range of situa…Finding Signal In The Noise - How To Avoid The Noise Bottleneck
The Art Of Being Wise Is The Art Of Knowing What To Overlook. We are blessed and cursed to live in the digital age. We have access to more information than we can possibly handle yet we struggle to fi…The Checklist Manifesto - Your Personal Safety Net
A Checklist Is A Safety Net That Encourages Better Results And Prevents Avoidable Mistakes. The volume and complexity of what we know has exceeded our individual ability to deliver its benefits correc…How Not To Be Stupid - 4 Key Tips
How To Avoid The 7 Causes Of Everyday Stupidity. We are all capable of everyday stupidity as we undertake routine tasks in our business and working lives and also in our personal lives. This is not ab…Beginners Mind And The Voice Of Experience
Only The Experts Survived Evolution.
"In the beginner's mind there are many possibilities, but in the expert's there are few." These are the famous words of Shunryu Suzuki in "Zen Mind, Beginner's Min…The Art Of Being Alone Is A Skill
Let's Make Today A Good Day. You may not have chosen the condition of being alone, and it may have been imposed upon you by circumstances beyond your control, but your response to the situation is wit…Intuition & Anxiety - Are There Angels Or Devils Calling Here?
How To Tell The Difference Between Intuition and Anxiety. How do you know whether the voice of your intuition is real or just the product of your inner anxiety? We all struggle with these inner voices…Like A Prayer - Life Is A Mystery
It Isn't The Process Of Prayer That’s The Problem, It’s The Way It’s Framed. Regardless of what we feel about Madonna or her song the topic of prayer often arouses strong reactions. Usually, it is som…Fear Of Missing Out - "I'll Have What She's Having!"
We Follow The Herd - We Mimic Other People's Choices. Fear of missing out - or FOMO as it is popularly referred to - is the feeling that everyone else in your peer group is having much more fun than y…Free - Self Improvement Resources
Exercising Balance and Discernment. I have just updated these self improvement resources with a number of additional sources of material that are practical and can help you change your life. Check it…Your Higher Self - Your Hardwired Portal To The Universe
This Is The You That Is Beyond Your Thinking Mind. This is the big you, the transcendent you, the you that is often referred to as your higher consciousness or higher self. This is the you that acts a…The Power Of Gratitude - It's Good For You!
Gratitude And Attitude Are Not Challenges, They Are Choices. The power of gratitude quite simply is that it is good for you! Many of us were raised by parents who instilled in us the social niceties o…Living With Your Thoughts - 4 Tools To Help You
Develop A Good Working Relationship With Your Mind. The first step in living with your thoughts is to understand and develop a good working relationship with all areas of your mind. The key to develop…You Are Not Your Thoughts - How To Escape This Ferris Wheel Of Suffering
The 4 Stages To Dealing With Your Thoughts. You are not your thoughts, at least that's what your rational mind tells you most of the time, but that's definitely not how it feels at 3 am when you have…Finish What You Start - Action Will Destroy Your Procrastination
4 Key Tips To Help You Finish What You Start. So why do you fail to finish what you start? Or in my case, why do I fail to even get started? There is a wealth of information out there about how to avo…Going The Extra Mile - The Power Of One More
Life Is Non-Linear, Which Makes The Rewards Of Continued Effort Disproportionately Big. In the pursuit of a long term goal you have to go countless extra miles to set yourself up for a big success. Go…Tao Te Ching - Connecting To Your True Source Of Power.
How To Be Lived By The Tao. The Tao Te Ching is one of those books that many people read, few understand, and even fewer put into practice. The only way to know the Tao is to experience it, and it is…Dying To Self & Dropping The Ego
Everything Does Not Revolve Around Me. The two phrases dying to self & dropping the ego are Christian and Buddhist expressions of the same fundamental truth - the need to correct the inbuilt, hardwire…The Confident Mind - 4 Keys To Peak Performance
Peak Performance Is Meditation In Motion. Peak performance is grounded in the Tao practise of Wu Wei which means "doing with out doing". This is all about how you take action by being so immersed in p…Going With The Flow - The Balance Of Being And Doing
Steer The Boat Each Day Rather Than Plan Ahead Way Into The Future. Going with the flow is about learning to become comfortable with uncertainty, and learning how to exploit the potential hidden withi…8 Principles For Success
A Timeless Foundation On Which To Build Your Success. There are Universal laws and processes that work to support us and guarantee that we achieve all our goals. But first we need to know what they ar…The Loving Father
A Love That Will Not Let You Go. I had a loving father. I knew that I did, but I never felt it. My father was a good man but very damaged and unable to show his feelings. So I grew up knowing in my he…Your Ebenezer Stone Of Help
How To Stay Connected To Your True Source Of Power. Approximately 2,500 years ago the Israeli town of Kafr Qasim was close to the site of two battles between the ancient Israelis and their neighbours…The Stockdale Paradox
How To Deal With The Attrition Factor. The Stockdale Paradox is named after the late James Stockdale, former vice presidential candidate, naval commander and prisoner of war during the American-Vietna…Embrace Your Fear Of Failure And Uncertainty
We Celebrate Success And Ignore Failure. Numerous studies have been undertaken on successful people to understand what it was that they did to become so successful, how they did it and when they did i…The Wilderness Years - 3 Keys To An Exit
How You Got Into The Wilderness Matters Less Than The Lessons To Be Learnt There. The wilderness years can be described as any lengthy time, longer than a year, that is spend aimlessly; without an imm…Boredom – Your Greatest Threat to Results
We Live In A World That Glorifies The Results And Not The Process. We all know what we should be doing, but the trouble with that is that sooner of later we get bored. The enjoyment fades, the excitem…