We can see that the probability of the woman has cancer is calculated as 7.76%. What I'm trying to say is that if builders or banks are in a period of decline then the answer is to avoid those sectors not to invest time and energy trying to pick the best stocks therein. medical tests, drug tests, etc. I'm currently intending to pursue the use of investment trusts to allow me to step back from stock selection and spend more time on sector selection. [This aligns the interests of the management with those of the shareholders and reduces the chances of fraud by the management. ( Log Out / Which might also strengthen the case for IT's or OEICs or ETF's which provide broad coverage of target sectors. I found it a bit confusing when I first read it, because I had wrongly assumed from the title that it is about the Bank of England's base rate, but of course it is nothing to do with that! Bayes’2. Worldwide around 90 per 100,000 people are exhibiting this auto-immune disease. I concluded that what was needed was a historically successful set (or sets) of screening criteria and an investment approach that suits your personality so you stick with it. So even if he had selected his stocks at random from the pool that remained after removing those stocks that did not satisfy his rules, I suspect he would still have done very well over the years (although perhaps not as well as he actually has done after using his skill and judgement in selecting individual stocks from that pool). The axioms of probability are these three conditions on the function P: 1. I really think you are talking about something quite unrelated to the subject under discussion here. Fill in your details below or click an icon to log in: You are commenting using your WordPress.com account. Get an intuition of what Bayes theorem is: One great example of the Bayes theorem and how it impacts our daily decision making is the base rate fallacy. Namely, if the Base rate is low, say 0.1%, the probability is practically zero. Hi Ian, One night, a cab is involved in a hit and run accident. My own experience is that it has several times been possible to call the oil sector and to position oneself with advantage. This is however much, much lower than lactose intolerance, with 0.09%. You are told that “John is a man who wears gothic inspired clothing, has long black hair, and listens to death metal.” You are then asked “How likely is it that he is a Christian, and how likely is it that he is a Satanist?”. 2 Review of Bayes’ theorem Recall that Bayes’ theorem allows us to ‘invert’ conditional probabilities. Be able to use Bayes’ formula to ‘invert’ conditional probabilities. By your logic almost all successful investors could be said to be applying Bayes Theory. Base rate fallacy/false positive paradox is derived from Bayes theorem. Despite John’s appearance increasing the probability that he considers himself a Satanist, the fact is that there are around 2 billion Christians in the world and very few Satanists. Obviously you would want to invest in companies in that sector. generic, general information) and specific information (information only pertaining to a certain case), the mind tends to ignore the former and focus on the latter. Spare production capacity was at an all time low. - He looks for moderately optimistic or better chairman's / CEO's most recent comments. General explanation from Wikipedia:. Bayes’Theorem and Base-Rate FallacyTheorem and Base-Rate Fallacy 3. When I started more serious investing I spent a lot of time reading over 50 books and looking for web based information that would give me an edge over the market. Not a bad shout to get it as an audio book too - I spend a lot of time reading (too much according to some) and have been looking around for material to listen to while I run etc. - He tries to buy stocks that are on modest valuations, which he defines as stocks that have an attractive yield and a low price earnings ratio and /or a discount to net asset value / real worth. [I think another way to look at this rule is he is using negative momentum to make some selling decisions, and it is well known that stocks with recent negative momentum tend to under-perform the market as a whole over the short-term.] [This greatly reduces his transaction costs, and transaction costs act like a tax on performance, so I think this is likely to improve his long-term results.] Ian, I've just finished reading the book 'How to make a million - slowly' by Lord John Lee, who has been an extremely successful private investor over many years. In short, it describes the tendency of people to focus on case specific information and to ignore broader base rate information when making decisions involving probabilities. It is remarkable just how many of these US "Guru" screen selections have beaten the US market, without direct human intervention. Understand the base rate fallacy thoroughly. Multiple sclerosis is one of the more common, rare diseases. Ask Question Asked 6 years, 3 months ... ("prevalence" or base rate probability). Birn-baum showed that behavior described as "ne-glect of base rate" may be consistent with ra-tional Bayesian utilization of the base rate. Base rate fallacy example. Why are doctors reluctant to randomly test or screen patients for rare conditions? noted that research on the "base-rate fallacy" used an incomplete Bayesian analysis. Value stocks, for example - it seems self evident that buying dollars for 50 cents will always prove to be profitable. Not a single scientifically hold belief for something, let’s say that mitochondria are the “powerhouses” of the cell, is based on only one assumption or observation. If you will allow me to play Devil's advocate for a minute though, how would you say that picking sectors is different from picking stocks? The scenario looks at a driver being stopped and breathalysed and aims to calculate the probability that a driver who fails the test is actually over the limit. The chance that somethingin the outcome space occurs is 100%, because the outcome space contains ever… The English statistician Thomas Bayes has done an interesting experiment on how to visualize that. I chose the title because the dash of alliteration made it sound punchy (at least in my mind...). In short, it describes the tendency of people to focus on case specific information and to ignore broader base rate information when making decisions involving probabilities. Answer to the Thought Experiment: The exact answer to this problem depends upon what percentage of the population is homosexual. But if we do the test with 100,000 people again, we get: Due to the rare occurence of this disease the confidence in the test, even though the test is as good as the one above, goes down to less that 50%, i.e. Why would I be more likely to get it right just because I'm analysing a different aspect of the future? Good luck with your investing, Conclusion They know about it. [Again, this reduces the chances of fraud by the management at the expense of shareholders.] He asked his servant (in yellow) to throw a ball on the table and mark the position, where the ball has landed. Student of Life Suppose you came to the realisation that the oil sector was poised to outperform. In that case, each new ball (new information) updated his belief. Now, lets say, that a similar test as above is developed for this disease, i.e. [Small companies tend to perform better over the long-run than larger ones, although that is not the case in every year.] Pretty much any house builder you bought a few years ago would have done extremely well and if you knew the sector was undervalued, you could have saved yourself a lot of effort by just buying a basket of them. Therefore, in practice we almost always have to expand: Bayesian theorem basically tells us to look at all the cases where the evidence is true and then looking at the proportion of these evidences, where the hypothesis is also true. Applications and examples. Bayes’ theorem was developed by Rev. Why do knowers of Bayes's Theorem still commit the Base Rate Fallacy? Another early explanation of the base rate fallacy can be found in Maya Bar-Hillel’s 1980 paper, “The base-rate fallacy in probability judgments”. Interesting what you say about picking sectors, it makes sense in the Bayesian context and the house builders you mention are quite a good example. Conclusion5. 8.5 The Base Rate Fallacy. 6. Bayes’ theorem has been a controversial idea during the development of statistical reasoning, with many authorities dismissing it as an absurdity. is has the same 99.9% true positive rate and the probability of being tested negative, while still developing MS is also pretty low (false positive: 0.02 %). Be able to organize the computation of conditional probabilities using trees and tables. ( Log Out / The evidence would suggest that experts and amateurs alike are poor forecasters whether it comes to company earnings or macro events - it seems the future just isn't all that clear, whatever the scale! So we are restricting our view to where the evidences holds. When given relevant statistics about GPA distribution, students tended to ignore them if given descriptive information about the particular student even if the new descriptive information was obviously of little or no relevance to school performance. The base-rate fallacy only occurs with frequentist methods because they cannot use prior information in a straightforward way. This means that the odds are still overwhelmingly in favour of John being a Christian. Bayes Theorem is a mathematical equation where you can input the Base Rate for an event along with the probabilities associated with new information to get the actual overall probability for the event. We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. Tom, http://www.aaii.com/stock-screens?a=menubarHome. Using Baye's theorem, we get actual probabilities of competing hypotheses. In fact it is the opposite of drunken rationale and takes you though a history of the development of randomness theory and the need for the evolutionary human brain to look for cause and effect patterns that are either not there, or that we misinterpret. This updated belief (the resulting posterior probability) incorporates all the evidence of that claim. Base Rate Fallacy: This occurs when you estimate P(a|b) to be higher than it really is, because you didn’t take into account the low value (Base Rate) of P(a). An example is scrutiny (and subsequent demolition) of Fortune 500 companies who hire or fire their CEO's for what turns out to be random short term financial success of failure. I cannot find any of that reflected in your discussion of John Lee's approach that will help others to emulate it. The probability of every event is at least zero. The rate at which something happens in general is called the base rate. In other words, he greatly improved his 'base rate' probabilities of investing success by following those rules...." Have a good evening, "If you will allow me to play Devil's advocate for a minute though, how would you say that picking sectors is different from picking stocks? Tom, I think your article is excellent, but it's use of the mathematical term Bayes Theorem might frighten a lot of people who are not mathematicians. Quite a few of his examples relate to gambling, but they could equally as well be attributed to our "investment" decisions. People tend to simply ignore the base rates, hence it is called (base rate neglect). 2 Conditional Probability. This and other experiments led eventually to a mathematical formulation of Bayes theorem. I have been listening to an excellent audiobook in the car (also available as a book) called, "The Drunkard's Walk: How Randomness Rules" by Prof L. Mlodinow . In retrospect perhaps I should have opted for plain old clarity instead. Christians might possess the same characteristics only rarely but their numbers are big. might address those concerns. This equation is completely fine like it this, but let me expand on P(E), the probability of seeing the evidence, a little bit more. I'm read Kahneman so have already grappled with Bayes Theorem and found it fascinating to see how absolutely counter intuitive the outcomes are when it's applied to apparently simple problems. Why would I be more likely to get it right just because I'm analysing a different aspect of the future? Change ), You are commenting using your Twitter account. Explained based on a test for a rare disease: Basically, when the percentage of people with a disease is lower than the test’s false positive rate, the chance of getting a false positive is higher than actually having the disease. Better still when my logic and high Stockrank numbers happen to coincide, or is this just another random event? Easy Definition of Base Rate Fallacy: Don't think "99% accurate" means a 1% failure rate.There's far more to think about before you can work out the failure rate. support the ongoing hypothesis or refute the held beliefs. I am familiar with Bayes theorem and I am a big fan of StockRanks but I hadn't made the connection. Interesting, thanks for getting back to me. This video by Julia Galef explains more about how you can use Bayes’ theorem, not just to avoid the base-rate fallacy, but also to improve your thinking more generally. Tom, Thanks for the feedback - I quite enjoyed writing this one. Consumption was growing strongly. If we test 100,000 people with this test, we get: As a person that receives a positive test result, how confident should you be in trusting that result? Example 1 given on the Wikipedia page is clear and easy to picture. Is it easier? The base rate fallacy and its impact on decision making was first popularised by Amos Tversky and Daniel Kahneman in the early 1970’s. 1. Bayes' theorem for the layman. 47.37% (90 / (90 + 100)). 2. - He prefers conservative, cash-rich companies or those with low levels of debt. If a woman has breast cancer, the probability that she tests positive is 90% ("sensitivity" or reliability rating). From a personal perspective, I am still a little wary as I do not have full faith in my ability to reliably identify such trends in a timely manner due to my inexperience, ignorance and so on. Change ), How to do Science: Bayes Theorem and the base rate fallacy, Distinction between Frequentist and Bayesian Approaches, being identified positive, given that you’re sick, being identified positive, given that you not carry the disease, being identified negative, while not carrying the disease, being identified negative, but actually having the disease.
2020 base rate fallacy bayes