Testimonials, Covariation Tables and Magic Pills
Scott Plous, in 1993, wrote an excellent text or compilation of the then current state of the art on the Psychology of Decision making, and included a number of test exercises at the beginning of the text. The tricks of the mind that psychologists have discovered are very ably covered by Plous. There is a general agreement amongst scientists is that there are heuristic tools that we use to break down a decision problem, which generally work well. But we overuse these heuristics and make predictable mistakes.
Here is an example about the use of testimonials to aid our decision making. If I am going to buy a new computer, I will generally read product reviews, especially from authorities. Some of these reviews provide important information. When the "discovery" of a Mac OSX virus was a hot topic, reading authoritative reviews helped my understand what the problem was and how to gain protection until Apple released its security update. In general, relying upon authorities is a good idea and trying to identify an authority is a worthwhile exercise.
Where is this a bad strategy is when the event you are seeking information about is highly unusual, rare, or wildly out of the norm. How important are authorities in these cases? Consider the following problem. You are a member of jury in personal injury lawsuit. The plaintiff has testified that the car that hit him was yellow. Let's make the plaintiff an "authority" on spotting cars: if a car is yellow, then the plaintiff will correctly identify it as yelllow 90s% of the time. Further suppose that there are only two colors of cars: yelllow and green. Our authority is just as good as spotting yellow cars as green cars. You also know the total number of cars is 200, and 180 are green. What are the chances that if the plaintiff says that the car was yellow, it was yellow, given that he is an authority. Well, let's count the cases.
| A = number of cars that are yellow & are identified as yellow. | C=number of cars that are green & are identified as green. |
| B = number of cars that are yellow & are misidentified as green |
D= number of cars that are green & are misidentified as yellow. |
From the population totals, we know that A + B + C + D = 200, and that A + B = 20, while C+ D = 180. We also know that our authority doesn't make mistakes very often, so that A/A+B = .9, or A= 18 and B =2. Similarily, we know that C/C+D = .9, so C = 162 and D =18. What is the universe or total collection of cars and identifications in which our authority says that the "car was yellow". It is the information state containing A and D, a total number of 36 different states. Only in half of those states, A, does our authority get it right despite his 90% ability. This is because the chances of getting a yellow car are small, 20/200. So when our expert identifies a yellow car, he has picked a relatively rare event to be right about. As David Hume said, and I paraphrase, when presented with testimony as to a very rare event, I have to judge whether the rare event happened or that testifier has made some mistake.
What does this have to do with magic gas pills?
Technorati Tags: hot topic, authoritative reviews, heuristic tools, test exercises, mac osx, gain protection, topic reading, psychologists, new computer, predictable, compilation, state of the art, scientists, psychology, testimonials, discovery, heuristics, product reviews, scott plous
At this site there are a series of "testimonials" from happy campers about the efficacy of Bioperformance. One individual goes so far as to draw an illusory correlation between posters who have purchased the produce and their satisfaction with it versus all the other posters who have panned the product on scientific grounds. The suggestion is that if you have not tried it, then you cannot comment on the efficacy of the magic pills.
But that is just wrong. Magic pills are a rare event, not non-existent because science does produce amazing results. But when judging, we are entitled to use the rarity of magic pills as valid information. Could the testimonials, assuming that they are even honest, be wrong in some detail? A very good statement of the minimal conditions for the testimonial to be correct was stated at the Quatloos site,
Post the mileage logs -- before and after using the magic pills.
Also, include comments as to any changes in your driving habits or routes and any service records other than normal minor maintenance.
This information should cover a statistically significant period and should show a comparison of the same season for the before and after.
Anything less is subject to too many variables to attribute it to an untested product.
Technorati Tags: hot topic, authoritative reviews, heuristic tools, test exercises, mac osx, gain protection, topic reading, psychologists, new computer, predictable, compilation, state of the art, scientists, psychology, testimonials, discovery, heuristics, product reviews, scott plous
