Anecdotes as evidence

An anecdote is a tale, story or an account of events that is sometimes humorous or meant to convey a moral, but is often taken as evidence. In this latter sense there are two broad categories. Firstly, there is testimony, i.e., inferring that such-and-such happened or is true because someone said so. For simplicity's sake, we can further divide testimony into a bunch of different subtypes, most notably, into eyewitness testimony, expert testimony and hearsay. The second broad category of anecdotal evidence is personal experiences (or, if you will, ‘personal testimony’) that takes the form 'I saw so-and-so, therefore it is reasonable for me to believe such-and-such'. Note the key similarity and key difference between these two categories: in both cases someone's experience (or alleged experience) is taken as evidence for some claim, but, for testimony and not for one's own experiences, one has to believe a particular experience occurred on someone else's say-so. Now, it seems perfectly reasonable to believe some things on testimony or personal experience - indeed life would be impossible without it. I am currently having the personal experience of sitting on my couch with my notebook on my lap while typing this post. Hyperbolic doubt aside, I have no good reasons to doubt that his is what is really going on, and you have little reason to doubt my testimony. On the other hand, however, it is common for people to believe wild or improbable things – that aliens regularly visit earth, that highly diluted substances can cure any illness, etc. – solely on the strength of anecdotes. So what exactly is the evidentiary status of anecdotes?

It is vital, first, to distinguish between two types of proposition that anecdotes can be claimed to be evidence for: causal and observational propositions. A causal claim is of the form “P caused X” (or “P, Q and R caused X”) and an observational proposition is “P happened” (or “P occurred, then Q happen and after that R”). This distinction is important for a simple reason: an anecdote can in isolation (almost) never establish the truth or falsehood of a causal claim, it can only be evidence for observational propositions. Why this is the case will be clearer if we think through some concrete examples.

Consider the claim “Mary cheated on John with Bob”. This kind of proposition is pretty straightforward. I can convince myself it is true simply by determining whether Mary and John are in an exclusive relationship, and, if I see Mary make out with Bob, I can reasonably conclude the proposition is true. Now, obviously, there are a bunch of ways I could get it wrong: maybe the woman involved wasn’t Mary, maybe it was simply a friendly hello kiss, or maybe Mary had broken up with John (so it’s making out, but not cheating). It’s clear, though, that if I’m just a little careful, there are a wide variety of circumstances in which I could be very confident Mary did in fact cheat based on my personal experiences. There are a couple of extra complications when John has to decide whether to believe my testimony – maybe I’m lying, for example – but, again, these are not difficult to understand even if they’re difficult to deal with in practice. In other words, we here have a clear case of an anecdote – both in the sense of personal experience and testimony – that can be a good reason to accept the truth of some proposition. Notice two things, though: the claim is not a causal one, and the plausibility (or prior probability) of it being true is pretty high since we know people do in fact cheat on each other regularly. I’ll explain what the latter means in a bit, but let’s move on to anecdotes as evidence for causal claims.

So consider the causal proposition “I took medicine X, I got better, therefore I got better because of medicine X”. This claim is much more complicated than the one about Mary, and you can’t determine whether it’s true simply by looking or making a few observations. Why? Because causality is counterfactual: to say A caused B, is to say B would not have occurred if A had not occurred. (There is a long-standing and complicated philosophical debate about causality. Take it from me: steer clear and stick to the counterfactual view). So, on our example, when you claim “medicine X caused my recovery” you’re committed to the counterfactual proposition that you wouldn’t have gotten better had you not taken medicine X. And you simply cannot, even in principle, know this. For one thing, you have an immune system (gasp!) which may have fought off your infection irrespective of whether you took the medicine. Alternatively, you could have ingested some other substance - maybe you took medicine Y as well, maybe you ate something therapeutic - that took care of the infection. In other words, there are a whole bunch of things - i.e. confounds - that could have caused your recovery, and a single observation – the source of an anecdote – simply cannot distinguish between them. In general, the only reliable way to establish the truth or falsity of causal claims is to do controlled experiments. Let’s look at this a bit more closely.

Assume we are trying to understand the causal relationship between four dichotomous and independent variables (A-D) and a particular dichotomous outcome (O). Assume also that these four variables exhaust the universe of all variables even conceivably related to the outcome. Our aim is to make either inclusion inferences (i.e. conclude the relevant variable has a causal relationship to the outcome) or exclusion inferences (i.e. that the variable does not have a causal relationship to the outcome). Given these assumptions, inclusion inferences are valid only when, from:

A1   B1   C1   D1  =>  O1
A2   B1   C1   D1  => O2

it is concluded that A1 caused O1. The inference is valid because all the variables except one (A) was controlled – held constant – and given the outcome changed, it follows that A is causally related to O. Exclusion inferences are valid only when, from:

A1   B1   C1    D1   =>  O1
A2   B1   C1    D1   =>  O1

it is concluded A1 is not causally related to O1. Again, the validity of the argument is assured because all the variables save one (A) was controlled. Given that A varies while O does not, it follows that A is not causally related to O. Notice that in both cases we are comparing one outcome with a counterfactual. The process of engineering such comparisons is called experimentation and it is at the very heart of the scientific method. (These, by the way, are versions of Mill's Methods).

If we want to determine whether medicine X can cure some disease, we cannot rely on anecdotes because they don’t allow for counterfactual comparison. Assume, for example, that A1 is taking medicine X, and A2 is not taking it; that B1 is having an immune system, and B2 not; that C1 is taking medicine Y, and C2 not taking it; that D1 is being overweight, D2 is not being overweight; and, finally, that O1 is getting better and O2 is not getting better. When we have a single observation - we know that Thaba over there took medicine X, has a healthy immune system, that he isn't taking medicine Y, that he's rather overweight and that he got better after a few days - all we have is:

A1   B1   C1    D2   =>  O1

We have no proper counterfactual: we have not controlled for variables B, C, or D so we can't make any logical inferences about variable A. (Making this inference is the post hoc fallacy). At best, we can say that taking the medicine is possibly related to getting better, but then the same goes for B, C and D. Note also that adding more anecdotes does not resolve our problem: in the real world there are many more than just four variables so things are much more complicated, experiments involving humans are always possibly confounded by the placebo effect, and, importantly, the variables may interact in complex ways. In the oft repeated phrase, the plural of anecdote is anecdotes, not data. Thousands of anecdotes are no more convincing that a single anecdote. As a result, then, anecdotes cannot in general (that is, barring extreme exceptions) establish the truth or falsity of causal propositions.

As I showed above anecdotes can reasonably be taken as convincing evidence for observational claims. But that does not mean we should believe every anecdote (concerning observational propositions). Bob Carrol of Skepdic (an excellent resource worth referring to often, by the way) nicely enumerates the possible problems:

Anecdotes are unreliable for various reasons. Stories are prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on. Most stories get distorted in the telling and the retelling. Events get exaggerated. Time sequences get confused. Details get muddled. Memories are imperfect and selective; they are often filled in after the fact. People misinterpret their experiences. Experiences are conditioned by biases, memories, and beliefs, so people's perceptions might not be accurate. Most people aren't expecting to be deceived, so they may not be aware of deceptions that others might engage in. Some people make up stories. Some stories are delusions. Sometimes events are inappropriately deemed psychic simply because they seem improbable when they might not be that improbable after all. In short, anecdotes are inherently problematic and are usually impossible to test for accuracy.

In other words, while anecdotes can be good evidence for believing observational propositions - "x happened" - for the reasons listed above, we certainly can't accept all anecdotes uncritically. So what to do? Life is impossible if we dismiss all anecdotes, but we'll be led astray if we accept all anecdotes. The solution is skepticism: that is, being open minded but then filtering beliefs through a bullshit detector. Doing this is simple in principle, but incredibly difficult in practice, so some examples are in order. (Recommended books: Thinking About Thinking by Anthony Flew, The Demon-Haunted World by Carl Sagan [my review], Truth by Simon Blackburn [my review], and Mistakes Were Made (But Not by Me) by Travis & Aronson [my review]).

Megan Fox: not in Jeff's league. 

One of the most important and useful bullshit detecting skills is weighing up the evidence against the plausibility of the claim (here is an example of me doing this). Determine, firstly, how plausible the claim is given everything else we know. For example, given everything I know about my friend Jeff, people in general and the state of technology, the proposition that he has flown in an airplane at least once is highly plausible. (He is middle-class, airplane tickets are cheap and abundant, I've seen him in other cities, etc.). On the contrary, given everything we know, it is extremely implausible that he once had a threesome with Megan Fox and Jessica Alba. (For one thing, he is short and balding. For another,  he's never been to the US). The threesome claim is an extraordinary one: given what we know about attractive celebrities, balding South African men, sexual psychology, and so on, Jeff having a threesome with Fox and Alba is just not the kind of thing we expect to happen. Having determined the plausibility of a claim, the next step is to assess the strength of the evidence. In our example, all the evidence we have is Jeff's testimony. As Bob Carrol explained above, anecdotes are often unreliable because people suffer from innumerable cognitive biases and, more obviously, they sometimes lie. Since in the Alba-Fox example Jeff has a strong motivation to lie - having it believed is highly status-enhancing - his testimony is further undermined. What we have, then, is an extraordinary claim, the only evidence for which is very weak. It is reasonable, then, to withhold assent until better evidence is provided.

A somewhat more enlightening example is alien abduction. People from all over the world claim to have been abducted by extraterrestrials and then molested, lectured on the necessity of world peace, and so on. So, step one: how plausible are these claims? Given what we know about physics and human psychology, not very. First of all, we currently have no evidence that life - let alone intelligent life - exists anywhere else in the universe. (My gut tells me alien life is abundant, but as Carl Sagan pointed out, we shouldn't think with our guts). Secondly, if intelligent life does exist, the aliens will in all likelihood be tens of light-years or more distant, and, since we have no reason to think faster than light travel is practicable, there is no known way for aliens to get to earth in a reasonable period of time. Step two: what about the evidence? Again we have anecdotes: tales from people who claim to have been abducted. Significantly for this example, there is a highly plausible alternative explanation that undermines the evidentiary status of the accounts, namely, hypnogogia and hympnopompia. Briefly, these are vivid hallucinations that occur as you're falling asleep or waking up that are accompanied by sleep paralysis. Typically, a person wakes up terrified and unable to move, senses (an often malevolent) 'presence' in the room, and may also experience a variety of visual, auditory and proprioceptionary hallucinations. Tellingly, this well-studied phenomenon closely mirrors accounts of alien abduction, which often feature extreme fear, a 'presence' in the room, and being unable to move. Since these experiences are accompanied by visual hallucinations and alien visitation is a common trope in popular culture, the other reported experiences are easily accounted for. Importantly, also, hypnogogia and hypnopompia are common (much more common than claims of alien abduction). Extraordinary claims require extraordinary evidence. Since claims of abduction are extraordinary, anecdotes of being abducted are far from extraordinary evidence, so, until much more evidence is provided, it is reasonable to withhold assent.

So... what exactly is the evidentiary status of anecdotes? In summary: (1) anecdotes on their own can never establish the truth or falsity of causal propositions. (2) While anecdotes can be evidence for observational propositions, the plausibility of the claim must be taken into account. To be believed, highly implausible claims require much, much more than mere anecdotes.