I moved from Johannesburg to Durban in early 2007 and my fiancée did the same in early 2009. Possibly as a result of her comments about how much it has been raining in Durban, I came to believe that 2009 had been an especially wet year: I thought it must be the wettest since I'd moved here. I knew, of course, that the only way to establish this for sure was to look at actual statistics because our memories are flawed and we use the availability heuristic to make inferences about trends. But... I didn't bother to check for a while. When I finally did, it became quite clear that my intuitive sense about Durban's weather was spectacularly wrong. The wonder that is Wolfram Alpha let me create the following two graphs: the first shows the total estimated yearly precipitation (rain, for Durban's purposes) for the last 5 years, and the second shows (I think weekly) rainfall amounts over the same period.
Kahneman, Slovic & Tversky, 1982).
The second example concerns bias and rather nicely illustrates the importance of blinding. If you had asked me a while ago what the best search engine was, I would have said: "Google - and by a wide margin". Until I found BlindSearch, that is. Branding biases our judgments and Google's brand is so powerful that being objective while knowing which search engine's results you're looking at is extremely difficult. BlindSearch remedies this problem: it lets you search Bing, Yahoo and Google simultaneously, presents the results in three columns, and blinds you to which search engine produced which results. You look through the results, vote for the one you prefer, and then only are the brand names revealed. I've now used BlindSearch dozens of times and a clear pattern has emerged: Google isn't nearly as superior as I once thought it was. While I still tend to prefer Google's results a plurality of the time, Bing and Yahoo do get my vote more often than I would have thought. For the sake of concreteness, here are ten searches I did with my vote listed next to it. I tried to pick topics that were either obscure or controversial to 'test' the search engines, since search terms with obvious results aren't exactly indicative of quality. Also, I verified some of these results by checking whether my vote stayed the same later (it did in all cases).
- Science - Bing
- Islamic terrorism - Difficult call, but Yahoo
- Evolution - Yahoo
- iPad - Google (though it was close)
- Michael Meadon - Google
- Sex - Yahoo (though Bing doesn't work for some reason)
- Homeopathy - Google (by a large margin)
- Jacob Zuma - Bing (close)
- Richard Dawkins - Google (Yahoo sucked)
- Sitcky the stick insect - Google, by far (also wins on Olaf the Hairy)
These are just two, small, inconsequential examples, of course. They illustrate an important point though: if you want to be right, you have be be skeptical, self-critical, willing to reconsider and admit error, cautious, and scrupulously careful with facts and arguments. Or, to corrupt a glorious quote misattributed to Thomas Jefferson: the cost of truth is eternal vigilance.