The Cost of Truth is Eternal Vigilance

A recurring theme on this blog is that it is unwise to rely on 'everyday' or uncritical thinking because our minds are liable to innumerable biases, failures of memory, and so on. An important part of being a good thinker, then, is to submit ideas - and especially our own - to critical scrutiny. I am not, obviously, immune to these biases, in fact, I am as liable to them as anyone else. I do work hard to scrutinize my beliefs carefully, though, and I regularly give up previously held beliefs as a result. To demonstrate not only the dangers of uncritical thinking, but also that I (try to) practice what I preach, here are two recent instances of having to change my mind. Both are pretty unimportant beliefs, but they illustrate the issues nonetheless.

I moved from Johannesburg to Durban in early 2007 and my fiancée did the same in early 2009. Possibly as a result of her comments about how much it has been raining in Durban, I came to believe that 2009 had been an especially wet year: I thought it must be the wettest since I'd moved here. I knew, of course, that the only way to establish this for sure was to look at actual statistics because our memories are flawed and we use the availability heuristic to make inferences about trends. But... I didn't bother to check for a while. When I finally did, it became quite clear that my intuitive sense about Durban's weather was spectacularly wrong. The wonder that is Wolfram Alpha let me create the following two graphs: the first shows the total estimated yearly precipitation (rain, for Durban's purposes) for the last 5 years, and the second shows (I think weekly) rainfall amounts over the same period.

As should be abundantly clear, 2009 is not the wettest year since I moved to Durban, it is in fact the driest. Now, it could be the case that 2009 had less total rainfall, but more rainy days, so I could have been misled for that reason. The second graph, though, is only mildly suggestive on that front and I can find no other data (that's free). So it seems fair to conclude that I was led astray by thinking intuitively when I should have known not to trust my intuitions about trends in complex, variable systems. (For detailed evidence that people are spectacularly bad at thinking statistically, see Kahneman, Slovic & Tversky, 1982).

The second example concerns bias and rather nicely illustrates the importance of blinding. If you had asked me a while ago what the best search engine was, I would have said: "Google - and by a wide margin". Until I found BlindSearch, that is. Branding biases our judgments and Google's brand is so powerful that being objective while knowing which search engine's results you're looking at is extremely difficult. BlindSearch remedies this problem: it lets you search Bing, Yahoo and Google simultaneously, presents the results in three columns, and blinds you to which search engine produced which results. You look through the results, vote for the one you prefer, and then only are the brand names revealed. I've now used BlindSearch dozens of times and a clear pattern has emerged: Google isn't nearly as superior as I once thought it was. While I still tend to prefer Google's results a plurality of the time, Bing and Yahoo do get my vote more often than I would have thought. For the sake of concreteness, here are ten searches I did with my vote listed next to it. I tried to pick topics that were either obscure or controversial to 'test' the search engines, since search terms with obvious results aren't exactly indicative of quality. Also, I verified some of these results by checking whether my vote stayed the same later (it did in all cases).

• Science - Bing
• Islamic terrorism - Difficult call, but Yahoo
• Evolution - Yahoo
• iPad - Google (though it was close)
• Michael Meadon - Google
• Sex - Yahoo (though Bing doesn't work for some reason)
• Homeopathy - Google (by a large margin)
• Jacob Zuma - Bing (close)
• Richard Dawkins - Google (Yahoo sucked)
• Sitcky the stick insect - Google, by far (also wins on Olaf the Hairy)

So that's 2 for Bing, 5 for Google and 3 for Yahoo. Without blinding, my guess would have been that I would have preferred Google 9 times out of 10. Turns out I was wrong. And, contrary to what I'd like to believe, branding works on me too. Bottom line: our biases affect our decisions and our judgments, so when those decisions or judgments are important (which is not the case with search engines), appropriate blinding is vital.

These are just two, small, inconsequential examples, of course. They illustrate an important point though: if you want to be right, you have be be skeptical, self-critical, willing to reconsider and admit error, cautious, and scrupulously careful with facts and arguments. Or, to corrupt a glorious quote misattributed to Thomas Jefferson: the cost of truth is eternal vigilance.