Are most experimental subjects in behavioral science WEIRD?

Note: here is a follow up post.

My supervisor David Spurrett and I have a commentary on an important paper - "The weirdest people in the world?" (pdf) - in the most recent edition of Behavioral & Brain Sciences. The authors of the paper, Canadian psychologists Joseph Henrich, Steven Heine and Ara Norenzayan, argue that most experimental subjects in the behavioral sciences are WEIRD - Western, Educated, Industrialized, Rich, and Democratic - and thus weird - not representative of most human beings. And this, if true, is a very serious problem indeed. Behavioral scientists (anthropologists, psychologists, behavioral economists and so on) are often interested in explaining the brains, minds and behavior of Homo sapiens as a species. (Some scientists, of course, are only interested in understanding specific cultures or what makes us different, but one important goal of the behavioral sciences has long been to explain universal human behavior). As evolutionary psychologists John Tooby and Leda Cosmides have put it, they "seek to characterize the universal, species-typical architecture of [the information-processing mechanisms that generate behavior]".

But... Henrich and his colleagues review a large body of literature that seems to show that, across several domains, Western undergraduates - the workhorses of the behavioral sciences - are extreme outliers. In other words, if they are correct, most of the data behavioral scientists have used to test hypothesis and to drive theorizing derives from subjects who are possibly the least suited for generalizing about the human race. Take as an example the Müller-Lyer illusion. In the diagram below, the lines labeled "a" and "b" are exactly equal in length, but many subjects perceive "b" as longer than "a".

This finding (which goes back all the way to 1889) has been used to make deductions about how the human visual system works. The Wikipedia article on the illusion, for example, states that one possible explanation for the effect is that "the visual system processes that judge depth and distance assume in general that the 'angles in' configuration corresponds to an object which is closer, and the 'angles out' configuration corresponds to an object which is far away". Plausible enough. Except that for some people - San foragers, for example - the illusion does not exist, and in many other non-WEIRD societies the effect size is significantly smaller. Henrich and his colleagues cite the work of Segall et. al. (1966), who worked out the magnitude of the illusion across 16 societies by varying the relative lengths of "a" and "b" and then asking subjects to indicate when they thought the lines were equal. The percentage by which "a" must be longer than "b" before the lines are adjudged equal - what they call the "point of subjective equality" (PSE) - varies substantially between subjects from different cultures - and, importantly, WEIRD-subjects are extreme outliers. The results are summarized in the following graph:

Both WEIRD adults and children (aged 5-11) require "a" to be 18%+ longer than "b" before they're perceived as equal, but for the San and South African miners, the illusion simply does not exist - their PSEs are not statistically distinguishable from 0. Why this difference arises is unknown, but Segall et. al. claim it is due to WEIRD people's visual systems developing differently because modern environments expose them to ("unnatural") shapes like 'carpeted corners', thus calibrating their visual systems in a way that favors the emergence of the illusion. Whatever the true explanation, however, it is clear that it is not permissible to use the existence of the illusion among WEIRD subjects to make inferences about the visual system. This is especially true since the San subjects were hunter-gatherers, just like all people for the vast majority of human evolutionary history. Given that species-typical features of the visual system would have evolved in this period, it is particularly telling that PSE seems to be positively correlated with the 'modernity' of the societies in question. (Warning: this is an "eyeball" observation; I haven't done a proper statistical analysis. Caveat emptor).

This is one example from an extremely long paper, but it conveys a flavor of the kind of evidence the authors present. (For much more, see "We agree it's WEIRD, but is it WEIRD enough?" over at Neuroanthropology). Having read the article very carefully, and despite some concerns, I think Henrich, Heine and Norenzayan are right: the Western undergraduate is often unrepresentative of humanity, and the behavioral science literature needs a lot of fixing as a result. (Most obviously, we need far more large, highly-powered, globally representative, prospectively designed, cross-cultural studies). Serious as this is, unfortunately, it gets worse... Since David and I worked extremely hard to present our argument clearly and concisely in our commentary (pdf - our piece starts on p. 44 of the pdf, paginated by BBS as p. 104), and I doubt I could improve on it, what follows is a slightly edited - simplified and somewhat de-academicized - version of the meat of our argument. (Note: each issue of BBS consists of a "target article" - in this case, Henrich et. al. - and 20 or so short peer-commentaries).

Henrich et al. underplay – to the point of missing – that how the behavioural sciences research community itself is constituted introduces biases. That the subject-pool of behavioural science is so shallow is indeed a serious problem, but so is the fact that the majority of behavioural researchers are themselves deeply WEIRD. People in Western countries have, on average, a remarkably homogeneous set of values compared to the full range of worldwide variability (Inglehart & Welzel 2005), and the data Henrich and his colleagues present suggest similarly population-level homogeneity in cognitive styles. Moreover, academics are more uniform than the populations from which they are drawn, so it is likely behavioral scientists are even WEIRDer than their most common subjects. Henrich and his colleagues review a bunch of studies and experiments that did not strike those who designed and conducted them as focused on outliers. Intelligent scientists acting in good faith conducted, peer-reviewed, and published this research, in many cases honestly believing that it threw light on human nature. This forcefully illustrates the power of the biases on the part of researchers themselves. It also suggests that, besides widening the pool of subjects, there are significant gains to be made by broadening the range of inputs to the scientific process, including in the conception, design, and evaluation of empirical and theoretical work. Given that diverse groups are demonstrably better at some kinds of problem solving, as things stand, the WEIRD-dominated literature is robbed of potentially worthwhile perspectives, critiques, and hypotheses that a truly global research community could provide. Clearly, simply increasing the number of behavioural sciences researchers will, in general, be beneficial. Our key contention, though, is that the marginal benefits of additional Western researchers are much smaller than the marginal benefits of more non-Western researchers, among other things, just because they are non-Western.

The non-Western world, in short, can contribute not only additional subjects to experiment upon – the main focus of the target article’s recommendations – but also additional researchers, with novel perspectives and ideas and who are less affected by WEIRD biases. (Naturally, these researchers will have biases of their own. Our claim is not that there is someone who consistently knows better; it is that diverse groups of investigators can avoid some kinds of error.) Clearly, these researchers will have to be educated, will likely be middle class, and, since science flourishes in politically open societies, they will tend be concentrated in liberal countries. Nevertheless, additional non-Western researchers, even if they are educated and relatively wealthy, could be a boon to the behavioural sciences.

A direct and powerful way to remedy both sources of bias – too many WEIRD subjects and too few non-WEIRD researchers – is to foster research capacity in the non-Western world. Non-WEIRD researchers tend to study non-WEIRD subjects, so increasing their number will deepen the subject pool and widen the range of inputs to the scientific process at the same time. Building research capacity, however, should not merely involve collaborations led by WEIRD researchers; it should aim to generate studies led and initiated by non-Western researchers. Committed and long-term inter-institutional collaboration between Western and non-Western universities focused on remedying the deficits in the behavioral sciences literature should include internships at Western universities for non-Western researchers, stints at non-Western universities for WEIRD researchers, and extensive student exchange programs (especially for graduate students). Unlike many existing scholarship and exchange programs in the sciences, a key point of the necessary programs should be for the learning to proceed in both directions.

----------------------------------
Henrich, J., Heine, S., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33 (2-3), 61-83 DOI: 10.1017/S0140525X0999152X

Meadon, M., & Spurrett, D. (2010). It's not just the subjects – there are too many WEIRD researchers Behavioral and Brain Sciences, 33 (2-3), 104-105 DOI: 10.1017/S0140525X10000208