Tim Harford has written an interesting book called The Logic of Life. I may check it out. It is, after all, intrinsic to my interests in BI and Knowledge Management, which is for those of you who may not be aware, the art and science of figuring out what information is computable and collecting, assembling and disseminating it for the purposes of augmenting decision making. Over the past 20 years, this area of computing has been commercially successful primarily in the form of financial management and reporting because people are fairly disciplined when it comes to money, but it also applies more generally. My interests and career are in the specifics of Business Intelligence, but also in the general of computer augmentation of human decision making. If you ever wanted a formal framework for why I've been writing online for 15 years, that would be a good start.
3. In response, Tim could object that I've overlooked a subtler way for statistical discrimination to harm a group. After all, he heavily emphasizes a few experiments showing that statistical discrimination could be a "self-fulfilling prophesy." For example, he describes a resume experiment where otherwise identical fake resumes with "black names" were less likely to get a response. "High-quality applicants were more likely to be invited for an interview, but only if they were white. Employers didn't seem to notice whether black applicants had extra skills or experience." If that is how employers treat black applicants, what's the point of trying? As Tim asks, "Why bother to get a degree or work experience if you are young, gifted, and black?"
But is it really true that the market fails to reward blacks for getting more education? Is it even true that the market rewards them less? I tested these claims using one of the world's best labor data sets, the NLSY. The results directly contradict Tim's self-fulfilling prophesy story. Blacks actually get a substantially larger return to education than non-blacks! The same goes for experience, though the result is not statistically significant. The real lesson of the data is that if you are young, gifted, and black, you should get a ton of education, because it has an exceptionally large pay-off.
Why would this be so? I'm not sure, but one simple story is that counter-stereotypical behavior stands out. When my sons were young, my wife was working a lot, so I often took my kids places on my own. Funny thing: Time and again, strangers came up and said, "Wow, you're such a great dad!" But there were moms of young kids doing the same thing in plain sight, and the strangers rarely praised them. Why not? Because a dad taking care of two babies is counter-stereotypical, which grabs people's attention.
Purely anecdotal, yes. But it is consistent with the small academic literature on counter-stereotypical behavior. If you clearly violate expectations, people not only notice; they often over-react.
The upshot is that stereotypes may actually be self-reversing rather than self-fulfilling. The marginal payoff of distinguishing yourself from the pack is high if people think poorly of the typical member of the pack.
Black on black discrimination for 'acting-white' does have an intrinsic link to statistical discrimination, you may just be unaware of the statistics. Roland Fryer did the study several years ago.
In any case I tend to look at all matters of this type of research to exhibit a sort of confirmation bias and the bits about self-fulfillment or self-reversal to be post-hoc rationalizations. Stereotypes are, after all, a shorthand way of thinking and in the interests of generating policy, so are statistics. There are very few aspects of human behavior that can be accounted for on the basis of key performance indicators, nor are statistics likely to be taken on the rationale. And so we tend to focus on the few statistics that can be collected reliably and then, post-hoc, try to make sense of them.
Take the example of the federal consent decree set for the LAPD in the wake of the CRASH scandal and general public concern about the beating of Rodney King. There was a stereotypical assumption that white police officers were stopping black motorists because of racist reasons. And so in order to monitor this, a set of new reporting regime & requirements involving racial checkboxes was established. At some point, we would have a fairly representative sample of which officers were citing which people by race, but none of that statistical information gives any outsider a better understanding of the reasons why police do what they do. In otherwords we created a statistical abstract, an incomplete model of officer behavior, which is much more complex than can ever be statistaclly represented - especially on moral questions like discrimination.
So we can generate statistics on discriminations, but can we generate statistics on the rationales behind those discriminations?
It seems to me, that for the purposes of policy and remedy, we are always going to be at a loss with our abstractions, and that those who are close enough to the transactions in question cannot be both efficient at their jobs and effective in generating documentation bringing the layman close enough. We will always use some stereotypical and reverse-stereotypical thinking. It's how human mind copes with realtime performance without crashing.
I believe there are intuitive ways to deal with even counter-intuition. Partisanship is a fairly reliable way to establish a finely tuned and coherent set of double standards, and we are all very familiar with how that works.
But moreover the question of statistical morality is one that I keep coming back to over and over. My prelimnary conclusion is this: If we are to rid ourselves of the human abstraction problem, then we must work within a framework that subordinates our our natural thought processes to its own logic. I often think of Allen Wheelis' Spirit, as the essence of that tale, but one could think of religion or science equally so, or more commonly that process that happens to a teenager when they walk around all day listening to their iPod. We systematically 'let go' of our judgments in order to have some measure of fidelity to external imperatives and in this we find consistency of a sort despite the fact that we defy aspects of our own nature.
spirit is the traveller, passes now through the realm of man. we did not create spirit, do not possess it, cannot define it, are but the bearers. we take it up from unmourned and forgotten forms, carry it through our span, will pass it on, enlarged or diminished, to those who follow. spirit is the voyager, man is the vessel.
So in writing this I have added a new entry of Statistical Morality to my Cobb notebook.
Americans tend to speak about the status of groups of people in statistical terms in order to introduce themselves to subjects concerning them. This is troublesome for two reasons. On the one hand it traffics in stereotypes and reverse-stereotypes about in-groups and out-groups, but does not deal with causality or individuals. It therefore cheapens the quality of discussion and sets up litmus tests and glass ceilings and false dichotomies. Secondly it does not begin to approach with respect to which the dynamic in question is transient. It goes against the principle of discovery.
For these reasons I find sociology to be a discipline that warrants suspicion, the same kind of suspicion economics warrants. However I find that economists tend to be a bit less dogmatic and more self-deprecating in their predictions than sociologists. That is because I believe economists have a great deal more accurate and detailed data than sociologists and thus realize how contingent their models are on complexities beyond reckoning.
Note also these two blog entries: