I’ve been railing against education policy arguments based on identifying places with successful outcomes and then claiming we should imitate certain practices or policy features of those places to achieve success elsewhere. I argued forcefully against it in my review of Marc Tucker’s book. Anytime someone says Finland (or Massachusetts or whoever) is doing well because they have high standards or little choice or no school athletics, or whatever, they are just engaging in quackery — pure BS.
It is a matter of basic logic that one cannot know what causes success only by looking at a successful place (or set of successful places). You cannot know whether any factor contributes to success without also considering unsuccessful places and examining whether that same factor tends to be more present or absent in successful relative to unsuccessful places. This is an error known as “selection on dependent variable” and it is taught in any decent introduction to research methods course.
People who regularly draw policy recommendations based on the Finland du jour should be made to hold up a giant sign that says “I do not understand the basic research methods of the field in which I claim to be an expert.” And that’s just it. These people aren’t really experts. There used to be a time when a clever writer could be considered an expert on education policy simply by virtue of articulating a clear-sounding argument. That time is gone.
Even journalists should be expected to have some basic understanding of the methods in education policy research. No one would accept that science reporters could be ignorant of fundamental principles of the scientific method. No one would accept that diplomatic correspondents would have no knowledge of diplomatic history. Even journalists — especially when they are writing book-length arguments — need to have some understanding of research methods that would include the obvious point that no causal claims can be made from selection on dependent variable analyses.
So, I was delighted to see the Brookings Institution’s Russ Whitehurst take on David Kirp’s book, Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools, which draws lessons from an examination of one “successful” district — Union City, NJ. Whitehurst begins by illustrating why selection on dependent variable does not allow for causal claims:
A while back, I read a journalistic account of a small island off Italy in which people live to be quite old. Could it be because they sleep late, or drink lots of wine, or live a communal existence, or don’t eat refined sugar? Unfortunately there is no way to know based on the information provided in the article if it is one or more of the lifestyle characteristics the author identified as distinctive, or whether something else is going on. Even the claim of unusual longevity is questionable since there is no birth registry. And taking everything at face value, maybe there are other places in Italy in which people live as long or longer with different lifestyles.
Whitehurst goes on to question whether Union City is really such a successful district. And then he applies the problems of his Italy example to Kirp’s use of Union City to make causal conclusions about how school improvement can be achieved. Lastly, Whitehurst concludes with:
Once we have valid descriptions of the distinctive operational differences between good and not-so-good schools, controlling for differences in student background and out-of-school factors that are beyond district control, the social science of district reform can move to planned and carefully evaluated interventions. That is our playbook. That is where we need to be. I don’t think the path goes through Union City.
Amen, Brother Russ.
-Jay P. Greene
