It has been almost a decade since the publication of Education Myths.  At that time I was concerned that education policy was largely driven by emotional appeals and assumed facts.  ”Don’t you love children?” and “we know education spending is plummeting” used to be regular arguments in policy discussions.  Ed Myths was a response to that type of policy argument and an attempt to demonstrate how systematic evidence could be used to inform policy-making.  I (along with co-authors Greg Forster and Marcus Winters) understood that the particular evidence we were citing would soon enough be out of date, but we hoped that our approach to using rigorous evidence could serve as a model for future policy debates.

Much progress has been made over the last nine years in the use of systematic evidence in education policy-making.  Mountains of data are being collected and scores of well-trained social scientists are applying cutting edge techniques and clever research designs to draw useful lessons from those data.  And policy-makers increasingly rely on this rigorous evidence when making decisions.  Emotional appeals and assumed facts have diminished in their influence.

Despite this progress, the field just can’t seem to shake the enduring attraction of the flim-flam man.  The flim-flam man resembles a social scientist and cites evidence to make his case, but often relies on faulty evidence drawn from flawed research designs as well as selective and distorted interpretations of evidence.  The flim-flam man uses the veneer of social science to disguise an agenda-driven or ill-conceived argument.

The most common manifestation of the flim-flam man in ed policy is the practitioner of “selection on dependent variable” arguments.  This is the person who says that Finland or South Korea or Massachusetts produce really great results, so we should do something that they do to make similar progress.  Of course, it is impossible to know from an examination of successful places why they are successful.  For all I know, Finland’s success is a function of the heavy concentration of reindeer, South Korea thrives because of delicious kimchee, and Massachusetts is blessed with excellent human capital to staff its schools and has had super folks like Sandra Stotsky and Bob Costrelll to direct its efforts.

Just because I choose to focus on Finland’s rigorous teacher preparation, South Korea’s emphasis on test-preparation over athletics, and Massachusett’s standards does not mean that those factors caused the success in question.  Identifying causation requires, at a minimum, observing that certain practices or policies tend to be present where there is success and absent where there is failure.  Only looking at these successful places tells us nothing about why they are successful because we do not know if those same practices or policies also existed in other, unsuccessful places.

The flaw of selection on dependent variable analyses is so obvious that it is shocking that education policy debates continue to be shaped by them.  I know that it is tempting to look at some place that is doing well and wonder what you could imitate to get the same results, but we just need to stop considering that evidence in education policy debates.

We should take a pledge — No more selection on dependent variable analyses!  No more divining the secrets of success in Finland!  Anyone who continues to present this kind of argument as evidence should be shunned as a flim-flam man.  A quack.

I take some heart from seeing that folks have started to critique Malcolm Gladwell as a flim-flam man.  Yes, he writes well.  Yes, he tells appealing stories.  But his reading of social science evidence is thin, selective, and often distorted. :

Mr. Gladwell has not changed his own strategy, despite serious criticism of his prior work. What he presents are mostly just intriguing possibilities and musings about human behavior, but what his publisher sells them as, and what his readers may incorrectly take them for, are lawful, causal rules that explain how the world really works. Mr. Gladwell should acknowledge when he is speculating or working with thin evidentiary soup. Yet far from abandoning his hand or even standing pat, Mr. Gladwell has doubled down.

If psychology is turning on Gladwell, then maybe education can turn on its own flim-flammers.

-Jay P. Greene