I’ve written several times recently about how short term gains in test scores are not associated with improved later life outcomes for students. Schools and programs that increase test scores quite often do not yield higher high school graduation or college attendance rates. Conversely, schools and programs that fail to produce greater gains in test scores sometimes produce impressive improvements in high school graduation and college attendance rates, college completion rates, and even higher employment and earnings. I’ve described at least 8 studies that show a disconnect between raising test scores and stronger later life outcomes.
Well, now we have a 9th. Earlier this month MDRC quietly released a long-term randomized experiment of the effects of the SEED boarding charter school in Washington, DC. Because SEED is a boarding school, there was a lot of hope among reformers that it might be able to make a more profound difference for very disadvantaged students by having significantly more time to influence students and structure their lives. Of course, boarding schools also cost significantly more — in this case roughly twice as much as traditional non-residential schools.
While the initial test score results are very encouraging, the later life outcomes are disappointing. After two years students admitted to SEED by lottery outperformed those denied admission by lottery by 33% of a standard deviation in math and 23% in reading. If we judged the quality of schools entirely based on short term changes in test scores, as many reformers would like to do, we’d say this school was doing a great job.
In fact, SEED may be doing a great job in a variety of ways, but when we look at longer term outcomes for students on a variety of measures the evidence demonstrating SEED’s success disappears or even turns negative. Of the students accepted by lottery to SEED 69.3% graduate from high school after four years compared to 74.1% for the control group, a difference that is not statistically significant. And when asked about their likelihood of attending college, there was no significant difference between the two groups. SEED students also score significantly higher on a measure of engaging in risky behavior and lower on the grit scale.
We’ve seen this pattern before. Research by Marty West and colleagues of no excuses charter schools in Boston found large gains in test scores but also significantly lowered student performance on noncognitive measures. And Josh Angrist and colleagues found that those schools actually decrease four year high school graduation rates despite large gains in test scores. In their words:
Perhaps surprisingly given the gains in test score graduation requirements reported in column 2 of table 4, the estimates in column 4 of this table suggest not. In fact, charter attendance reduces the likelihood a student graduates on time by 14.5 percentage points, a statistically significant effect.
It’s time that people start paying a lot more attention to this pattern of a disconnect between short term test score gains and long term life outcomes. We can’t just dismiss this pattern as fluke. And the reduction in noncognitive skills may be important for explaining this pattern. Reduced grit scores may not just be the product of reference group bias. It appears that certain types of charter schools that are able to produce large test score gains also lower character skills and fail to yield long term improvements in life outcomes. Conversely other types of charter and private schools in choice programs fail to improve test scores but yield large gains in later life outcomes.
If we think we can know which schools of choice are good and ought to be expanded and which are bad and ought to be closed based primarily on annual test score gains, we are sadly mistaken. Various portfolio management and “accountability” regimes depend almost entirely on this false belief that test scores reveal which are the good and bad schools. The evidence is growing quite strong that these strategies cannot properly distinguish good from bad schools and may be inflicting great harm on students. Given the disconnect between test scores and later life outcomes we need significantly greater humility about knowing which schools are succeeding.
– Jay Greene
