CHECKED: Robert C. Pianta and Arya Ansari, “Does Attendance in Private Schools Predict Student Outcomes at Age 15? Evidence from a Longitudinal Study,” Educational Researcher, Online First.
Checked by Patrick J. Wolf
“New Study Finds Low-Income Students Do Not Benefit from Private Schooling: Findings refute administration’s call for $1 billion for private school vouchers.” Such was the headline of a press release issued by the University of Virginia’s Curry School of Education. Not to be outdone, perennial school choice critic Valerie Strauss of the Washington Post announced, “No, Private Schools Aren’t Better at Educating Kids than Public Schools: Why this study matters.”
Of course, headlines about studies sometimes exaggerate the findings to draw attention to researchers and their research. In the text of the release, however, lead author Robert C. Pianta, Dean of the Curry School, states that the “assumption that private schools are more effective…is demonstrably ineffective and potentially harmful.”
So, has this new study, forthcoming in the prestigious journal Educational Researcher, proven that private-school vouchers harm children? Hardly. It isn’t even a study of school vouchers. It isn’t designed to determine what caused the student outcomes it examines. Its findings are inconclusive, not negative. And, finally, it doesn’t have a large enough sample to prove much of anything.
The study grew out of a larger project designed to measure an extensive set of school factors and find out which ones correlate with student outcomes. A comparatively small sample (more on that later) of 1,067 children who were newborns in 1991 were followed until age 15 in the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development. Participants were recruited from 10 cities across the country, none of which was home to a private-school voucher program during any of the 15 years of data collection. The research team measured students’ achievement, psychological well-being, and expectations for reaching various educational attainment goals, as well as their school type and family background. The measures of psychological well-being, including externalizing or internalizing behavior, social skills, risky behaviors, victimization, and future outlook, are especially important and distinctive features of this study. Data collection took place in four waves.
The authors, Pianta and his colleague Arya Ansari, analyzed the data observationally, meaning they looked for correlations between their variables of interest—whether a student had attended a private school between kindergarten and 9th grade and, if so, for how many years—and student outcomes. They report that the simple correlations between private schooling and desirable student outcomes are positive and statistically significant for every outcome measure. When variables are included in statistical models to control for student and family background characteristics, however, they find that all of the correlations between private schooling and student outcomes become statistically insignificant. It is from this “no effects of private schooling” finding that Pianta concludes that private schooling through vouchers harms children.
It bears repeating that the Pianta-Ansari study is not of a school-voucher program. Their unrepresentative sample of the private schools in 10 cities does not include a single private school serving a single student with a school voucher. The authors seem to admit to limitations in their private-school sample when they state that a “second consideration in evaluating private-school effects is the exceptionally wide variation in private schooling across the United States.” They can’t generalize their findings to all private schools in the country, and if their findings don’t necessarily apply to all private schools, then they almost certainly don’t apply to the distinctive set of private schools that educate students in voucher programs. The authors disregard their own concerns, however, and irresponsibly generalize the findings from private schools not participating in voucher programs in the 1990s and early 2000s to private schools participating in voucher programs now and in the future.
A second limitation of this study is its non-experimental research design. I have also performed a few non-experimental studies of school vouchers but only when an experiment proved to be infeasible. The authors acknowledge that experimental studies using random assignment of students to treatment and control groups “to support causal inferences regarding the impacts of private schooling are among the strongest scientific examinations of such effects.” They justify their use of a weak non-experimental design by claiming that prior school-voucher studies draw similar conclusions, whether the studies are experimental or non-experimental. That claim is false.
In a chapter in the forthcoming book School Choice at the Crossroads, I present the results of a systematic review of the existing research on school vouchers, which demonstrates that the academic effects of vouchers from experimental studies are more positive than those from non-experimental studies such as this one. In a reanalysis of the data from the first evaluation of the D.C. school voucher program, Kaitlin Anderson and I show that the results produced by non-experimental analyses are less positive than the true causal impacts revealed experimentally. If rigorous experimental findings point in one direction while less-rigorous observational findings point in the other direction, we should believe the experiments. The authors of the Curry School study instead privilege their non-experimental results regarding “non-vouchers” over the stronger and different experimental findings.
Third, there is no logic to the claim that non-significant findings from a study suggest that an education intervention harms children. No effect means no effect, not a negative effect.
Finally, the main findings of the study were essentially preordained given important features of its sample and analysis. Not only is the study sample unrepresentative of the private-school population nationally, but it is also different in ways that bias the findings against private schools. Only 45 percent of the study targets actually participated, for example. Children of non-English-speaking mothers, a large subpopulation in many school-voucher programs, were excluded from the sample. Also absent were cities famous for their large and reputable private-school sectors, such as New York, Indianapolis, and Minneapolis-St. Paul. What the authors found is that, more than a decade ago, private schools not known for being particularly effective may not have been particularly effective.
The researchers analyze outcomes from only 1,097 students, which they describe as a relatively large sample but is small in comparison to other school-voucher evaluations currently underway in D.C., Louisiana, and Indiana. The authors report a lot of missing data for these students on the “type of school attended” variable, which is central to their study. Whether or not a student attended a private school was unknown for between 7 and 16 percent of the students annually, depending on grade, leaving only 90 percent of the students with even six years of certain data regarding private schooling, out of a possible 10 years. The authors then use the least-scientific method available to account for these missing data, essentially assuming that a student attended a public school for any year in which their school type was unknown, and thus introduce extensive measurement error into their key variable of interest. This measurement error increases the likelihood of non-significant results, especially when analysts introduce control variables with less measurement error into their statistical models, as the authors do here.
The authors claim their study had the statistical power to detect private-school effects as small as 10 percent of a standard deviation overall and 20 percent for subgroups. Taken at face value, this implies that they would fail to detect school-voucher effects of the size typically found in actual voucher studies, which average 7 to 12 percent of a standard deviation. Moreover, their own results disprove their claim about the statistical power of their study, as a handful of private-school effects larger than 10 percent of a standard deviation are not statistically significant in their article’s crucial Table 4. An extensive set of study limitations converged to make the non-significant findings a virtual certitude, regardless of how effective the private schools actually were.
Pianta and Ansari conclude: “In sum, we find no evidence for policies that would support widespread enrollment in private schools, as a group, as a solution for achievement gaps associated with income or race.” Their lack of significant findings was obtained from an outdated, non-experimental, underpowered, sample-of-convenience analysis of places and people that were not participating in actual private-school voucher programs. They add the requisite, “findings should be interpreted with caution.” Readers would be wise to follow the sage advice that the authors themselves ignored.
Patrick J. Wolf is professor in the department of education reform at the University of Arkansas.
