Just ten years ago, the only data available on the impact of school vouchers came from a poorly designed public-choice program conducted during the 1960s in Alum Rock, California. But the early and mid-1990s brought new privately and publicly funded voucher programs to cities such as Milwaukee; Dayton; Cleveland; Indianapolis; San Antonio; Washington, D.C.; and New York City. With them came a wealth of new research opportunities.

The foundation in New York City offered 1,300 scholarships, each worth up to $1,400 annually toward tuition at a private school.

The privately funded voucher programs in New York City, Dayton, and the District of Columbia are especially conducive to study. In each city, vouchers were awarded randomly, generating treatment and control groups that are statistically indistinguishable from one another. Before conducting the lotteries, our evaluation team collected data on student test scores and family background characteristics. One and two years later, we retested the students. Since the two groups of students-the lottery’s winners and losers-had similar average abilities and family backgrounds, any subsequent achievement differences observed between them can be attributed to the effects of the vouchers.

As a result, our evaluations of the New York, Dayton, and D.C. voucher programs have yielded the best available information on students’ test-score outcomes and parental assessments of public and private schools. Here we use the data from all three cities to analyze the one- and two-year effects on academic performance of switching from a public to a private school. We find that vouchers have a moderately large, positive effect on the achievement of African-American students, but no discernible effect on the performance of students of other ethnicities.

The Literature

Earlier comparisons of public and private schools generally have found that low-income and African-American students who attend private schools outperform their public-school peers. For instance, University of Wisconsin economist Derek Neal’s analysis of the National Longitudinal Survey of Youth found that, even after adjusting for family background characteristics, students from Catholic schools were 16 percentage points more likely to go to college than were public-school students. The gap between Catholic-school students and public- school students was largest among urban minority children. Other studies have reached similar findings. University of Wisconsin political scientist John Witte’s review of the literature on school effects led him to conclude that studies of private schools “indicate a substantial private-school advantage in terms of completing high school and enrolling in college, both very important events in predicting future income and well-being.”

All of these studies, however, have one important limitation. They can account for only observed family background characteristics, such as the mother’s educational level, a student’s ethnicity, or family income. There is no assurance that these studies have successfully controlled for an intangible factor: the willingness of parents to pay tuition to send their children to private school and all that this implies about the value they place on education. As a result, it remains unclear whether these studies have unearthed actual differences between public and private schools or simply differences in the kinds of students and families attending them.

The best way to compensate for this limitation is to assign students randomly to experimental and control groups whose only substantive difference is whether they are offered a voucher. Past evaluations of voucher programs have not been able to take full advantage of a random-assignment research design. Consequently, the findings from New York, Dayton, and D.C. provide a unique opportunity to examine the effects of school vouchers.

The Programs

In several key respects, the three voucher programs followed similar designs. All were privately funded; all were targeted at students from low-income families, most of whom lived in the inner city; all provided only partial vouchers, expecting the families to supplement them; and all of the students in the evaluations previously had been attending public schools. Brief descriptions of the three programs follow.

New York City. The School Choice Scholarships Foundation (SCSF) in New York City offered 1,300 scholarships worth up to $1,400 annually toward tuition at a private school for at least three years. To qualify for a scholarship, children had to be entering grades 1 through 4, live in New York City, attend a public school at the time of application, and come from families with incomes low enough to qualify for the U.S. government’s free or reduced-price school-lunch program. More than 20,000 students applied between February and late April 1997. By the end of the scholarship program’s second year, 64 percent of the lottery-winning students were attending a private school.

Dayton, Ohio. In the spring of 1998, Parents Advancing Choice in Education (PACE) offered low-income students in grades K-12 the opportunity to win a scholarship to attend private school. For the 1998-99 school year, PACE offered scholarships to 515 students who were in public schools and to 250 who were already enrolled in private schools in the Dayton metropolitan area. During the program’s first year, the PACE scholarships covered 50 percent of tuition at a private school, up to $1,200. Support was guaranteed for at least four years, with a possibility of continuing through high school, provided funds remained available. Of those students offered scholarships, 49 percent enrolled in a private school during the second year of the program.

Washington, D.C. Established in 1993, the Washington Scholarship Fund (WSF) is the oldest of the three programs. By the fall of 1997, the WSF was serving approximately 460 children at 72 private schools. On receiving a large infusion of new funds from two philanthropists, the WSF announced a major expansion in October 1997.

To qualify, applicants had to reside in Washington, D.C., and be entering grades K-8 in the fall of 1998. Families with incomes at or below the poverty line received vouchers that equaled 60 percent of tuition or $1,700, whichever was less. Families with incomes above the poverty line received smaller scholarships. Families with incomes higher than two-and-a-half times the poverty line were ineligible. The WSF claims that it will maintain tuition support for at least three years and, if funds remain available, until students complete high school. In April 1998, the WSF awarded more than 1,000 scholarships by lottery, with the majority going to students previously attending a public school. Of those students offered scholarships, 35 percent were still using them to attend a private school in the second year of the program.

Evaluation Procedures

The evaluation procedures used in all three studies conformed to those used in randomized field trials. Our evaluation team collected baseline test scores and family background information before the lottery, administered the lottery, and collected follow-up information one and two years later.

Students took the Iowa Test of Basic Skills (ITBS) in reading and mathematics. Students who were entering grades 1-4 in New York City and grades 2-8 in Dayton (and other parts of Montgomery County, Ohio) and Washington, D.C., were included in the evaluations. Parents responded to survey questions about their satisfaction with their children’s schools, their involvement in their children’s education, and their demographic characteristics. Students in grades 4 and higher completed similar surveys. In all three cities, the follow-up procedures replicated the pre-lottery procedures: students again took the ITBS in reading and math; parents and older students filled out surveys about their backgrounds and educational experiences.

More than 5,000 students participated in pre-lottery testing in New York City. Of the families that did not win the lottery, approximately 1,000 were selected at random to compose a control group of approximately 960 families. All of these students were attending public-schools at the time. In Dayton, 1,440 students were tested before the lottery; 803 of them were attending public schools at the time. In Washington, D.C., 2,023 students were tested before the lottery; 1,582 of them were attending a public school. In Dayton and in D.C., separate lotteries were held for students who were enrolled in public and private schools at the time of application. The fact that only public school children were eligible to apply for a scholarship in New York obviated the need to hold separate public and private lotteries there. In all three cities, only those students who were in public schools at the time of the lottery are included in this study.

In New York City, 42 percent of the students participating in the second year of the evaluation were African-Americans; in Dayton, 74 percent; and in D.C., 94 percent. Hispanic students accounted for 51 percent of the New York City group and 2 percent and 4 percent of the Dayton and D.C. groups, respectively. Whites accounted for 5 percent of New York City’s evaluation group, versus 24 percent in Dayton and 1 percent in D.C. The remaining students came from a variety of other ethnic backgrounds.

In New York City, 80 percent of the students included in the evaluation attended the first-year testing sessions; 66 percent attended the second-year sessions. In D.C. the response rate after one year was 63 percent; after two years, 50 percent. In Dayton, 57 percent of families attended follow-up sessions after one year, 49 percent after two years.

We are reasonably confident that these modest response rates do not undermine the integrity of our findings. First, with the exception of the second year in New York, response rates were similar for both the treatment and the control groups after one and two years in all three cities. Second, comparisons of baseline test scores and background characteristics revealed only minor differences between the composition of the test and control groups in all three cities. Finally, to account for the minor differences between respondents and nonrespondents that we did observe, the test scores of children who, based on their demographic characteristics, were more likely to attend follow-up sessions were weighted less heavily, while the test scores of children who were less likely to attend follow-up sessions, but nevertheless did, were weighted more heavily. Given the slight differences between respondents and nonrespondents, however, the weights had little effect on the results.

Vouchers have a moderately large, positive effect on the achievement of African-American students, but no discernible effect on the performance of students of other ethnicities.

The randomized lottery ensured that lottery winners as a group were not significantly different from the control group (those who did not win a scholarship). In all three cities, the demographic characteristics and pre-lottery test scores of scholarship winners and losers (the treatment and control groups, respectively) resembled one another. Only in Dayton were there minor differences in the pre-lottery test scores: those offered a voucher scored 6.5 percentile points lower in math and 3.1 points lower in reading than those not offered a scholarship, a statistically significant difference.
To measure the effect on children’s test scores of switching to a private school, we estimate a statistical model that takes into account whether a child attended a public or a private school, as well as baseline reading and math test scores. Baseline test scores were included to adjust for the minor baseline differences between the treatment and control groups on the achievement tests and to increase the precision of the estimated impact.

The lottery generated two groups: those who were offered a voucher and those who were not. We’re not interested, however, in the effect of being offered a voucher. Rather, we’re interested in the effect of using a voucher to attend a private school. A significant number of the students who were offered vouchers did not use them; similarly, a smaller proportion of those students not offered a voucher attended a private school anyway. Therefore, a simple comparison between public and private school students is inappropriate because certain students may be more likely to take advantage of a voucher. Their parents may place greater value on education and be more willing to supplement the voucher, or they may live in a neighborhood with a broader selection of private schools. If these children differ from students who won a voucher but failed to use it in ways that are related to student achievement, it could bias our findings. To solve this problem, we used as an instrumental variable whether or not a student was offered a voucher to predict the probability that she attended a private school; with these predicted values, we can provide an unbiased estimate of the actual impact of switching from a public- to a private-school. This two-stage regression technique was first used in medical research and is now commonplace in econometric studies.

Results

Our findings varied by ethnic group. In all three cities, there were no significant differences between the test-score performance of non-African-American students who switched from a public to a private school and the performance of students in the control group-after either one or two years. For African- American students, however, the receipt of a voucher made a substantial difference. In the three cities combined, African-American students who switched from public to private schools scored, after one year, 3.3 percentile points higher on the combined math and reading tests (expressed as National Percentile Ranking [NPR] points, which run from 0 to 100 with a national median of 50). After two years, African- American students who used a voucher to enroll in a private school scored 6.3 percentile points higher than African-American students who remained in public schools (the control group) (see Figure 1).

We emphasize the overall test scores, which represent the average of the math and reading components. When using one-hour testing sessions to gauge student performance, combined reading and math scores serve as a better indicator of student achievement than either test separately. Theoretically, the more test items used to evaluate performance, the more likely performance will be measured accurately.

Nevertheless, the differences after two years were approximately the same for both the reading and the math tests. On average in the three cities, African-American students who switched from public to private schools scored 6.3 percentile points higher than their peers in the control group on the reading portion of the test and 6.2 points higher on the math portion.

The largest test-score differences between African-American students in private schools and African-American students in public schools were observed in the D.C. program. Black students who attended D.C. private schools for two years scored 9.0 percentile points higher on the two tests combined than did students in the control group. The smallest differences after two years were observed in New York City, where the combined test scores of African-American students attending private schools were 4.3 percentile points higher than those of the control group. In Dayton the difference was 6.5 percentile points for African-American students.

The trend over time also varied from city to city. In New York City, at the end of the first year, African-American students in private and public schools displayed substantial differences in test scores, but these diminished slightly in the second year. After two years the difference in scores is 4.3 percentile points, which is slightly but not significantly (in statistical terms) less than the 5.8 percentile point difference observed after one year. It is reasonable to conclude that African-American students’ initial gains in the New York City school voucher program were preserved but did not increase between year one and year two.

In Dayton, there appears to be a steady upward trend in the combined test-score performance of African-Americans. African-American students who switched from public to private schools performed 3.3 percentile points higher on the combined test in year one and 6.5 percentile points higher in year two.

In some ways, the most striking results in terms of trends over time concern African-Americans in D.C. After one year, no significant differences were observed for African-American students as a group, but older and younger students experienced significant differences. While younger students may have benefited slightly from the voucher program after one year, the older students who switched to private schools scored significantly lower than their public- school peers after one year. By the end of the second year, however, these students seemed to have overcome the initial challenges of changing schools. Both younger and older African-American students who switched from public to private schools posted positive and significant gains. On the combined reading and math tests, younger students in private schools scored 9.3 percentile points higher than those who remained in public schools. Older African-American students in private schools scored 10.3 percentile points higher.

The lottery ensured that scholarship winners as a group were not significantly different from those who did not win a scholarship. In all three cities, the demographic characteristics and pre-lottery test scores of scholarship winners and losers resembled one another.

Controlling for Demographics

Most research on the impact of private schools attempts to control for differences in family income and other background characteristics among students attending public and private schools. When a lottery is used to assign research subjects to experimental and control conditions, however, such statistical adjustments are generally unnecessary simply because the two groups being compared are virtually identical.

After two years, African-American students who used a voucher to enroll in a private school scored 6.3 percentile points higher than African-American students who remained in public schools.

Nonetheless, after the release of our study, some analysts objected to the apparent absence of controls for family background characteristics. Bruce Fuller and his colleagues at the University of California, Berkeley, for instance, argued, “The experimental group may have been biased as some of the most disadvantaged voucher winners did not switch to a private school, and therefore were excluded from the group (possibly boosting mean achievement levels artificially).” An interest group, People for the American Way, lodged a similar complaint: “The study’s key finding improperly compares two dramatically different groups and may well reflect private-school screening-out of the most at-risk students.”

In the three cities roughly half the students initially took the voucher that was offered to them (the takers), and about half did not (the decliners). Takers had higher family incomes in New York and D.C., but lower incomes in Dayton. The New York and D.C. findings are not surprising, given that the voucher awards did not cover all the costs of a private education. These additional costs were the reason most frequently given by families for not using the voucher. Presumably acceptance rates would rise if the monetary value of the vouchers were increased.

However, we did not drop the decliners from the analysis, as some of our critics have charged. All voucher applicants were invited to follow-up testing sessions, and each of the families who participated, including those who declined a scholarship, is included in the analysis. To estimate the impact of switching from a public to a private school, we did not simply compare those students who used a voucher to enroll in a private school with all those who did not. Such a comparison would have introduced bias and squandered all the advantages of a random-assignment evaluation. Instead, we used a familiar technique, often used in medical and econometric research, that preserves the essence of a random-assignment evaluation. The outcome of the lottery, a random event, was used to create what statisticians refer to as an instrumental variable, which obtains unbiased estimates of the effects of attending private school on students’ test scores. According to the statistical theory that underpins this technique, results from lotteries are powerful instrumental variables, because the lottery, being a random event, is not directly related to students’ test-score performance. In other words, the use of this statistical technique fully corrects for any differences that arise from the fact that not all of the families who were offered a voucher made use of one.

When similar results emerge from school voucher programs in three very different cities, we can be fairly confident that the intervention is the main cause of differences in achievement.

To see whether the instrumental variable worked in practice as it should in theory, we conducted a second analysis in which we controlled not only for the students’ pre-lottery test scores but also for their mothers’ educational level, her employment status, family size, and whether the family received welfare. If the critics were correct, the introduction of these background characteristics into the analysis should have diminished the estimated effect of attending a private school, because only after these adjustments were made would the analysis have adjusted for the background differences between those who used the voucher and those who did not. But if the use of the lottery as an instrumental variable works in practice as it is expected to work in statistical theory, it would already have corrected for these differences. The results should remain essentially the same.

As statistical theory anticipates, the average difference in the combined reading and math test scores of African-Americans in all three cities remained exactly the same-6.3 NPR points-after the adjustments for family background characteristics were introduced. Minor differences in the two estimates were observed within each city. The impact of switching to a private school without controlling for family background in New York City was originally estimated to be 4.4 NPR points; after accounting for family background, the impact was estimated to be 4.2 NPR points. Introducing controls in Dayton decreased the estimated impact from 6.5 to 5.9 NPR points. In Washington, D.C., the estimated impact increased from 9.0 to 9.1 NPR points. In New York and Washington, the estimated impacts, after adding controls for family background, remain statistically significant. In Dayton, the impact just missed the standard threshold for statistical significance.

Discussion

It is possible that conditions specific to each city or minor fluctuations in testing conditions might skew results one way or another. But when similar results emerge from the evaluations of school voucher programs in three very different cities, we can be fairly confident that the intervention is the main cause of the differences in achievement.

In general, we found no evidence that vouchers significantly improved the test scores of ethnic groups other than African-Americans, most notably Latinos in New York and whites in Dayton. The impact of vouchers for African-Americans, however, was moderately large. After one year, black students who switched to private schools scored 0.17 standard deviations higher than the students in the control group. After two years, the difference grew to 0.33 standard deviations, roughly one-third of the test-score gap between blacks and whites nationwide. These effects are approximately the same as those observed in Tennessee when class sizes were reduced from 24 students to 16 students, a much more costly intervention.

Whether the gains from these small, private scholarship programs will translate to large-scale, publicly funded school-choice programs in urban areas is unknown. Only a small fraction of low-income public-school students in New York, Dayton, and D.C. were offered vouchers, and these students made up a small share of the cities’ private-school populations. A much larger program carried out for longer periods of time could yield quite different outcomes. But we’ll never know unless we try. The nation’s capital, the city where the largest effects were observed, would be a good place to begin.

-William G. Howell is an assistant professor of political science at the University of Wisconsin-Madison. Patrick J. Wolf is an assistant professor of public policy at Georgetown University. Paul E. Peterson directs the Program on Education Policy and Governance at Harvard University, where David E. Campbell is a research associate. To view their study in its entirety, log on to www.edmattersmore.org.