For several decades pollsters have asked American citizens to grade the nation’s public schools, both nationally and within their local community.  Yet we know next to nothing about how citizens go about answering.

It is often noted, for example, that Americans tend to rate their local schools more favorably than those of the nation–much as they regard their own members of Congress quite highly while disdaining Congress as a whole.  At the same time, the 2008 EdNext-PEPG survey revealed that citizens assign far lower grades to their community’s schools than they do to its police force and post office.

But do the ratings on which these comparisons are based reflect a school’s actual performance?  Or do they instead reflect such factors as the racial or socioeconomic makeup of their students?

Matt Chingos, Mike Henderson, and I explore these questions in a new study (“Grading Schools”) from the Fall 2010 issue of Education Next.*  In a nutshell, we asked a nationally representative sample of American adults to identify their local elementary, middle, and high school and to grade them on a standard “A” to “F” scale.  We then linked the grades given to each school to data on the school’s characteristics: its size, the size of classes at the school, the  racial and ethnic composition of its students, the percentage of students from poor families, and the percentage of students performing at proficient levels on state reading and math tests.

Our findings are encouraging in many respects and troubling in others.  Let’s start with the positive:

1. Student achievement matters (especially to parents):  Citizen ratings of specific local schools do reflect publicly available information on the level of student achievement in those schools.  After adjusting for student demographics and other school characteristics, schools with 25 percentage points more proficient students are rated 22 percent of a letter grade higher.  Parents of school-aged children rate such schools nearly half a letter grade higher.

2. Race doesn’t matter (not even to parents):  Neither citizen nor parent ratings appear to be influenced by a school’s racial or ethnic composition.  This is not to say that high-minority schools do not receive lower grades (they do).  But this relationship dissipates once poverty rates and student achievement are also considered.

3. Poor and minority citizens are just as informed:  Although many have speculated that low-income and minority citizens are less informed about or interested in school quality than more advantaged groups, we found no evidence that this is the case.

So far, so good: it would appear that citizens (and especially parents) have the information they need to evaluate schools in a way that lines up with student performance.  As suggested above, however, other findings are more disconcerting:

1. School poverty matters: Unlike race, the share of a school’s students who are poor remains a strong predictor of citizen ratings even after taking into account student achievement.

2. Grades reflect achievement levels, not gains: For residents of Florida, we were able to check whether citizen ratings more closely reflect the level of student of achievement in a school or how much its students are learning over time.  We found that levels matter more–despite the fact that they are influenced by factors outside of the school’s control.

3. Differences in state standards are ignored: It is well known that the definition of proficiency varies widely from state to state.  Our main analysis deals with this fact by comparing only respondents within the same state.  But we also looked to see whether citizens take into account the difficulty of their state’s standards when assigning school grades.  The short answer: they don’t.  A school with an 80 percent proficiency rate in, say, Massachusetts is rated no more highly than a school with the same proficiency rate in Texas–despite the fact that the state test in Massachusetts is far more difficult.

This last finding is of special interest given the ongoing push for common standards across states.  It may be that a common definition of proficiency would increase pressure for reform in states where many students perform poorly relative to the nation as a whole but are deemed proficient by their state.  U.S. Secretary of Education Arne Duncan has said that many states are “lying to children and parents” by setting low standards.  Our evidence suggests that parents are believing them.

NB: I discuss the findings of the study with Ed Next’s Paul Peterson in this video: How Good Are Parents At Rating Schools?

* Readers interested in the details of the analysis can find the full PEPG Working Paper here.