“We know what works” has been the traditional refrain of the teachers unions.  If only everyone else would just do what we think is best, they suggest, education would be much improved.  With its rating of teacher prep programs, the National Council on Teacher Quality has joined the “we know what works” chorus — the only difference being that NCTQ is singing from a different hymnal.  Unfortunately, NCTQ is no more able to identify the true faith of education policies and practices than are the teachers unions.  Education reform would benefit from less false confidence in knowing what works and more encouragement of experimentation, choice, and competition.

Given that there are competing visions of the one true way, it’s reasonable for skeptics to wonder how NCTQ knows that there is a correct approach to preparing teachers and that they have managed to discover it.  NCTQ claims that the 18 standards they developed to judge ed schools are well-established criteria:

Our standards for the first edition of the Teacher Prep Review are based on research; internal and external expert panels; the best practices of other nations and the states with the highest performing students; and, most importantly, what superintendents and principals around the country tell us they look for in the new teachers they hire.

So, NCTQ claims that it knows the right way to prepare teachers because it is relying on research, the authority of experts and school leaders, and their observation of the “best practices” among successful organizations.

The latter two methods for identifying the right way to prepare teachers are obviously unreliable.  There is no consensus among experts, whether they are researchers or school leaders, about the correct way to operate an ed school.  So how does NCTQ know that it has selected the right experts to identify the right way to prepare teachers?  And how do the experts they choose to consult know what the correct approach is?  When self-described experts have strongly divergent visions, organizing panels of the experts with whom NCTQ happens to agree does not resolve the dispute.  It only recycles NCTQ’s opinion in the disguise of expert authority.

Also, I’ve written previously about the fundamental flaw of relying on the “best practices” of successful organizations.  To repeat briefly the argument, best practice methods that only look at successful organizations lack variation in the dependent variable.  We can never know from best practice approaches whether organizations are successful because of — or in despite of — a particular practice because we do not see whether unsuccessful organizations also engaged in the same practice.  The identification of a causal relationship involves, at a minimum, discovering that the variation in a practice is related to variation in success.  Because best practice approaches truncate variation in the dependent variable by examining only successful organizations, they cannot identify the practices that caused success.

This leaves us with research as the major basis for NCTQ’s claim that they know how teacher prep should be done.  But by NCTQ’s own description, they have “strong research” supporting only 8 of the 18 standards by which they judge ed schools.  Unfortunately, even in those 8 cases where NCTQ claims to have strong research, they mis-read much of that literature, are selective among the “strong research” they choose to embrace, and often have to make a series of leaps from the research cited to support the standards.  The result is that NCTQ doesn’t really have 8 of their 18 standards supported by strong research, so they can’t claim to know the right way to run a teacher prep program based on their review of research.

Here’s an example of how NCTQ mis-reads the research: Standard 13 is “Equity” and judges teacher prep programs based on whether “The program ensures that teacher candidates experience schools that are successful in serving students who have been traditionally underserved.”  The only “strong research” NCTQ cites for support of the claim “that entering teachers learn crucial methods of instruction and management through observation of and supervised practice in schools where staff are successfully teaching students living in poverty” is a study by Matthew Ronfeldt.

Unfortunately, Ronfeldt’s study appears to make the opposite claim.  He finds that it is more important for student teachers to be trained in schools with low staff turnover that tend to have more advantaged students.  He concludes:

Should we place student teachers in “difficult-to-staff, underserved” schools to learn to teach? The main findings of this study suggest otherwise – learning to teach in difficult-to-staff field placement schools is associated with lower teacher effectiveness and retention. Moreover, the results demonstrate that being trained in field placements with higher concentrations of poor, black, and lowest-achieving students has no significant effect on teacher retention or effectiveness.

Another example: Standard 2 on “Early Reading” assesses whether “The program trains teacher candidates to teach reading as prescribed by the Common Core State Standards.”  Never mind that most people who support Common Core repeatedly insist that Common Core does not dictate curriculum or pedagogy.  NCTQ, which also supports Common Core, does not appear to be working off of the same talking points because they are under the impression that Common Core prescribes a particular approach to teaching reading.  Somehow NCTQ knows what reading pedagogy Common Core requires even while other supporters of Common Core deny that it requires any particular pedagogy or curriculum.

The only “strong research” they cite in support of the “Early Reading” standard examines whether teachers are familiar with the “five components of effective reading instruction,” and whether teachers who are certified and have masters degrees are more likely to know those five components.  It turns out teachers are generally not familiar with the five components and are no more likely to know them if they are certified or have a masters’ degree.

That’s all very nice, but isn’t the “strong research” supporting the standard supposed to show that knowledge of the five components actually leads to improved reading by students?  The research cited by NCTQ says it doesn’t: “This study also found no relationship between teachers’ knowledge of these components and their students’ reading growth – with the notable exception of third-grade students.”  So, even the research cited by NCTQ to support the “Early Reading” standard fails to find a broad relationship between the standard and desired outcomes.

A final example: Standards 1 and 6 address whether teacher prep programs select “teacher candidates of strong academic caliber” and whether “teacher candidates have the broad content preparation necessary to successfully teach to the Common Core State Standards.”  In both cases the “strong research” on which these standards rely is a study by Boyd, et al examining the relationship between teacher characteristics and student achievement.  NCTQ acknowledges that “strong research” by Harris and Sass as well as Peterson and Chingos fail to support these NCTQ standards.  But rather than describe the strong evidence as mixed, they disregard the contrary findings and choose to believe only the research that agrees with their view.

Even the Boyd, et al study does not support NCTQ as strongly as they suggest.  Boyd, et al warn against making causal claims given non-random assignment of teachers to students in their study.  In addition, Boyd, et al only examine whether the improvement of a basket of teacher characteristics was associated with improvement in student achievement in New York City.  They cannot isolate with confidence the effect of individual criteria, like whether a teacher is certified in the subject or has higher SAT scores because “many of the measures of teachers’ qualifications are highly correlated with each other.”  In fact, if you look at Table 4 in their paper you will find that teachers with higher verbal SAT scores are negatively associated with student gains in math achievement.  Boyd, et al are not surprised by this finding because verbal SAT scores are highly correlated with other measures in their model, but this hardly proves the case that ed schools should be graded by establishing higher SAT requirements for admission.

To be fair, NCTQ acknowledges that quality research on effective education practices is in short supply: “To the extent that high-quality research can inform how teachers should be prepared, NCTQ uses that research to formulate standards. Unfortunately, research in education that connects preparation practices to teacher effectiveness is both limited and spotty.”  But this lack of evidence does not prevent NCTQ from confidently declaring that they know what teacher prep programs should be doing and judging them on that basis.  If quality research is so limited, how does NCTQ know what everyone else is supposed to be doing?

I’m sure that there is considerable room for improvement in teacher prep programs.  Many of NCTQ’s recommendations are probably sensible, even if they aren’t backed by “strong research.”  The problem is not so much that NCTQ is suggesting bad ideas as that they are claiming to know much more than they actually know.  And they are willing to boss around everyone else despite not knowing as much as they think.

Maybe we’d make more progress in improving teacher prep programs if we were more upfront about what we didn’t know and encouraged more experimentation and data-collection so that we can learn more.  And given that different circumstances may call for different practices, maybe we should be open to a variety of Ed School approaches rather than attempting to impose the one true way on all teacher prep programs.

-Jay P. Greene

For a response by Kate Walsh, please read “A Response to Jay Greene“