Students learn principally through interactions with people (teachers and peers) and instructional materials (textbooks, workbooks, instructional software, web-based content, homework, projects, quizzes, and tests). But education policymakers focus primarily on factors removed from those interactions, such as academic standards, teacher evaluation systems, and school accountability policies. It’s as if the medical profession worried about the administration of hospitals and patient insurance but paid no attention to the treatments that doctors give their patients. With over half of fourth graders doing math problems from their textbooks daily, we surely ought to care about what’s in those books.
There is strong evidence that the choice of instructional materials has large effects on student learning—effects that rival in size those that are associated with differences in teacher effectiveness. For example, in a large-scale methodologically rigorous evaluation of the differential impact of four leading mathematics curricula, second-grade students taught using Saxon Math scored on average 0.17 standard deviations higher in mathematics than students taught using Scott Foresman-Addison Wesley Mathematics. By way of comparison, the difference in the impact on student achievement of a teacher at the 75th percentile of effectiveness compared to an average teacher is only 0.11 to 0.15 standard deviations. But whereas improving teacher quality through changes in the preparation and professional development of teachers and the human resources policies surrounding their employment is challenging, expensive, and time-consuming; making better choices among available instructional materials should be relatively easy, inexpensive, and quick.
Administrators are prevented from making better choices of instructional materials by the lack of evidence on the effectiveness of the materials currently in use. The vast majority of materials either have no studies of their effectiveness or have no studies that meet reasonable standards of evidence. Not only is little information available on the effectiveness of most instructional materials, there is also very little systematic information on which materials are being used in which schools. In every state except one, it is impossible to find out what materials districts are currently using without contacting the districts one at a time to ask.
This scandalous lack of information will only become more troubling as two major policy initiatives—the Common Core standards and efforts to improve teacher effectiveness—are implemented. Publishers of instructional materials are lining up to declare the alignment of their materials with the Common Core standards using the most superficial of definitions. The Common Core standards will only have a chance of raising student achievement if they are implemented with high-quality materials, but there is currently no basis to measure the quality of materials. Efforts to improve teacher effectiveness will also fall short if they focus solely on the selection and retention of teachers and ignore the instructional tools that teachers are given to practice their craft.
In our Brookings Institution report, we show how this problem can be fixed by states with support from the federal government, non-profit organizations, and private philanthropy. First, state education agencies should collect data from districts on the instructional materials in use in their schools. The collection of comprehensive and accurate data will require states to survey districts, and in some cases districts may need to survey their schools. In the near term, many states can quickly glean useful information by requesting purchasing reports from their districts’ finance offices. Building on these initial efforts, states should look to initiate future efforts to survey teachers, albeit on a more limited basis.
The federal government’s National Center for Education Statistics should aid states in this effort by developing data collection templates for them to use through its Common Education Data Standards (CEDS), and providing guidance on how states can use and share data on instructional materials. The most recent version of CEDS contains 679 data elements for K–12 education, none of which relate to instructional materials in use.
Organizations with an interest in education reform should support this effort. For example, the National Governors Association (NGA) and Council of Chief State School Officers (CCSSO) have put their reputations on the line by sponsoring the Common Core State Standards Initiative. Research based on current and past state standards indicates that this initiative is unlikely to have much of an effect on student achievement in and of itself. The NGA and CCSSO should put their considerable weight behind the effort to improve the collection of information on instructional materials in order to create an environment in which states, districts, and schools will be able to choose the materials most likely to help students master the content laid out in the Common Core standards.
States facing severe budgetary pressures may be reluctant to undertake new data collection efforts. Philanthropic organizations such as the Bill and Melinda Gates Foundation and the Lumina Foundation for Education could have a major impact by providing the start-up funding needed to collect data on instructional materials and support the research that would put those data to use.
In 1955, educational psychologist Lee J. Cronbach wrote that “The sheer absence of trustworthy fact regarding the text-in-use is amazing.” It is more than a half-century later and we still don’t know. How can we tolerate ignorance on something that is as critical to student learning as instructional materials?
Matthew M. Chingos and Grover J. (Russ) Whitehurst, who are research director and director of the Brown Center on Education Policy at the Brookings Institution, are the authors of Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core.
