Saw Brad Pitt’s new flick Moneyball the other week. Good, not great; thought the book was better. A lot of the interesting stuff gets lost in translation. I’ve noted the same thing when K-12 thinkers latch onto the “moneyball” analogy. K-12 enthusiasts point out that Billy Beane used sophisticated statistical analysis to build winning teams, and sensibly presume that the same kinds of tools can help drive school improvement. (Back in 2003, when the book was published, the edu-analogies consisted mostly of paeans to data dashboards; today, it’s all about “value-added” metrics.)
Here’s the problem. Author Michael Lewis made it real clear in the book (though it’s less clear in the movie, which features scouts talking about whether players have attractive girlfriends) that the problem in baseball prior to Beane’s revolution in Oakland was not an absence of data. In fact, baseball has been a geek haven for generations because of all its statistics. The problem? The stats in question–typically home runs, runs batted in, and batting average–are flawed measures of individual performance. They routinely understate (or overstate) a player’s value by ignoring the stadium he plays in, how often his teammates get on base, how selective he is at the plate, how well he fields, and so on. A big part of the problem wasn’t a lack of numbers; it was a reliance on overly simplistic measures. Consequently, players who hit a lot of home runs or who hit for a high average were massively overpriced, while players who walked a lot or hit a lot of doubles were undervalued.
This is where value-added enthusiasts come in. Value-added is a potentially very useful (if limited) tool, but it’s one that’s still in its relatively infancy. It can tell us what we might otherwise overlook or fail to see, helping correct our tendency to overvalue or undervalue certain teachers and techniques. The problem is our impatience and, sometimes, hubris. There’s a sense among too many would-be reformers that our new edu-statistics are ready for prime-time, and even an inclination to imagine that they can render judgment and common sense superfluous. Nope.
Look, it’s frustrating, but today’s data dashboards and crude value-added measurements only mean we have finally caught up to the pre-“moneyball” era. We finally have simple, incomplete performance measures like home runs and batting average. These tell us something useful, but they can provide a distorted picture or lead us astray if not used with care. Today’s metrics conflate the effect of support staff and teachers of record, capture only a narrow slice of instructional quality, are exceedingly imprecise, and are relevant (even incompletely) for no more than perhaps 30 percent of teachers. This is a far cry from counting and measuring everything that matters, and then allowing calculations of cost-effectiveness to guide hiring and staffing decisions.
Paul DePodesta, the inspiration for Jonah Hill’s ubergeek statistician in the movie, has explained that the “moneyball” idea was not to scrap baseball’s traditional metrics or scouting systems. (Again, this kind of gets lost in the film version; and even in Lewis’s book.) Rather, DePodesta has pointed out that baseball execs are “constantly trying to predict the future performance of human beings. We’re trying to get our arms around that uncertainty. Scouts really help you deal with that uncertainty. On the other hand, we looked at it and said, ‘How can we further decrease that uncertainty?’ And being able to use data was one of the ways we could do that.”
It’s not that “moneyball” is a bad analogy. It’s a terrific analogy. But you’ve got to use it right. And I fear that the value-added enthusiasts who imagine they’re right now gearing up to play moneyball in K-12 are actually going to find, to their chagrin, that they’re the potbellied scouts hoping to sign an overpriced free agent because the guy drove in 100 runs for the Yankees last year.
– Rick Hess
This post appeared earlier on Rick Hess Straight Up.
