The Trump administration has gutted the Institute of Education Sciences. It’s yanked contracts and zeroed out staff without explanation. This could be a big problem. Federal data like the National Assessment of Educational Progress (NAEP), data on school staff and spending, and measures of college outcomes are vital if we’re to sensibly discuss K–12 and higher education.

But, while I’m very concerned about what’s in store for collecting these statistics, I’m far less sympathetic to the complementary claim that these cuts are costing us valuable insights into “what works.” I just don’t feel like I encounter much education research that provides credible, concrete guidance when it comes to pedagogy, instruction, or classroom management.

Am I unduly skeptical? Well, spurred by fears that the knowledge in ED’s What Works Clearinghouse might be lost, The Learning Agency (TLA) stepped in and released an AI chatbot that helps users navigate the “Doing What Works Library,” which is a federally funded effort designed to make the body of research in the What Works Clearinghouse more accessible and actionable for researchers. This offers a terrific opportunity to gauge the practical value of the accumulated research.

For once, I am not making satirical use of chatbots. This is real. You can try the chatbot yourself here. I appreciate TLA’s moxie in standing it up and was curious to see how helpful the results really were. So, I asked it 10 questions and then judged each answer as Informative, Obvious, or Unhelpful. You can score them yourself as we go along. Agree or disagree, let’s keep track and then compare notes at the end.

(Now, it’s important to keep in mind that the value of the responses will speak both to the research itself and to the caliber of TLA’s chatbot, meaning that any conclusions should be appropriately hedged.)

Here we go:

Informative.

Informative, but also problematic in that it promotes a contested view as conclusive fact.

Obvious, and not really about classroom management per se. But not bad instructional advice.

Informative.

Informative.

Unhelpful, no specifics or clarity provided.

Unhelpful, irrelevant—and really should just reverse the prior response, right?

Unhelpful, though appealing in a certain Zen-like way.

Informative, if pretty vague.

Unhelpful, and oddly ignores the actual question.

What’d you get? I judged five answers to be informative, one to be fairly obvious, and four to be unhelpful. But of the five informative answers, one struck me as problematic and another as pretty nebulous.

Bottom line: Only three out of ten answers struck me as clearly useful—and two of those had to do with early reading. The other responses ranged from vague folk wisdom to problematic answers that offered dubious advice. When it came to homework or discipline, it felt mostly akin to asking a medical chatbot how often I should administer aspirin to a four-year-old and being told to give them a “moderate amount in a reasonable timeframe.”

I don’t want to be overly critical. The results reflect on a few different things: the chatbot, the Doing What Works Library, and the underlying research. The chatbot is fairly new, and the answers we get from AI are a function of the question, which is why it’s useful to rephrase queries or ask follow-ups. (In this case, the real takeaway may be the limits of my interrogation skills). And getting any useful responses this easily is certainly better than nothing. So, there are plenty of reasons to treat this exercise with appropriate caution.

That said, I’m not sure the results here were any more useful or compelling than what I could’ve found on Pinterest. That can’t help but raise hard questions about the utility of education research, the accessibility of results, and how findings are being translated into practice. As much as I’m concerned about  the fate of NAEP or IPEDS, this little exercise reminds me why I’m leery of emphatic claims that cuts to education research are going to have devastating consequences for teaching and learning.

Frederick Hess is an executive editor of Education Next and the author of the blog “Old School with Rick Hess.”