Last week, I offered thoughts on the Fordham Institute’s research paper on e-schools in Ohio titled, “Enrollment and Achievement in Ohio’s Virtual Charter Schools.”
It is a strong study. As I wrote though, it has four significant limitations in the research—not of the analysts’ fault but instead because of limitations intrinsic to the data that are currently available to measure the outcomes of e-school students—that should give us pause in the conclusions we are able to draw from it.
It’s worth discussing in more depth one of the limitations, as well as exploring the policy implications from Fordham’s study.
Understanding students’ jobs to be done
The Fordham study controls for both the demographic and prior achievement variables that are measured at the state level of e-school students. This is useful for giving us a window into the quality of a school on average, but it misses understanding at an individual level the true circumstances under which and motivation for why a student enrolls in a full-time virtual school. In our research on innovation, we refer to this underlying causal reason as the student’s “job to be done.” It therefore does not allow researchers—and this report in particular—to actually compare like situations and results.
Demographic data misleads researchers in lots of circumstances. For example, in business, most companies simply look at the average demographic likely to use a given service and what features on average those people say they want. They then make improvements accordingly. But customers often don’t follow what the average in their demographic is likely to do, which is a big reason why, even in the era of big data, the vast majority of new product and service launches fail. Instead, jobs arise in people’s lives—a particular circumstance in which they need to make progress—and they hire different services to help them do those jobs. Understanding the job allows us to understand causality. For example, the traditional demographic data can’t explain why a man might take a date to a movie on one night but order in pizza to watch a DVD from Netflix the next night. In each case, a different job has arisen that has caused the behavior.
In the case of virtual schools, a significant percentage of students enroll because of a medical issue or bullying, for example. For a student in this circumstance, she may be “hiring” the virtual school to “help me escape from my current situation”—with little regard to what that decision means for her future. Academic considerations are secondary at best. Understanding this dynamic is critical to making a valid comparison and to understanding what success is.
Policy implications
Despite its limitations, the Fordham Institute’s policy recommendations are solid and recognize some of the nuances in the full-time virtual school landscape. Fordham helpfully does not call for a ban on full-time virtual schooling. It recognizes that, for many individual students, virtual schools have been critical to their academic and life success. Fordham instead presents constructive recommendations for improving virtual schools—and online learning and schooling more generally.
One of the key recommendations it makes in the foreword and conclusion is that policymakers should “explore ways to improve the fit between students and e-schools.” This could help tremendously in addressing the job for which students are hiring virtual schools—and help them see that there may be other options to get that job done better than the particular virtual school they have chosen. For example, the report says, “Ohio recently enacted a provision requiring e-schools to offer an orientation course—a perfect occasion to set high expectations for students as they enter and let them know what would help them thrive in an online learning environment (e.g., a quiet place to do schoolwork, a dedicated amount of time to devote to academics).” For students enrolling who are not considering the future and are just looking for an escape from the present, helping quickly shift their focus to the future could help them improve their decision-making so they don’t get stuck in a situation they need to escape yet again. I would love to see schools also adopt a page from the new EQUIP higher education program and bring in an outside quality assurance entity that would audit the claims on outcomes—across a wide range of objectives—a school makes to drive better student and family decision-making.
Additionally, Fordham recommends that state lawmakers explore rules that would allow e-schools to be selective in which students they serve, such that they could optimize their operations for a particular set of jobs students have and only serve students who have those jobs such that success is more likely. Offering students and families the services of an advisor as they consider transitioning to a full-time virtual school to help them better understand their options and goals is another recommendation I would add.
The call to support statewide online course access also makes sense. This could allow students who need access to certain courses unavailable at their school or who are interested in online learning for part of their day to avoid making an all-or-nothing decision between transferring to a “full-time e-school or stay[ing] in their traditional school and potentially be denied the chance to take any tuition-free credit-bearing virtual courses aligned to Ohio state standards.” This would create options better aligned to students’ actual jobs to be done.
Finally, in the foreword, Fordham recommends that policymakers “should adopt performance-based funding for e-schools. When students complete courses successfully and demonstrate that they have mastered the expected competencies, e-schools would get paid. Implementing performance-based funding policies would create incentives for e-schools to focus on what matters most—academic progress—while tempering their appetite for enrollment growth and the dollars tied to it. It would also encourage them to recruit students likely to succeed in an online environment—a form of ‘cream-skimming’ that is not only defensible but preferable in this case.”
At a high level, others who study online learning and I believe this idea makes sense. But how it is done matters. An all-or-nothing funding formula would not work, as it ignores the fact that full-time schools must make significant up-front investments and do have fixed costs to serve students. A sounder way forward would give virtual schools some up-front funds for these fixed administrative and technology costs (such as furnishing students with a laptop and home Internet connection), and then fund schools with a weighted per-pupil rate as students mastered competencies and made academic progress, which embeds performance funding in the formula and is similar to how New Hampshire’s Virtual Learning Academy Charter School is funded. In other words, a student’s academic progress would replace seat time—such as average daily attendance—as the measurement that determines funding levels. As a simple example, when a student mastered 10 percent of a course, the school would receive 10 percent of funding for that course; when the student mastered the next 10 percent, the school would receive the next 10 percent of funds. External state assessments could be used to make sure schools’ more micro-assessments of learning were rigorous and valid.
Those from the online learning community who say they like this performance-funding mechanism but will agree to it only when all brick-and-mortar schools are subject to it as well are thinking too narrowly. First, seat-time funding is totally anathema to virtual education—far more so than for brick-and-mortar schools that have been funded based on seat time for generations. Second, we need to try out a competency-based funding model somewhere in the system first to learn how to make it work correctly. Starting small is important. Third, the traditional schooling system will never let performance-based funding start there; arguing for waiting is akin to saying “no.” And fourth, the new upstart system is always going to look “worse” than the traditional system because it’s not how “things have always been done.” Why not step up and show that the upstart system performs better on a more rigorous standard as a way to get early wins and build momentum, flip the public relations narrative, and then have the conversation about funding for the larger system?
Ultimately, as I’ve written, virtual schools must step up their game. Although it’s unlikely they will ever grow to educate more than five to 10 percent of K–12 students, given the important service they provide for many individual students and families, we need to better understand the value they are delivering. Harnessing their benefits while reining in their downsides is critical.
—Michael B. Horn
Michael Horn is a co-founder of and a distinguished fellow at the Clayton Christensen Institute for Disruptive Innovation.
This post first appeared on Forbes.com
