When behavioral nudges using text messages became the flavor of the month a few years ago I expressed some serious reservations. In general, I was concerned that nudges substitute the preferences of distant experts for those of people who may understand their own situation better, thereby pushing people to do things against their better judgement. These interventions may appear successful in the short run, especially when we examine near-term outcomes that are over-aligned with the intervention, but they may harm people over the long run.
In particular, I was responding to texting nudges being advocated by Ben Castleman and others to reduce “summer melt” by getting students to complete the FAFSA and enroll in college. I wrote:
…even if sending text messages is successful at getting more low-income students to complete the FAFSA and enroll in college in the fall, it is unclear whether this ensures a positive outcome. Students who start college but then fail to finish may be hurt by forsaking employment and other training opportunities and taking on significant debt for a credential they never earn. The students who are accepted to college but then decide not to enroll may have just been deterred by an intimidating form, as Castleman suspects, or they may know things about themselves that made them rationally decide not to pursue a degree they are unlikely to complete. The 160-character solution may unwittingly push students into making decisions that are against their better judgment and end up harming them. Castleman has not reported retention and graduation rates from the texting intervention, so we do not know whether this behavioral nudge is helping or hurting students in the long term.
Well, Ben Castleman and Lindsay Page have finally released results on the longer-term effects of their “summer melt” texting nudge and they are pretty clearly negative. That is, students randomly assigned to receive texts to remind them to complete the FAFSA while they are seniors in high school are significantly less likely to complete an AA or BA degree than those who were not nudged into completing the forms necessary to get financial aid and enroll in college. As can be seen in Table 10, the treatment group was 1.7 percentage points less likely to complete a BA degree in 4 years (p< .01). The treatment group was also .8 percentage points less likely to complete an AA degree after 2 years and 1.0 percentage point less likely to complete an AA after 3 years (p < .1).
Castleman and Page then focus on the subset of subjects who were in the uAspire program and for whom they had outcomes after 6 years. At the end of 6 years the students who were randomly assigned to receive texting nudges were 2 percentage points less likely to have earned an AA degree (p < .1) and they were no more likely to have earned a BA degree. (See Table 12).
Castleman and Page also report results on whether students already enrolled in college are more likely to complete their degrees if they are randomly assigned to receive texts reminding them to renew the FAFSA so they can continue receiving financial assistance. As shown in Table 18, students reminded to renew the FAFSA are no more likely to complete an AA or BA degree.
So, the longer-term results of these texting nudges are generally null or negative despite initially encouraging results that the intervention got more students to complete a form and enroll in college. The problem is likely to be exactly what I suspected. Students were being pushed into doing things that were against their own better judgement.
Researchers and foundation officials may think everyone should enroll in college but they don’t know each student’s circumstances and are very poorly positioned to know what is best for others. Students and the advisors (family, counselors, educators, etc…) who know them and with whom they have authentic relationships understand the context better and are more likely to make good decisions. Technocrats are inclined to manage things from afar, but this approach is very likely to end up hurting students.
The early, positive results for texting nudges received an enormous amount of attention, including an endorsement from Bill Gates and being featured on NPR’s Hidden Brain. Given that we now know that this type of nudge intervention may harm students, I hope there is a comparable amount of attention given to the release of the longer-term negative results. There is no shame inherent in an intervention failing, but there are serious problems if we only tout temporary successes while ignoring long-run damages.
To their credit, Castleman and Page appear to be turning their attention in their newer research to “higher-touch” interventions that may cost more but may also have a better chance of providing guidance better suited to each student’s situation. Higher-touch interventions also seem to acknowledge that success for students typically requires much more than a reminder or some information. To succeed students need character traits that will help them make better decisions for themselves over and over as life presents an endless string of challenges. To shape character requires human interaction and meaningful relationships. That’s something that a “bot” or text message simply cannot do.
— Jay Greene
Jay P. Greene is endowed chair and head of the Department of Education Reform at the University of Arkansas.
This post originally appeared on his blog.
