The Education Exchange: Higher Ed Has Withstood Past Innovative Shocks. AI Is Hitting Different.

PAUL PETERSON, HOST:

This is the Education Exchange. I am Paul Peterson, Director of the Program on Education Policy and Governance at Harvard University. Thank you for joining us. Many of the best educated people in the United States are afraid of losing their jobs, and they’re not going to lose them to somebody else, but to the monster known as artificial intelligence. In the past, we didn’t worry about this. So, we said, okay, technology advances, but new jobs are made, and they’re made at a faster rate than the ones that are destroyed. Well, some people worried about it, but not the educated people, because the educated people could see that the new kinds of jobs coming along were those that required a college degree or studying advanced beyond college. So going to college, more education was always the solution to advances in technology. But now artificial intelligence could be different. It could be destroying white-collar jobs more rapidly than blue-collar jobs. Artificial intelligence is targeting jobs in administrative services, bookkeeping, computer science itself. So, any job that can be simplified into step-by-step procedures is potentially at risk. That has created a lot of concern about artificial intelligence. Now, Jacob Light, he’s an economist at the Hoover Institution at Stanford University; he’s taking a good look at what’s happening inside the colleges and universities themselves. And a soon-to-be-released paper entitled “How Exposed is Higher Education to Artificial Intelligence?” I almost went to check my website to see if I, or my email, to see if Harvard had fired me, but they hadn’t yet. So, “How Exposed is Higher Education to Artificial Intelligence” is the topic that he is going to address with us today, and I’m delighted to have him on the Education Exchange. Thank you for joining us, Jacob.

JACOB LIGHT: Thank you so much for having me, Paul.

PETERSON: Well, Jacob, your paper is a sophisticated analysis of college courses that students are taking. But before we get into your data and your methodology, as important as it is, I think our listeners want to know, first of all, what are your major findings? What do you know about our educational system now that you didn’t before you took on this research?

LIGHT: So, the project asks two questions. First, if we are going to be concerned about the exposure of higher education to AI, where should we be concerned? Which fields of study have the most overlap between the things that students do in their classes and what AI is capable of doing? And the second question is, how are universities and how are students responding to the specific shock represented by the release of ChatGPT? So, on the first question, I study how different fields of study differ in their exposure to artificial intelligence. What I find is that, in contrast with previous technology shocks, which, as you said, kind of advantage the skills of more-educated workers compared to less-educated workers, we’re looking at a shock where the technology can do a lot of what we’re training students to do in their college courses. The overlap is particularly strong in fields that involve data analysis and writing. So, we should think about this as kind of computer science, but also the quantitative social science fields. And then on the second question, I study how students’ enrollment patterns and instructors’ course offerings have changed since the release of ChatGPT. And I find that despite this very large shock to how we conduct our courses, there’s very little change both in the types of courses that students are taking, and in the way that instructors are administering their courses, especially if we compare the release of ChatGPT to a similar shock, the pandemic, that instructors are changing the way that they structure the courses, the kind of weights that they put on exams versus problem sets, much less than they did during the pandemic.

PETERSON: So, Jacob, are you really saying here that we should all be taking English and history instead of statistics and economics? You’re an economist yourself, you know.

LIGHT: Right. And I think one of the concerning findings is that the things that we teach students to do in their economics classes, analyzing data, using models, evaluating policy, these are things that large language models seem to be quite capable of doing. But I think it’s important to stress that what I think of as this exposure measure is not a— it’s not a siren that we shouldn’t be studying economics, that we shouldn’t be studying computer science, but particularly when we think about how universities will have to adapt in their ability to kind of detect whether students are developing the skills that they were developing before the release of these large language models. I think this exposure measure is really telling us that there’s a risk that if students are substituting AI for their own learning, that we are at risk of producing economists with less human capital, with fewer skills than the economists that we produced before the release of ChatGPT.

PETERSON: I want to ask you about that. But before I do so, I think it’s good for our audience to understand exactly what kind of data you’re looking at. So how do you know what’s going on? You know, colleges and universities are secret organizations. It’s trying to figure out what a professor is doing behind that closed door or a teacher in the classroom, which is what I said. What do they do behind that closed door? How did you penetrate to find out what’s going on?

LIGHT: The data that I work with is a data set that I’ve collected over the course of many years by scraping university course catalogs and course schedules. What this allows me to do is develop a sense of what are the skills that a student develops in the typical economics class or the typical math class. I do this by pooling together course descriptions from about a thousand U.S. universities, which collectively enroll over half of U.S. undergrads. The data set includes 55 million course sections that have been offered at US colleges and universities since the beginning of the century. So, it’s really rich, comprehensive information about the exact things that students are doing in their college classes, kind of at a scale that existing data sets don’t really allow us to unlock. I use these data to extract from course descriptions the types of things that students are doing in their courses. So, in computer programming or in computer science courses, we see students specializing in skills related to computer programming, applying algorithms, analyzing data.

PETERSON: Yeah. Just let me interrupt you on that point, Jacob, because I do these syllabuses that you’re looking at. How do you get them? Do you sneak into the classroom? You know, syllabuses are not generally…you’re not required to put this out on the internet. So how do you find these things?

LIGHT: So, I’m working with two different types of data. The first is course catalog and course schedule data, which I have for this larger sample of universities, these thousand universities that I mentioned. And there what I’m doing is I’m scraping public websites that the universities put up and that they archive over many years that allows me to see section-level offerings at the university. So, Paul Peterson’s Intro to Government course offered in 2010. I can see how many students enrolled in the course, and I can see a brief text description of what are the skills that students will learn in that course. Or…

PETERSON: What do you do with that?

LIGHT: So, I could construct a Paul—

PETERSON: You assume that the syllabuses accurately reflect what’s going to go on in the classroom.

LIGHT: Yeah, so I could construct a Paul Peterson’s Intro to Government specific AI exposure measure, but these course descriptions are widely available for a large number of universities. Now I use syllabus data for the specific part of the project where I’m studying university responses to the release of ChatGPT and as you point out most instructors are not required to publish their syllabus on a public facing website so, for a smaller number of universities, about 25, whether it be a university policy or in response to some state regulation, I’ve downloaded the syllabi from about 25 universities. And so, it’s a smaller sample, but it allows me to track the way that the courses are structured over time. So not just what are the high-level skills and topics that a student is going to be exposed to, but how is the class administered? And does the way that the class is administered change in the presence of a kind of threat to the instructor’s ability to monitor student learning?

PETERSON: Well, that’s a very impressive data set. I must say it’s amazing. Nobody else has ever done this before, I take it.

LIGHT: That’s correct. Yeah. There are some researchers who are working with similar syllabus data, but the course offerings data and the enrollment, the kind of comprehensiveness of that data set, I think is really unique in the field.

PETERSON: Well, congratulations on pulling that all together. But now what you’re saying is that the students are studying the wrong thing in these classes. Now, how do you say that figure that out, that there’s an overlap between what the students are studying in statistics and what AI could do for them, and they don’t really need to learn that?

LIGHT: So, the way that I calculate these exposure measures is I extract from the course descriptions the verb-object pairs, the kind of tasks that students do in their courses. This is adapting methods that labor economists use to measure the exposure of occupations to technology but taking it one level upstream at the university level. So, examples of these verb-object pairs that students in economics classes learn to analyze data, they use models, and they evaluate policy. And the intuition for this exposure measure is if we see that AI is very good at analyzing data, using models, and evaluating policy that we might think of economics as being a field that’s highly exposed to artificial intelligence. And I should say that this measure is agnostic about the implications of exposure. It’s not strictly the case that exposure means that students should not be learning how to do this. We teach students many skills from a very young age that are completely automated. We teach students basic arithmetic. We teach students spelling. And we do that because that knowledge unlocks higher order thinking that technology can’t do. And also, because we think that those skills are generally valuable, even when students have access to calculators and spellchecks. So, I see that this first exposure measure part of the paper is telling us where should we be concerned that students have the ability to substitute AI tools for their own work? And that has implications for how instructors should be administering their courses. But I want to be cautious about saying that this exposure is in some way bad, both from the perspective of labor market implications, as well as kind of the value of the skills that students are developing.

PETERSON: Well, that’s good that you’ve explained that, because when I was reading the paper, at times I said to myself, well, just because AI can do it doesn’t mean that I shouldn’t know how to do it myself. Because I’ve got to understand what AI is doing and not if I can’t do it myself, I’ll never understand what it’s doing for me.

LIGHT: Yeah, absolutely. And if we’re moving to a place where we want students to have expertise, or we want workers to have expertise beyond the capabilities of what AI can do, then you’re right. It is essential that students know how to do the things that the AI is doing for them so that they can assess whether the quality of the output matches what they wanted.

PETERSON: But you’re saying the other side of the coin is that it may not be so easy to get a job if this is all you’re learning. If you’re not going beyond that in the course, you may be just learning material that somebody is going to be able to do for you or something out there, some avatar can do, and you won’t have a job.

LIGHT: Yeah, so I think that the predictions from the labor economists suggest that if it really is truly the case that all that a student, all that a worker is capable of doing is something that technology can do at a much lower price than what that worker would demand, then we would expect fewer of those workers to be employed. And I think that would potentially be concerning. What I really stress in the paper is if we put aside the risks in the labor market, what this exposure measure is telling us is that students can use these tools both to help them in their courses, that you have access to a bespoke tutor that can help you do your very tough computer science problem sets, but you also run the risk of students being able to completely substitute these tools for their own work, that I can generate using AI answers to my economics problem set in a way that five years ago, I would not have been able to. And so, there are certain screening methods that instructors have used for a long time, such as out-of-class problem sets, that no longer carry the same information about how much students are learning in the class. And I think that’s the big concern that this AI exposure measure addresses.

PETERSON: So, let’s say that people are being asked to do things that AI could do, and that’s okay. But then how do you make sure, how does the instructor make sure then that AI isn’t just answering the questions, and the student is just sitting there, you know, reading one of their comic magazines?

LIGHT: So yes, so the first part of this project really kind of points us to there are some fields of study, the quantitative social sciences, computer science, and statistics, where what AI broadly, as well as kind of large language models specifically, are capable of doing intersects with the exact things that students are learning to do in their courses. So, the question is, what can instructors do to make sure that even that these tools are being used as a complement to student learning rather than a substitute. And so for the second part of the project, I’m using this panel of syllabi from about 25 universities, and I extract two pieces of information from the syllabus. The first is whether the instructor has an AI policy at all. This tells us, are instructors thinking about the implications of AI for their courses? And the second thing that I extract is the weights that instructors put on different types of assessments. So, in Paul’s Intro to Government class, half of the grade is based on exams; 25% of the grade is based on problem sets; 25% is based on participation. And what I study is the changes in the weights that are put on the types of assessments that can’t be done with AI. So, in-class exams, presentations, participation, I look at whether the weights that instructors are putting on those types of assessments, whether those weights are increasing and substituting away from the types of assessments where students can very easily substitute AI for their own work. So out of class essays and problem sets.

PETERSON: Well, so what I’ve been doing in my classes, Jacob, is I’ve been asking students to write a response to a question when they come to class each day. The question comes from the material that they showed…that they exposed themselves to before coming to class. And then in class, they sit down and they write. And it’s a pencil and paper kind of thing, a pen and paper kind of thing. And then they hand that in. And then that’s going to be…I have increased the percentage of the grade that’s based on that significantly. And then I’m also asking them to give a presentation in class of a paper that they prepared. So, they do prepare a paper, which AI can help them with that, but I figure it’s going to be harder for them to substitute AI when they are actually presenting in the class. That’s some of the stuff that I’m trying to do. Is that the kind of thing you’re looking for? Is that a good thing or what would you do?

LIGHT: Yeah. So, in the class that I teach, I do something similar. I have students read a paper for every class. I ask them to prepare a couple of discussion slides and then I pick a student at random to present those discussion slides to the rest of the class. So, the hope is that the risk of being asked to present the paper is a mechanism that encourages the students to read the paper rather than outsourcing the reading. But I…if we take the results from my paper seriously, I think you and I are somewhat minorities in this case. What I find is that the weight that instructors are putting on assessments where students can substitute AI for their own work, these problem sets, out of class essays hasn’t really changed since the release of ChatGPT in late 2022. And it’s kind of instructive looking in the data to compare two different shocks that push assessment weights in opposite directions. So, during the pandemic, we had a shock that made it more difficult for instructors to administer in-class exams because a lot of instruction was occurring virtually. And what we see during the pandemic is a sharp decrease in the weight that instructors put on exams and a reallocation mostly towards problem sets. That change in assessment weights persists even in the years after the pandemic, after in-person instruction resumes. And if we can compare the size of that change to the size of the change since the release of ChatGPT, there’s really no comparison. We’re just kind of offering courses today where the assessment weights are very similar to the way that we administer the course prior to the release of ChatGPT.

PETERSON: So, this is an irony, isn’t it? I mean, this is a truly fascinating, ironic situation that during the pandemic, when we closed our colleges and our high schools, we then at the same time set up an examination system which addressed that closure but then prepared the groundwork for a world in which AI could pose a new threat to learning and the new threat the colleges are not responding to.

LIGHT: I think that that’s right. And there are certainly many instructors within these universities who are making adaptations. I don’t want to say that that’s not happening. You hear of stories like your class that are changing. But I think at scale, the nature of this large language model shock is different from the pandemic in the sense that during the pandemic, the instructions for how to change courses were more obvious in the sense that the move to online learning was a top-down policy from the university. And instructors could respond, or instructors were told how to respond versus the nature of this technology is that things are constantly changing. And it’s probably quite hard for instructors to identify exactly where are the vulnerabilities in their courses and how do we patch them through changes in the way that courses are being administered.

PETERSON: Well, maybe you’re just too hasty. Let me give you an example. When the AI first came out, first had this impact, Harvard Central Administration was sending out all kinds of emails that said, oh, don’t be afraid of this. You need to embrace it. We don’t know the right thing to do but whatever you do, make sure you do the right thing. And they gave us no guidance at all. But in the last six months, I see a change of tone. They are now saying, you should not have final exams that are take-home exams. You should have your final exams in class. It used to be that if you were going to have a final exam in class, Harvard would make it difficult for you. You had to do all kinds of things in a certain way to do that. If you had a take-home exam, they made it very easy for you. Now they’re shifting the rules and regulations in the opposite direction. So, I think there is some concern about this by central administration. They’re beginning to recognize the problem that you’ve identified and are beginning to adapt. Now, maybe that’s just Harvard, but I don’t know if you’ve seen anybody talk about that possibility elsewhere.

LIGHT: There are certainly…I’ve certainly seen discussions of this type of thing. I know an analysis in the Princeton Student Newspaper documented some similar trends in modest shifts away from out-of-class exams towards in-class exams. I think the results of the paper suggest that so far as of the end of fall 2025, at scale, we don’t see large adjustments in the sample of universities that I’m looking at, which span the selectivity spectrum from non-selective public universities all the way up to kind of very selective state flagship schools. So, it’s…it could be the case that instructors are adjusting but with a lag and that eventually we will see shifts of the kind that you and I are discussing. But in the meantime, we have a cohort of students who have gone through college in a way where we can’t assess, or we don’t have good measures of whether the skills that students have developed through their courses are the same as the skills that those degrees represented four or five years ago.

PETERSON: Well, just because economics and statistics are the fields that are most exposed to AI, does that mean that those fields won’t continue to be the fields that will yield the largest return in the future? I mean, it could be that they are exposed to AI, but they’re still going to be relatively attractive areas to go into. Do you have any evidence that bears on that question?

LIGHT: If we look at the past as precedent, previous waves of technological change have created more jobs than they have destroyed. And although the period of transition within the labor market can be painful for people who are currently employed or who are entering the labor market, on net, things generally are better. Now, whether it’s the case that job opportunities for people who have skills of the kind that economics graduates develop or that statistics graduate develop, whether those jobs will continue to be as lucrative or as abundant really depends on how workers incorporate AI into their workflow. It could be the case that the availability of these tools makes it possible for firms that otherwise would not hire a computer scientist, an economist, or a statistician because just having one on staff doesn’t make sense. And now hire a very productive analyst who has an army of AI agents to do work that those companies were not doing previously. I’m hesitant to make a long-term prediction at the long-term viability of an economics degree or a statistics degree is—I continue to think that the critical-thinking skills that a student develops in an economics major are incredibly valuable. And as long as students are developing those skills, they will be very productive members of the labor market.

PETERSON: So, I have a close relative working in a high-tech industry. And he tells me that he’s in a managerial position. And he tells me that they are under great pressure to reduce the number of staff in their company because the people who are investing in their company look at the revenue staff ratio. So, if you are generating a lot of revenue, but you have a lot of staff, that’s not as good as generating a lot of revenue with less staff. So that revenue staff ratio has become a key measure in the investment community. How does that interact with what you’re finding? The possibility would be that you really got to know AI, and you’ve got to be able to use AI, and you’ve got to be able to perform more work with AI if you’re going to be employable.

LIGHT: If we think about AI as any other technology change, a worker who is made more productive through access to these tools, offers more value to a company if the additional production that they can engage in exceeds the costs of the employee and the tools that they’re using. And I think right now companies are starting to learn about the costs of these AI tools, that you hear stories about firms reducing headcount and then exceeding their budgets for these AI tools. We’re in a learning period where everybody is deploying these tools without a strong understanding of exactly how much AI is needed to substitute for one worker. So, I think at the end of the day, firms will make a decision about deploying AI with or as a substitute for workers based on whether the extra output that you can get through the combination of these inputs is greater than the costs of either keeping an additional worker on staff or the compute costs of using AI with that worker. So, I think it’s a little hard to say. We’re in early stages, and this is a technology shock that’s very different from computers or robots in the sense that the technology was deployed very widely. This universal access means that many companies are learning in real time what the right level of AI versus human labor should be.

PETERSON: Now, one of the findings is that students are modifying their behavior in the courses that they’re taking is because you’re seeing some changes in enrollment, but you’re not seeing a lot. So, let’s explore that topic a little more. What do you find? Are they going into taking English classes or are they going into working as carpenters? What do you find in that domain?

LIGHT: So, for that analysis, I’m once again using the course catalog and course schedule data. So, it’s enrollment at the course level for about a thousand U.S. colleges and universities. And what I’m seeing is that there’s a slight decrease in the last couple of years in enrollment in courses that this exposure measure predicts are highly exposed to artificial intelligence. What this means in practice differs based on the type of school. So, we’ve seen broad increases in skilled trade enrollment at two-year community colleges, and that seems to be taking enrollment away from essentially every other field uniformly. At the highly selective R1 research universities, we see for the first time a decline in enrollment in computer science after kind of continuous growth in CS enrollment over the last 15 to 20 years. So, I think that’s kind of the leading indicator of at some universities, we see students moving away from computer science due to a perception of risk that CS skills are less valuable today than they were a couple of years ago. And to be fair, it’s not obvious that that would happen. On one hand, there’s more risk, but also CS classes are slightly easier because you have this tool that can help you do your homework and that can tutor you through the difficult parts of the classes. Where are those students going? Mostly to business and economics courses.

PETERSON: Which may not be any better option than the computer programming courses, right?

LIGHT: I think that we…if the response is less thinking about exposure of the skills that students are developing and more related to perceptions of risk in the labor market, then the response makes sense. But yes, students in the quantitative social science fields, including economics, are developing data analysis and writing skills that these large language models can do very well.

PETERSON: So, I’m thinking that actually they should be taking English courses or they should be taking the old-fashioned liberal arts courses. Do you see any hint of that?

LIGHT: I don’t see any hint of that initially. I am very sympathetic to students who are in college right now during a period of immense uncertainty about what the returns on different degrees and different fields of study are. But I think to the discussion that we had earlier, at least in the short term, there will continue to be strong value in developing the skills aligned with what these tools are capable of doing so that you can supervise the large language model’s output. That we will need, even if we need fewer entry-level analysts, entry-level paralegals; we will need perhaps even more managers to supervise the output of these tools. And we can’t create managers without the expertise that somebody develops in an entry-level position. So, I continue to see a lot of value in economics and business degrees, even if the graduates will enter jobs where AI tools are going to be increasingly a part of the workflow.

PETERSON: Well, Jacob, our listeners are fascinated, I am sure, by the conversation that we’ve had today. It’s really an important conversation because just exactly how to adapt to a world we barely understand. You’re shedding some light on this. So, thank you very much, Jacob, for your presentation and discussion today.

LIGHT: Thank you so much. I really enjoyed the conversation.

PETERSON: I’ve been speaking with Jacob Light. He’s a Hoover Fellow at the Hoover Institution at Stanford University. He’s the author of a forthcoming paper entitled The Attraction… I should say a just released paper because by the time this appears, you will be able to get this at Jacob Light’s website. Give the title, Jacob.

LIGHT: The title is, How Exposed is Higher Education to Artificial Intelligence?

PETERSON: Thank you. Thank you, Jacob, for joining me on the Education Exchange. I am Paul Peterson. Please join me for our new Education Exchange podcast released on the Education Next website every Monday at noon Eastern time.