Harvard Built an AI Academic Adviser. Here's How It Fared.

News Harvard Built an AI Academic Adviser. Here’s How It Fared. Harvard's new AI advising chatbot handles straightforward policy questions but sometimes stumbles on the nuanced ones that often send undergraduates to human advisers. | By Pavan V. Thakkar By Hugo C. Chiasson and Julia A. Karabolli , Crimson Staff Writers In April, Harvard’s Advising Programs Office announced it was building an AI chatbot to answer academic advising questions for the class of 2030 onward. The program — announced a month after the College ended a two-year pilot program pairing freshmen with pre-concentration advisors — made its way into incoming students’ hands over the summer, as they began selecting the courses that would make up their first semester. In announcing the program, Assistant Director for First-Year Academics Brooks B. Lambert-Sluder ’05 said it was “not meant to replace human advising,” but instead to connect students with existing information online and direct them to appropriate resources. The Crimson tested Harvard Student Compass, the ChatGPT Edu-powered chatbot, and spoke with students and departmental directors of undergraduate studies about where it worked — and where it broke down. The bot handles straightforward questions grounded in official policies, but its limits became clearer on the more nuanced questions that often send students to human advisers. Student Compass can draw only from select policy documents and Faculty of Arts and Sciences websites, including the Student Handbook and departmental websites. It cannot access the Q reports, the live my.Harvard course search, course sites, or some syllabi. Several directors of undergraduate studies said they had not learned much about the program or had the opportunity to use it. Economics Director of Undergraduate Studies Jeffrey A. Miron said the department had gotten “not too much” information about the chatbot in advance of its rollout. He said that the department’s emphasis was mostly on existing concentrators, and the new chatbot did not present “an obvious set of action items.” For straightforward questions with answers in the handbook or on a department website, students said the bot delivered. Advertisement Shriya Dharmapurikar ’30 said she used it to find a general education course that would also count toward her prospective concentration — a task she would have otherwise spent time scrolling through requirement pages for or waiting for a reply from an adviser. Get The Crimson in your inbox. Sign up “That really literally saved me like an hour of time,” Dharmapurikar said, adding that she appreciated the bot’s citations, which allowed her to verify its generated answers. The bot links to each of the sources it pulls from when responding to a query and ends with a reminder to consult human advisers for time-sensitive questions. “These references are provided to support your planning as a Harvard College student. For time-sensitive or high-stakes decisions, confirm with official sources, such as your Resident Dean or the Advising Programs Office,” the bot writes. But when the questions got more complicated, the bot began to stumble. Joseph K. Blitzstein, director of undergraduate studies in the Statistics department, tested it against questions that frequently come up in concentration advising meetings and found that it often gave wrong answers. “A chatbot is a poor substitute for in-person advising from a human expert,” Blitzstein wrote in a statement. “Even if the chatbot were infallible, there are many reasons why it is valuable for students to meet with advisors in the concentrations they are considering.” Advertisement Some students saw the same limitation. Joseph S. Saba ’30 said the impersonal, rigid nature of an online chatbot made it hard to encourage students to step outside their comfort zone. He said that Harvard Compass does not have the same insight as a faculty member in a related department and cannot match the fluidity of an in-person conversation. “The AI is kind of rigid,” Staba said. “Our first semester specifically, you should be as flexible as you can. And while you can tell it to try to be flexible, again, it’s going to keep on giving structured answers, which I think — at this point in time — isn’t what students need.” Angela S. Allan, associate director of History and Literature, said the bot performed well on general factual questions, but fell short on the more detailed, idiosyncratic aspects of the program — the kind of nuance she said only becomes clear through a conversation with an advisor. Allan added that the bot’s picture of studying History and Literature skewed toward more traditional, text-based work, such as court cases and archival research, without capturing the concentration’s flexibility, which allows students to work with film, comics, or video games. “I would love for students to come in for advising conversations because we really think about them as conversations where we also want to learn more about students and what they’re interested in,” Allan said. “That’s the kind of thing that you’re not going to get from a chatbot.” Miron struck a middle ground. He found the chatbot’s answer to common questions from students in the department useful, but agreed that there will remain a need for human advising. “I’m on the moderately skeptical side of the view that AI is going to solve all our problems,” Miron said. “AI is a tool; it can be a useful tool in plenty of settings, but I don’t think it’s going to miraculously make things dramatically better.” Advertisement In a statement to The Crimson, Lambert-Sluder reaffirmed that Student Compass was designed to be a “starting point” rather than a replacement for human advising, providing answers for questions “easily addressed by extant resources” while directing students to individuals with more expertise otherwise. “We welcome feedback on this tool, from advisors, PAFs, and students, as part of our regular process of improving resources for students to chart their own path,” he wrote. —Staff writer Hugo C. Chiasson can be reached at [email protected] or on Signal at hcc.35. Follow him on X @HugoChiassonn . —Staff writer Julia A. Karabolli can be reached at [email protected] . Want to keep up with breaking news? Subscribe to our email newsletter Have a tip for The Crimson? Share it confidentially Share See more on: College , Student Life , Academics