CICS Researchers Recognized at International AI-in-Education Conference

Content

A team led by Manning College of Information and Computer Sciences (CICS) third-year doctoral student Alexander Scarlatos and their doctoral advisor, Assistant Professor Andrew Lan, recently received the Best Student Paper Award at the 2024 International Conference on Artificial Intelligence in Education (AIED), held in Recife, Brazil.  

Co-authored by researchers at the educational software company Eedi, the paper, “Improving the Validity of Automatically Generated Feedback via Reinforcement Learning,” addresses challenges in the generation of automated feedback within intelligent tutoring systems and online learning platforms. 

In subjects like mathematics, where accuracy and a detailed understanding of errors are essential to promoting student understanding and fostering a growth mindset, the generation of automated feedback can be especially difficult. The paper highlights the importance of generating feedback that is both mathematically correct and aligned with pedagogical best practices, improving both the correctness and educational relevance of this feedback. 

The team first introduces a rubric-based framework for evaluating feedback quality that measures various aspects of feedback, such as correctness, whether it avoids revealing the answer, whether it accurately diagnoses the student’s error, and whether it encourages positive learning behaviors. By employing large language models like GPT-4, the research demonstrates how reinforcement learning could be used to train AI systems to generate more accurate and pedagogically aligned feedback.  

Additional Recognition from AIED 2024 

“Affect Behavior Prediction: Using Transformers and Timing Information to Make Early Predictions of Student Exercise Outcome” 
Hao Yu, Danielle A. Allessio, William Rebelsky, Tom Murray, John J. Magee, Ivon Arroyo, Beverly P. Woolf, Sarah Adel Bargal, and Margrit Betke  

Nominated for Best Paper Award, Best Student Paper Award

To further optimize feedback, the researchers utilize an approach called Direct Preference Optimization (DPO), a form of reinforcement learning, to train the AI to generate feedback that better meets the rubric’s criteria. This technique improves the AI’s ability to identify where students made errors and how to guide them toward a better understanding of the material without simply providing the correct answer. 

“The AIED community's main goal is to provide students around the world with personalized, equitable, and high-quality education,” says Scarlatos. “Large language models can help us achieve this by helping students understand and correct their mistakes in real-time. But if we want these technologies to be effective, we need to increase the accuracy and mathematical understanding of these models, as well as keep them aligned with best practices set out by educators. Our work reveals that new training methods make huge gains in small models, but even the best models still have room for growth.” 

The AIED Best Student Paper Award is presented annually to recognize outstanding contributions to the development of AI technologies that enhance educational practices.