Generative AI Trends – Evaluating LLMs: Where Are We and What's Next?

Abstract: Generative AI, exemplified by Large Language Models (LLMs), has transformed the landscape of natural language processing. In this research talk, Reshmi will delve into the current state of evaluating LLMs and explore the exciting trends that will shape the future of this field. She will begin by assessing the existing evaluation methodologies and metrics used to gauge LLM performance, emphasizing the need for a holistic framework encompassing fluency, coherence, relevance, and safety. She will also address the challenges associated with benchmark datasets and their limited representation, which can lead to biased assessments.

As we anticipate a future where LLMs are applied across diverse domains, we also discuss the need to expand LLM capabilities to handle cross-lingual and multimodal inputs, making them more versatile and applicable on a global scale. Join Reshmi for an insightful exploration of the present and future of LLM evaluation in the rapidly evolving world of generative AI.

Bio: Reshmi Ghosh is an Applied Scientist at Microsoft’s Search, Assistance, and Intelligence team. She has played a pivotal role in a small team of developers working on M365 CoPilots over the past year. Currently, she is focused on deploying Generative AI solutions for consumer and enterprise customers in a responsible and secure manner, ensuring a sustainable user experience. Before the announcement of CoPilots, Reshmi made significant contributions towards integrating intelligence into Microsoft’s productivity applications, including Word and PowerPoint as well as the cloud infrastructure (Azure). She holds a Ph.D. in data-driven decision-making methods for climate change using deep learning/NLP from Carnegie Mellon University. With over 5 years of experience in Machine Learning and AI, both in academia and industry, Reshmi also leads efforts to advocate for more opportunities for women in ML/AI through her involvement in WiDS and WiMLDS.