Ellipsis-dependent Reasoning: A New Challenge for Large Language Models

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

37 Downloads (Pure)

Abstract

We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.
Original languageEnglish
Title of host publicationProceedings of the 61st Annual Meeting of the Association for Computational Linguistics
EditorsAnna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Volume2
Place of PublicationStroudsburg
PublisherAssociation for Computational Linguistics
Publication date2023
Pages39-47
ISBN (Print)9781959429715
Publication statusPublished - 2023
EventThe 61st Annual Meeting of the Association for Computational Linguistics - Westin Harbour Castle, Toronto, Canada
Duration: 9 Jul 202314 Jul 2023
Conference number: 61
https://2023.aclweb.org/

Conference

ConferenceThe 61st Annual Meeting of the Association for Computational Linguistics
Number61
LocationWestin Harbour Castle
Country/TerritoryCanada
CityToronto
Period09/07/202314/07/2023
Internet address

Cite this