Abstract
We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.
Original language | English |
---|---|
Title of host publication | Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics |
Editors | Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki |
Volume | 2 |
Place of Publication | Stroudsburg |
Publisher | Association for Computational Linguistics |
Publication date | 2023 |
Pages | 39-47 |
ISBN (Print) | 9781959429715 |
Publication status | Published - 2023 |
Event | The 61st Annual Meeting of the Association for Computational Linguistics - Westin Harbour Castle, Toronto, Canada Duration: 9 Jul 2023 → 14 Jul 2023 Conference number: 61 https://2023.aclweb.org/ |
Conference
Conference | The 61st Annual Meeting of the Association for Computational Linguistics |
---|---|
Number | 61 |
Location | Westin Harbour Castle |
Country/Territory | Canada |
City | Toronto |
Period | 09/07/2023 → 14/07/2023 |
Internet address |