Ellipsis-dependent Reasoning: A New Challenge for Large Language Models

Daniel Hardt

Ellipsis-dependent Reasoning: A New Challenge for Large Language Models

Daniel Hardt

Department of Management, Society and Communication

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

38 Downloads (Pure)

Abstract

We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

Original language	English
Title of host publication	Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
Editors	Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Volume	2
Place of Publication	Stroudsburg
Publisher	Association for Computational Linguistics
Publication date	2023
Pages	39-47
ISBN (Print)	9781959429715
Publication status	Published - 2023
Event	The 61st Annual Meeting of the Association for Computational Linguistics - Westin Harbour Castle, Toronto, Canada Duration: 9 Jul 2023 → 14 Jul 2023 Conference number: 61 https://2023.aclweb.org/

Conference

Conference	The 61st Annual Meeting of the Association for Computational Linguistics
Number	61
Location	Westin Harbour Castle
Country/Territory	Canada
City	Toronto
Period	09/07/2023 → 14/07/2023
Internet address	https://2023.aclweb.org/

Access to Document

daniel_hardt_ellipsis-dependent_reasoning_publishersversionFinal published version, 286 KBLicence: CC BY

https://aclanthology.org/volumes/2023.acl-short/Licence: CC BY

Handle.net

Persistent link

Cite this

@inproceedings{66948743ec2e4c998b348d2db8af7a48,

title = "Ellipsis-dependent Reasoning: A New Challenge for Large Language Models",

abstract = "We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.",

author = "Daniel Hardt",

year = "2023",

language = "English",

isbn = "9781959429715",

volume = "2",

pages = "39--47",

editor = "Anna Rogers and Boyd-Graber, { Jordan} and Naoaki Okazaki",

booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "The 61st Annual Meeting of the Association for Computational Linguistics ; Conference date: 09-07-2023 Through 14-07-2023",

url = "https://2023.aclweb.org/",

}

Hardt, D 2023, Ellipsis-dependent Reasoning: A New Challenge for Large Language Models. in A Rogers, J Boyd-Graber & N Okazaki (eds), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. vol. 2, Association for Computational Linguistics, Stroudsburg, pp. 39-47, The 61st Annual Meeting of the Association for Computational Linguistics , Toronto, Ontario, Canada, 09/07/2023. <https://aclanthology.org/volumes/2023.acl-short/>

Ellipsis-dependent Reasoning: A New Challenge for Large Language Models. / Hardt, Daniel.
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. ed. / Anna Rogers; Jordan Boyd-Graber; Naoaki Okazaki. Vol. 2 Stroudsburg: Association for Computational Linguistics, 2023. p. 39-47.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Ellipsis-dependent Reasoning

T2 - The 61st Annual Meeting of the Association for Computational Linguistics

AU - Hardt, Daniel

N1 - Conference code: 61

PY - 2023

Y1 - 2023

N2 - We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

AB - We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

M3 - Article in proceedings

SN - 9781959429715

VL - 2

SP - 39

EP - 47

BT - Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics

A2 - Rogers, Anna

A2 - Boyd-Graber, Jordan

A2 - Okazaki, Naoaki

PB - Association for Computational Linguistics

CY - Stroudsburg

Y2 - 9 July 2023 through 14 July 2023

ER -