Ellipsis-dependent Reasoning: A New Challenge for Large Language Models

Daniel Hardt

Ellipsis-dependent Reasoning: A New Challenge for Large Language Models

Daniel Hardt

Department of Management, Society and Communication

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

143 Downloads (Pure)

Abstract

We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

Originalsprog	Engelsk
Titel	Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
Redaktører	Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Vol/bind	2
Udgivelsessted	Stroudsburg
Forlag	Association for Computational Linguistics
Publikationsdato	2023
Sider	39-47
ISBN (Trykt)	9781959429715
Status	Udgivet - 2023
Begivenhed	The 61st Annual Meeting of the Association for Computational Linguistics - Westin Harbour Castle, Toronto, Canada Varighed: 9 jul. 2023 → 14 jul. 2023 Konferencens nummer: 61 https://2023.aclweb.org/

Konference

Konference	The 61st Annual Meeting of the Association for Computational Linguistics
Nummer	61
Lokation	Westin Harbour Castle
Land/Område	Canada
By	Toronto
Periode	09/07/2023 → 14/07/2023
Internetadresse	https://2023.aclweb.org/

Adgang til dokumentet

daniel_hardt_ellipsis-dependent_reasoning_publishersversionForlagets udgivne version, 286 KBLicens: CC BY

https://aclanthology.org/volumes/2023.acl-short/Licens: CC BY

Handle.net

Persistent link

Citationsformater

@inproceedings{66948743ec2e4c998b348d2db8af7a48,

title = "Ellipsis-dependent Reasoning: A New Challenge for Large Language Models",

abstract = "We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.",

author = "Daniel Hardt",

year = "2023",

language = "English",

isbn = "9781959429715",

volume = "2",

pages = "39--47",

editor = "Anna Rogers and Boyd-Graber, { Jordan} and Naoaki Okazaki",

booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "The 61st Annual Meeting of the Association for Computational Linguistics ; Conference date: 09-07-2023 Through 14-07-2023",

url = "https://2023.aclweb.org/",

}

Hardt, D 2023, Ellipsis-dependent Reasoning: A New Challenge for Large Language Models. i A Rogers, J Boyd-Graber & N Okazaki (red), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. bind 2, Association for Computational Linguistics, Stroudsburg, s. 39-47, The 61st Annual Meeting of the Association for Computational Linguistics , Toronto, Ontario, Canada, 09/07/2023. <https://aclanthology.org/volumes/2023.acl-short/>

Ellipsis-dependent Reasoning: A New Challenge for Large Language Models. / Hardt, Daniel.
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. red. / Anna Rogers; Jordan Boyd-Graber; Naoaki Okazaki. Bind 2 Stroudsburg: Association for Computational Linguistics, 2023. s. 39-47.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Ellipsis-dependent Reasoning

T2 - The 61st Annual Meeting of the Association for Computational Linguistics

AU - Hardt, Daniel

N1 - Conference code: 61

PY - 2023

Y1 - 2023

N2 - We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

AB - We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

M3 - Article in proceedings

SN - 9781959429715

VL - 2

SP - 39

EP - 47

BT - Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics

A2 - Rogers, Anna

A2 - Boyd-Graber, Jordan

A2 - Okazaki, Naoaki

PB - Association for Computational Linguistics

CY - Stroudsburg

Y2 - 9 July 2023 through 14 July 2023

ER -