From bb69dcf083ec99fbd1d2ca3d17d2db0a684bcd74 Mon Sep 17 00:00:00 2001 From: anthology-assist Date: Tue, 17 Sep 2024 13:43:32 -0500 Subject: [PATCH] Paper Revision: {2024.acl-long.852}, closes #3879. --- data/xml/2024.acl.xml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/data/xml/2024.acl.xml b/data/xml/2024.acl.xml index f20229ebe1..8e53e8ad6d 100644 --- a/data/xml/2024.acl.xml +++ b/data/xml/2024.acl.xml @@ -11145,8 +11145,10 @@ ArmanCohanYale University 16103-16120 Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DocMath-Eval, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing financial documents containing both text and tables. We evaluate a wide spectrum of 27 LLMs, including those specialized in math, coding and finance, with Chain-of-Thought and Program-of-Thought prompting methods. We found that even the current best-performing system (i.e., GPT-4) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe DocMath-Eval can be used as a valuable benchmark to evaluate LLMs’ capabilities to solve challenging numerical reasoning problems in expert domains. - 2024.acl-long.852 + 2024.acl-long.852 zhao-etal-2024-docmath + + Included experimental results. Unintended Impacts of <fixed-case>LLM</fixed-case> Alignment on Global Representation