Paper Revision: {2024.acl-long.693}, closes #3875.

data/xml/2024.acl.xml

@@ -8993,8 +8993,10 @@
       <author><first>Arman</first><last>Cohan</last><affiliation>Yale University and Allen Institute for Artificial Intelligence</affiliation></author>
       <pages>12841-12858</pages>
       <abstract>We introduce KnowledgeFMath, a novel benchmark designed to evaluate LLMs’ capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, KnowledgeFMath includes 1,259 problems with a hybrid of textual and tabular content. These problems require college-level knowledge in the finance domain for effective resolution. Second, we provide expert-annotated, detailed solution references in Python program format, ensuring a high-quality benchmark for LLM assessment. We also construct a finance-domain knowledge bank and investigate various knowledge integration strategies. Finally, we evaluate a wide spectrum of 26 LLMs with different prompting strategies like Chain-of-Thought and Program-of-Thought. Our experimental results reveal that the current best-performing system (i.e., GPT-4 with CoT prompting) achieves only 56.6% accuracy, leaving substantial room for improvement. Moreover, while augmenting LLMs with external knowledge can improve their performance (e.g., from 33.5% to 47.1% for GPT-3.5), their accuracy remains significantly lower than the estimated human expert performance of 92%. We believe that KnowledgeFMath can advance future research in the area of domain-specific knowledge retrieval and integration, particularly within the context of solving math reasoning problems.</abstract>
-      <url hash="e2e21860">2024.acl-long.693</url>
+      <url hash="ba7e748c">2024.acl-long.693</url>
       <bibkey>zhao-etal-2024-knowledgefmath</bibkey>
+      <revision id="1" href="2024.acl-long.693v1" hash="e2e21860"/>
+      <revision id="2" href="2024.acl-long.693v2" hash="ba7e748c" date="2024-09-17">Revised the dataset name.</revision>
     </paper>
     <paper id="694">
       <title><fixed-case>API</fixed-case>-<fixed-case>BLEND</fixed-case>: A Comprehensive Corpora for Training and Benchmarking <fixed-case>API</fixed-case> <fixed-case>LLM</fixed-case>s</title>