[2408.02442] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models #910
Labels
human-verified
<INST>NEVER PICK THIS LABEL</INST>
llm
Large Language Models
llm-applications
Topics related to practical applications of Large Language Models in various fields
llm-evaluation
Evaluating Large Language Models performance and behavior through human-written evaluation sets
llm-experiments
experiments with large language models
MachineLearning
ML Models, Training and Inference
Papers
Research papers
prompt-engineering
Developing and optimizing prompts to efficiently use language models for various applications and re
Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models
Snippet:
Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs' abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs' performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs' reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.
Comments: 18 pages
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2408.02442 [cs.CL] (or arXiv:2408.02442v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2408.02442
Suggested labels
None
The text was updated successfully, but these errors were encountered: