Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[2408.02442] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models #910

Open
1 task
ShellLM opened this issue Aug 21, 2024 · 1 comment
Open
1 task

[2408.02442] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models #910

ShellLM opened this issue Aug 21, 2024 · 1 comment
Labels
human-verified <INST>NEVER PICK THIS LABEL</INST> llm Large Language Models llm-applications Topics related to practical applications of Large Language Models in various fields llm-evaluation Evaluating Large Language Models performance and behavior through human-written evaluation sets llm-experiments experiments with large language models MachineLearning ML Models, Training and Inference Papers Research papers prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re

Comments

@ShellLM
Copy link
Collaborator

ShellLM commented Aug 21, 2024

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Snippet:
Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs' abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs' performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs' reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.

Comments: 18 pages

Subjects: Computation and Language (cs.CL)

Cite as: arXiv:2408.02442 [cs.CL] (or arXiv:2408.02442v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2408.02442

Suggested labels

None
@ShellLM ShellLM added llm Large Language Models llm-applications Topics related to practical applications of Large Language Models in various fields llm-evaluation Evaluating Large Language Models performance and behavior through human-written evaluation sets llm-experiments experiments with large language models Papers Research papers labels Aug 21, 2024
@ShellLM
Copy link
Collaborator Author

ShellLM commented Aug 21, 2024

Related content

#762 similarity score: 0.86
#706 similarity score: 0.86
#333 similarity score: 0.85
#519 similarity score: 0.85
#876 similarity score: 0.85
#729 similarity score: 0.85

@irthomasthomas irthomasthomas added prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re MachineLearning ML Models, Training and Inference llm-benchmarks testing and benchmarking large language models human-verified <INST>NEVER PICK THIS LABEL</INST> and removed llm-benchmarks testing and benchmarking large language models labels Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
human-verified <INST>NEVER PICK THIS LABEL</INST> llm Large Language Models llm-applications Topics related to practical applications of Large Language Models in various fields llm-evaluation Evaluating Large Language Models performance and behavior through human-written evaluation sets llm-experiments experiments with large language models MachineLearning ML Models, Training and Inference Papers Research papers prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@irthomasthomas @ShellLM