Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

evaluate ChatGPT #1218

Closed
percyliang opened this issue Dec 6, 2022 · 3 comments
Closed

evaluate ChatGPT #1218

percyliang opened this issue Dec 6, 2022 · 3 comments
Assignees
@yifanmai
Labels
models
Milestone
v0.2.3

Comments

@percyliang
Copy link
Contributor

percyliang commented Dec 6, 2022 •
edited by teetone
Loading

Run https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs_chat_gpt.conf
@percyliang percyliang self-assigned this Dec 6, 2022
@teetone
Copy link
Member

teetone commented Dec 7, 2022 •
edited
Loading

I think supporting ChatGPT in HELM is doable. I tried different prompts with ChatGPT, and it seems to format answers the way we want them as long as we provide in-context examples. For zero-shot, we would need more descriptive instructions.

@teetone teetone added the models label Dec 7, 2022
@teetone teetone self-assigned this Dec 7, 2022
@teetone teetone mentioned this issue Dec 10, 2022
Merged
2 tasks
@teetone teetone added the v0.2.1 label Dec 11, 2022
@teetone teetone changed the title evaluate ChatGPT on one scenario evaluate ChatGPT Dec 11, 2022
@teetone
Copy link
Member

teetone commented Jan 31, 2023

OpenAI is releasing a ChatGPT API soon:

Screenshot 2023-01-31 at 9 49 20 AM

@yifanmai yifanmai added this to the v0.2.1 milestone Feb 2, 2023
@yifanmai yifanmai removed the v0.2.1 label Feb 2, 2023
@yifanmai yifanmai modified the milestones: v0.2.1, v0.2.2 Feb 3, 2023
@yifanmai yifanmai mentioned this issue Mar 3, 2023
Merged
@teetone teetone assigned yifanmai and unassigned teetone and percyliang Mar 14, 2023
@yifanmai yifanmai mentioned this issue Apr 3, 2023
Merged
@yifanmai
Copy link
Collaborator

yifanmai commented Apr 4, 2023

There was an OpenAI backend bug where using "\n" as a stop sequence caused the OpenAI API to return a HTTP 500; this has been fixed by OpenAI last week in openai/openai-python#290

@yifanmai yifanmai modified the milestones: v0.2.2, v0.2.3 May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yifanmai yifanmai

Labels
models
Projects
None yet
Milestone
v0.2.3
Development

No branches or pull requests
3 participants
@yifanmai @percyliang @teetone