`Input`, `Output` and `Adapter` refactor #1280

teetone · 2022-12-31T23:31:25Z

Changes

input of Instance is now an object.
output of Reference is now an object.
Broke apart Adapter by creating a class for each adaptation method. Adapter is now an abstract class with GenerationAdapter, LanguageModelingAdapter, MultipleChoiceJointAdapter, MultipleChoiceSeparateAdapter, MultipleChoiceCalibratedAdapter, and BinaryRankingAdapter child classes.
Moved out AdapterSpec, Prompt, RequestState, and ScenarioState from adapter.py to separate files.
Added text_code to ModelRunExpander for reasoning scenarios. We can't assume all models are text-only.
Some minor cleanup here and there.

The changes in this PR should not invalidate our cache. I did a full run with the offline models to ensure the changes in this PR are safe.

…adapter.py

percyliang · 2023-01-01T18:03:39Z

Wow, this is great! Since this is such a big PR, would you mind going through and putting a few comments to help understand what the changes are (e.g., this piece of code is new or simply moved from another file)?

src/helm/benchmark/adaptation/adapters/adapter.py

teetone · 2023-01-02T00:25:34Z

Wow, this is great! Since this is such a big PR, would you mind going through and putting a few comments to help understand what the changes are (e.g., this piece of code is new or simply moved from another file)?

I did a big run with the changes in this PR for the offline eval models, and the cache seems to be safe. I will work on adding comments on this PR next.

rishibommasani

LGTM fwiw - did you test, since I noticed there are some minor changes to the scenario files beyond the main Input/Output change (e.g. renaming variables in wikfact), so curious just to make sure it works.

src/helm/benchmark/scenarios/newsqa_scenario.py

src/helm/benchmark/run_expander.py

teetone

@percyliang I added comments in most files

teetone · 2023-01-02T01:43:56Z

src/helm/benchmark/adaptation/adapter_spec.py

@@ -0,0 +1,86 @@
+from dataclasses import dataclass, field


Moved out Substitution and AdapterSpec out from the old adapter.py to their own file.

teetone · 2023-01-02T01:45:21Z

src/helm/benchmark/adaptation/prompt.py

@@ -0,0 +1,65 @@
+import re


Moved out Prompt out from the old adapter.py to its own file.

teetone · 2023-01-02T01:48:54Z

src/helm/benchmark/adaptation/request_state.py

@@ -0,0 +1,75 @@
+from dataclasses import dataclass


Moved out RequestState out of the old adapter.py to its own file.

teetone · 2023-01-02T01:50:18Z

src/helm/benchmark/adaptation/scenario_state.py

@@ -0,0 +1,40 @@
+from collections import defaultdict, OrderedDict


Moved out ScenarioState out from the old adapter.py to its own file.

teetone · 2023-01-02T02:07:28Z

src/helm/benchmark/augmentations/contrast_sets_perturbation.py

@@ -66,18 +66,18 @@ def apply(self, instance: Instance, seed: Optional[int] = None) -> Instance:
        """
        rng: Random = self.get_rng(instance, seed)

-        perturbed_instance: str = instance.input


input of Instance is now an object instead of a string. Also, the variable nameperturbed_instance was a bit confusing.

teetone · 2023-01-02T06:01:02Z

src/helm/benchmark/adaptation/adapters/multiple_choice_separate_adapter.py

+from helm.common.request import Request
+from .in_context_learning_adapter import InContextLearningAdapter
+
+


The logic in this file was from the old Adapter.

teetone · 2023-01-02T06:01:29Z

src/helm/benchmark/adaptation/adapters/test_generation_adapter.py

+from helm.benchmark.adaptation.adapter_spec import AdapterSpec
+from .adapter_factory import AdapterFactory, ADAPT_GENERATION
+
+


Was part of the old test_adapter.py file.

teetone · 2023-01-02T06:01:36Z

src/helm/benchmark/adaptation/adapters/test_language_modeling_adapter.py

+from helm.benchmark.adaptation.adapter_spec import AdapterSpec
+from .adapter_factory import AdapterFactory, ADAPT_LANGUAGE_MODELING
+
+


Was part of the old test_adapter.py file.

teetone · 2023-01-02T06:01:43Z

src/helm/benchmark/adaptation/adapters/test_multiple_choice_joint_adapter.py

+from helm.benchmark.adaptation.adapter_spec import AdapterSpec
+from .adapter_factory import AdapterFactory, ADAPT_MULTIPLE_CHOICE_JOINT
+
+


Was part of the old test_adapter.py file.

teetone · 2023-01-02T06:16:37Z

src/helm/benchmark/adaptation/adapters/adapter_factory.py

+from helm.benchmark.adaptation.adapter_spec import AdapterSpec
+from helm.benchmark.window_services.tokenizer_service import TokenizerService
+from .adapter import Adapter
+from .generation_adapter import GenerationAdapter


All Adapters live in src/helm/benchmark/adaptation/adapters.

teetone · 2023-01-02T06:21:57Z

LGTM fwiw - did you test, since I noticed there are some minor changes to the scenario files beyond the main Input/Output change (e.g. renaming variables in wikfact), so curious just to make sure it works.

Yes, I tested extensively. I did a full run with the Together and TNLGv2 models to ensure the cache is safe.

src/helm/benchmark/scenarios/scenario.py

percyliang

Thanks again for the big refactor! Everything looks good - only thing is making TextInput separate from Input.

teetone · 2023-01-05T20:08:24Z

I will do another big rerun before I merge this PR.

teetone added v0.2.1 labels Dec 31, 2022

teetone requested a review from percyliang December 31, 2022 23:31

teetone added 13 commits January 1, 2023 04:47

progress

355493c

Input refactor

c0bb83b

input refactor

07be627

input refactor

d3d4650

fixed typo

639580d

moved adapter to src/helm/benchmark/adaptation

59a1306

Break apart AdapterSpec, Prompt, RequestState and ScenarioState from …

f750674

…adapter.py

Break apart AdapterSpec, Prompt, RequestState and ScenarioState from …

64dc713

…adapter.py

Evaluate all language models on reasoning scenarios (model=all_language)

d662e53

Output/Input refactor

234fb88

break apart Adapter

ac9892f

fixed babiqa

162d189

minor cleanup

3babb4b

teetone force-pushed the inputrefactor branch from c9a106d to 3babb4b Compare January 1, 2023 12:49

percyliang reviewed Jan 1, 2023

View reviewed changes

src/helm/benchmark/adaptation/adapters/adapter.py Outdated Show resolved Hide resolved

teetone added 2 commits January 1, 2023 15:47

Fixed IMDB contrast sets

a85c6ab

rename and improve comments

59f9902

rishibommasani reviewed Jan 2, 2023

View reviewed changes

src/helm/benchmark/scenarios/newsqa_scenario.py Outdated Show resolved Hide resolved

src/helm/benchmark/run_expander.py Outdated Show resolved Hide resolved

teetone added 2 commits January 1, 2023 22:03

minor

91f13d1

reuse test logic

7df73b9

teetone commented Jan 2, 2023

View reviewed changes

fix typo

4d169f7

teetone requested a review from percyliang January 2, 2023 06:22

percyliang reviewed Jan 5, 2023

View reviewed changes

src/helm/benchmark/scenarios/scenario.py Show resolved Hide resolved

percyliang approved these changes Jan 5, 2023

View reviewed changes

teetone added 2 commits January 5, 2023 09:55

rename to text_code

067820b

cleanup

1776e4e

teetone merged commit 0eaaa62 into main Jan 6, 2023

teetone deleted the inputrefactor branch January 6, 2023 18:24

yifanmai mentioned this pull request Jan 10, 2023

Fix frontend to reflect input and output refactor #1306

Merged

yifanmai mentioned this pull request Feb 1, 2023

Run helm-summarize based on downloaded JSON output files #1310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`Input`, `Output` and `Adapter` refactor #1280

`Input`, `Output` and `Adapter` refactor #1280

teetone commented Dec 31, 2022 •

edited

Loading

percyliang commented Jan 1, 2023

teetone commented Jan 2, 2023

rishibommasani left a comment

teetone left a comment

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone Jan 2, 2023

teetone commented Jan 2, 2023

percyliang left a comment

teetone commented Jan 5, 2023

		@@ -0,0 +1,40 @@
		from collections import defaultdict, OrderedDict

		from helm.common.request import Request
		from .in_context_learning_adapter import InContextLearningAdapter

		from helm.benchmark.adaptation.adapter_spec import AdapterSpec
		from .adapter_factory import AdapterFactory, ADAPT_GENERATION

Input, Output and Adapter refactor #1280

Input, Output and Adapter refactor #1280

Conversation

teetone commented Dec 31, 2022 • edited Loading

Changes

percyliang commented Jan 1, 2023

teetone commented Jan 2, 2023

rishibommasani left a comment

Choose a reason for hiding this comment

teetone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teetone commented Jan 2, 2023

percyliang left a comment

Choose a reason for hiding this comment

teetone commented Jan 5, 2023

`Input`, `Output` and `Adapter` refactor #1280

`Input`, `Output` and `Adapter` refactor #1280

teetone commented Dec 31, 2022 •

edited

Loading