Allow users to add additional models via configuration file #1761

yifanmai · 2023-08-01T21:56:27Z

This allows the user to register additional models via configuration file.

Example usage: run helm-run --run-specs mmlu:subject=anatomy,model=ai21/j2-light --suite v1 -m 5 --model-metadata-paths models_metadata.yaml --model-deployment-paths model_deployments.yaml.

model_metadata.yaml:

models:
  - name: ai21/j2-light
    display_name: Jurassic-2 Light (7.5B)
    description: Jurassic-2 Light (7.5B parameters) ([docs](https://www.ai21.com/blog/introducing-j2))
    creator_organization: AI21 Labs
    access: limited
    num_parameters: 7500000000
    release_date: 2023-03-09

model_deployments.yaml:

model_deployments:
  - name: ai21/j2-light
    model_name: ai21/j2-light
    # For now, we only support HuggingFaceWindowService and Hugging Face tokenizers
    # TODO: Support other window services and tokenizers
    tokenizer_name: "huggingface/gpt2"
    max_sequence_length: 8191
    client_spec:
      class_name: "helm.proxy.clients.ai21_client.AI21Client"
      args:
        url: "https://api.ai21.com/studio/v1/j2-light/complete"

credentials.conf:

deployments: {
    "ai21/j2-light": your_key_here
}

yifanmai · 2023-08-05T01:49:25Z

@percyliang could you take a look? There are three different groups that need this (safety evals, NeurIPS Efficiency Challenge, AWS) - the latter two urgently - so I'd like to get it in by Monday if possible.

The main thing to hammer down is the config file format:

Should format be YAML or HOCON? Currently: HOCON
Should the field name be model_type vs implemention_type vs something else? Currently: model_type
Should we use a short model_type string, or do something like ObjectSpec and require the class name? Currently: short string
Should the config file be a dict or list? Should name be the keys or a field in the dataclass? Currently: Dict, but I think it should be a list actually.

percyliang · 2023-08-05T20:21:30Z

src/helm/benchmark/model_registry.py

+    """Configuration for a registered model."""
+
+    model_type: str
+    """Name of the client type."""


Why different names - model type versus client type? Can we standardize and explain / give an example?

percyliang · 2023-08-05T20:25:56Z

src/helm/benchmark/model_registry.py

+    # TODO(#1673): Add tokenizer name and sequence length fields.
+
+    args: Optional[Dict[str, Any]] = None
+    """Configuration for the model"""


It's a bit mysterious what the args are supposed to be - can we explain what these are and give one example? Do these have to be untyped?

percyliang · 2023-08-05T20:26:58Z

src/helm/benchmark/run.py

@@ -256,6 +263,8 @@ def main():
        register_huggingface_hub_model_config(huggingface_model_name)
    for huggingface_model_path in args.enable_local_huggingface_models:
        register_huggingface_local_model_config(huggingface_model_path)
+    for model_config_path in args.model_config_paths:
+        register_model_configs_from_path(model_config_path)


Does this mechanism subsume the Hugging Face local model loading?

Yes, eventually this will replace the Hugging Face mechanism (after some additional refactoring).

percyliang · 2023-08-05T20:29:28Z

src/helm/proxy/clients/auto_client.py


            if get_huggingface_model_config(model):
                from helm.proxy.clients.huggingface_client import HuggingFaceClient

                client = HuggingFaceClient(cache_config=cache_config)
-            elif organization == "openai":
+            elif model_type == "openai":


I think the term 'model_type' is not exactly right - I think more of Transformer versus RNN. Why not stick with organization? It is really an organizational thing and has nothing to do with the underlying model.

I think we can just make this a ObjectSpec.class_name. See the new example model_deployments.yaml.

percyliang · 2023-08-05T20:30:31Z

src/helm/proxy/clients/ai21_client.py

@@ -20,6 +22,28 @@ class AI21RequestError(Exception):
    pass


+@dataclass(frozen=True)


I'm a bit confused why we need the custom model config stuff for AI21 given that is something we define and support as opposed to configured by users? Does that mean we need another file that we need to pass into HELM to run AI21 evals?

Cleaned this up; now we just pass in the additional parameters via AI21Client.__init__()

percyliang · 2023-08-05T20:34:00Z

There's also defining the model in a customizable schema.yaml - could we define configs in terms of that and have an implementations field? I think it'd be nice to start to unify our schemas a bit more to avoid redundancy / things falling out of sync.

yifanmai · 2023-08-08T00:45:53Z

Modified this to be closer to our discussion.

model_metadata.yaml:

models:
  - name: ai21/j2-light
    display_name: Jurassic-2 Light (7.5B)
    description: Jurassic-2 Light (7.5B parameters) ([docs](https://www.ai21.com/blog/introducing-j2))
    creator_organization: AI21 Labs
    access: limited
    num_parameters: 7500000000
    release_date: 2023-03-09

model_deployments.yaml:

model_deployments:
  - name: ai21/j2-light
    model_name: ai21/j2-light
    # For now, we only support HuggingFaceWindowService and Hugging Face tokenizers
    # TODO: Support other window services and tokenizers
    tokenizer_name: "huggingface/gpt2"
    max_sequence_length: 8191
    client_spec:
      class_name: "helm.proxy.clients.ai21_client.AI21Client"
      args:
        url: "https://api.ai21.com/studio/v1/j2-light/complete"

credentials.conf:

deployments: {
    "ai21/j2-light": your_key_here
}

percyliang · 2023-08-08T04:16:15Z

src/helm/benchmark/model_deployment_registry.py

+
+@dataclass(frozen=True)
+class ModelDeployment:
+    name: str


Add docstring about what a deployment is (as opposed to a model).

percyliang · 2023-08-08T04:17:42Z

src/helm/benchmark/model_deployment_registry.py

+    model_deployments: List[ModelDeployment]
+
+
+_name_to_model_deployment: Dict[str, ModelDeployment] = {}


Do we want to use a singleton here? I believe schemas are passed around, which might make things more modular / easier to test?

This may be the result of merging multiple configuration files (e.g. the repo's "defaults" configuration file + the user's or 3P repo's configuration files). So this has to be dynamically constructed. So we can't do the same thing as schema.py.

percyliang · 2023-08-08T04:40:43Z

src/helm/benchmark/model_metadata_registry.py

+class ModelMetadata:
+    name: str
+
+    # Organization that originally created the model (e.g. "EleutherAI")


Do we want these comments to be """ under the field so that they get pulled into documentation?

percyliang · 2023-08-08T04:43:08Z

src/helm/benchmark/model_metadata_registry.py

+    #   Note that this may be different from group or the prefix of the model `name`
+    #   ("together" in "together/gpt-j-6b") as the hosting organization
+    #   may be different from the creator organization. We also capitalize
+    #   this field properly to later display in the UI.


Not sure if we want to capitalize...because then we will have Meta and meta... I don't think we should conflate unique names with display names...for example, the display name might have a space, and I don't think we want the creator_organization to have a space.

In the version in schema.yaml, this was mixed case... maybe we should revisit later.

percyliang · 2023-08-08T04:43:56Z

src/helm/benchmark/model_metadata_registry.py

+    # but we set it as an int for plotting purposes.
+    num_parameters: Optional[int] = None
+
+    # Tags corresponding to the properties of the model.


Comment that this will probably go...this is more of a property of the deployment, right?

The problem is that some things are properties of the model (e.g. is this a language model or an image model?) and some are properties of the deployment (e.g. window size).

percyliang · 2023-08-08T04:44:27Z

src/helm/benchmark/model_metadata_registry.py

+    creator_organization: Optional[str] = None
+
+    # How this model is available (e.g., limited)
+    access: Optional[str] = None


can say more explicitly this is the maximum access over all deployments

…-crfm#1761)

yifanmai force-pushed the yifanmai/fix-aws-ai21 branch from 9fb8d51 to cdb46b7 Compare August 4, 2023 23:33

yifanmai requested a review from percyliang August 5, 2023 01:45

yifanmai marked this pull request as ready for review August 5, 2023 01:46

yifanmai changed the title ~~AI21 model URL registration~~ Allow users to add additional models via configuration file Aug 5, 2023

percyliang reviewed Aug 5, 2023

View reviewed changes

yifanmai mentioned this pull request Aug 5, 2023

Neurips client #1693

Merged

yifanmai requested a review from percyliang August 8, 2023 00:45

percyliang reviewed Aug 8, 2023

View reviewed changes

percyliang approved these changes Aug 8, 2023

View reviewed changes

yifanmai added 7 commits August 7, 2023 23:15

AI21 model URL registration

a1a995f

Model registration round 2

18dfbc7

Fixes

91647e7

Misc fixes

d0db8e6

Fix

7ac9812

Add missing files

9377622

Review fixes

7a4b92b

yifanmai force-pushed the yifanmai/fix-aws-ai21 branch from 423bbe5 to 7a4b92b Compare August 8, 2023 06:18

Argh

f8449a2

yifanmai added 2 commits August 7, 2023 23:35

Remove bad whitespace

ca41068

Update comments

fb707a9

yifanmai merged commit 3857be3 into main Aug 8, 2023
3 checks passed

yifanmai deleted the yifanmai/fix-aws-ai21 branch August 8, 2023 15:38

danielz02 pushed a commit to danielz02/helm that referenced this pull request Sep 7, 2023

Allow users to add additional models via configuration file (stanford…

5e4452e

…-crfm#1761)

danielz02 pushed a commit to danielz02/helm that referenced this pull request Sep 7, 2023

Allow users to add additional models via configuration file (stanford…

2e5399c

…-crfm#1761)

danielz02 pushed a commit to danielz02/helm that referenced this pull request Sep 7, 2023

Allow users to add additional models via configuration file (stanford…

95616c8

…-crfm#1761)

yifanmai mentioned this pull request Oct 7, 2023

Make it easy to run HELM on new models #1219

Closed

RawthiL mentioned this pull request Dec 7, 2023

Add tokenizer config path argument #2110

Closed

yifanmai mentioned this pull request Feb 2, 2024

Configuration file for adding models to HELM #1673

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow users to add additional models via configuration file #1761

Allow users to add additional models via configuration file #1761

yifanmai commented Aug 1, 2023 •

edited

Loading

yifanmai commented Aug 5, 2023 •

edited

Loading

percyliang Aug 5, 2023

percyliang Aug 5, 2023

percyliang Aug 5, 2023

yifanmai Aug 8, 2023

percyliang Aug 5, 2023

yifanmai Aug 8, 2023

percyliang Aug 5, 2023

yifanmai Aug 8, 2023

percyliang commented Aug 5, 2023

yifanmai commented Aug 8, 2023

percyliang Aug 8, 2023

percyliang Aug 8, 2023

yifanmai Aug 8, 2023

percyliang Aug 8, 2023

yifanmai Aug 8, 2023

percyliang Aug 8, 2023

yifanmai Aug 8, 2023

percyliang Aug 8, 2023

yifanmai Aug 8, 2023

percyliang Aug 8, 2023

yifanmai Aug 8, 2023

		@@ -20,6 +22,28 @@ class AI21RequestError(Exception):
		pass


		@dataclass(frozen=True)

		model_deployments: List[ModelDeployment]


		_name_to_model_deployment: Dict[str, ModelDeployment] = {}

Allow users to add additional models via configuration file #1761

Allow users to add additional models via configuration file #1761

Conversation

yifanmai commented Aug 1, 2023 • edited Loading

yifanmai commented Aug 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

percyliang commented Aug 5, 2023

yifanmai commented Aug 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifanmai commented Aug 1, 2023 •

edited

Loading

yifanmai commented Aug 5, 2023 •

edited

Loading