Running against local model service

[drisspg]

For the neurips competetion we woud like to be able to run Helm against a model running local to the machine. I.e run helm in a container and have it send requests to another container. I started implementing a new local http client following the "adding a model" readme.

However I see that there already appears to be something pretty similiar here:

helm/src/helm/proxy/services/server_service.py

Line 38 in 05ce7e8

class ServerService(Service):

Do you think this would fit our needs? How exactly does one use this with helm-run?
I would imagine it to be something like:

RUN echo 'entries: [{description: "mmlu:subject=philosophy,model=local_model", priority: 1}]' > run_specs.conf

helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10 --local-path="http://localhost"

[msaroufim]

cc @yifanmai - do you mind if we get label permissions only so we can tag features as competition

[yifanmai]

Granted permissions.

ServerService is probably too heavyweight. It basically runs a full "playground API" server (which we also call a "proxy" server elsewhere in the code), which has a lot of extra functionality that you don't need (e.g. user authentication, user quotas).

I think you would need to create a new kind of server and client that's basically subset of the full playground API:

• Make a new HTTP server library with make_request(), tokenize() and decode() endpoints similarly to the full playground server.
• Make a new Client subclass for which the endpoint URL(s) is configurable, either by flag or environment variable, that only implements make_request(), tokenize() and decode() similarly to the full playground client

I also think that the current abstractions would make it difficult to reuse code, so it might be better to make these separate implementations.

I'm not sure what the right names of these clients and servers would be... possibly CompetitionClient, or NativeClient (because it uses "native" CRFM JSON schemas).

[timothylimyl]

this feels like a feature that should be integrated into helm. I think most people will want to set up their own local models to run helm evaluation.

[drisspg]

This was landed in #1693 there are more changes incoming to make this solution more generic