-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add --no_balance
flag to not balance datasets
#287
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Run with elk elicit gpt2 imdb --no_balance True --disable_cache --max_examples 100 100 --num_gpus 1 --max_inlp_iter 4
and seems to work.
Added some comments though
@@ -65,6 +65,9 @@ class Extract(Serializable): | |||
binarize: bool = False | |||
"""Whether to binarize the dataset labels for multi-class datasets.""" | |||
|
|||
no_balance: bool = False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just make it
balance: bool = True ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would also avoid having that:
balance=not cfg.no_balance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because it would be unclear how to use the flag to disable balancing from the CLA. --balance False
or something is weirder than --no_balance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
--balance False does not seem weirder than --no_balance True to me.
But okay, it's fine for me
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think I agree with you now
@@ -212,12 +212,11 @@ def inlp( | |||
p = y.float().mean() | |||
H = -p * torch.log(p) - (1 - p) * torch.log(1 - p) | |||
|
|||
if max_iter is not None: | |||
d = min(d, max_iter) | |||
max_iter = max_iter or d |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's just some refactoring which has nothing to do with the balancing I guesS?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right, I also added a max_iter flag and this was a necessary refactoring
@@ -6,7 +6,7 @@ | |||
|
|||
|
|||
def train_supervised( | |||
data: dict[str, tuple], device: str, mode: str | |||
data: dict[str, tuple], device: str, mode: str, max_inlp_iter: int | None = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's a new feature not related to the balancing either, right?
No description provided.