Causal Classification Problem Type

[psinger]

New Problem Type for supporting Causal Classification.

Workflow:

• Answer column contains integer encoder binary or multiclass target
• Problem Type is Causal Classification Modeling
• Metrics are AUC and Accuracy
• Number of classes need to be selected via cfg
• Pushing to HF and subsequent loading with additional head supported

[maxjeblick]

Thanks a lot for adding the classification problem type! The code looks good (I've left some minor comments), and training also works as expected.

As the setup (loss-metric-num classes) is currently manual, I'd suggest to add some sanity checks (that raise an exception) in the code to check for user input errors. At some later point, we can potentially add automatic handling of the settings, based on the input data. I've ran into the following issues:

• Using a dataset where target values are [1, 2, 3 ,4, 5]. num_classes needs to be set to 6, as they are 0-indexed.
• Forgot to change metrics/loss accordingly. For metric, experiment will fail at 100%, only.

Apart from that, we probably want to show the predicted class in the validation prediction insights and also add it to validation_predictions.csv.

Some smaller things (can also be addressed later):

• Tokenized input text also shows the classification label.
• We probably want to hide the Chat Window (funny conversation though :D )

[psinger]

Thanks @maxjeblick - will address.

I discussed with @pascal-pfeiffer that we can expect here for users to provide the correct inputs, but I agree that exceptions should be more clear.

[maxjeblick]

LGTM, thanks a lot for the implementation!

I've left a small comment about ConfigNLPCausalClassificationPrediction, apart from that it looks good!

[samvelkoch]

Hello. I played a bit with LLM Studio in H20 Aquarium while solving H2O predict LLM Competition. Models fine-tuned for Casual Classification Problems have no classification_head.pth in downloaded or pushed to HF model files. Any ideas what I have missed in the settings or why that issue happens? Thank you!

[maxjeblick]

Hi, @samvelkoch, I'll have a look into this.
Could you create a new issue for better tracking? Thanks!

[samvelkoch]

Hi, @samvelkoch, I'll have a look into this. Could you create a new issue for better tracking? Thanks!

Thanks for quick reply. I've published it here since I'm not sure that it is a real issue or just my lack of experience with LLM Studio. Could you please check? I'll publish an issue now.