Update main to define arguments directly

[snova-zoltanc]

Summary

Change main to accept arguments so it can be called from python (not just CLI). This is necessary for Studio to be able to call main

PR Checklist

• My PR is less than 500 lines of code
• I have added sufficient comment as docstrings in my code
• I have made corresponding changes to the documentation

[snova-zoltanc]

Example of it working

>>> from generative_data_prep import main
>>> main("hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: main() missing 32 required positional arguments: 'output_path', 'overwrite_output_path', 'log_file_path', 'pretrained_tokenizer', 'tokenizer_class', 'vocab_file', 'merges_file', 'special_tokens_dict', 'categories_path', 'input_path', 'disable_space_separator', 'keep_prompt_only_sequences', 'ignore_input_format_error', 'prompt_keyword', 'completion_keyword', 'shuffle', 'num_workers', 'do_not_balance_hdf5', 'keep_split_jsonls', 'max_seq_length', 'input_packing_config', 'packing_boundary', 'attention_boundary', 'num_training_splits', 'num_dev_splits', 'num_test_splits', 'dev_ratio', 'test_ratio', 'prompt_prefix', 'prompt_postfix', 'apply_chat_template', and 'silent

[snova-jerrym]

Looks like some tests are still failing. Is it possible to do something like a proxy function? Like the old way of calling main is kept and create a new jumper function that takes args and expands it to call the new main.

[snova-edwardm]

It seems likely that this line causes the error. vars can be used for a class but not for a dict.