Language Modelling Tasks as Objects (LaMoTO) provides a framework for language model training (masked and causal, pretraining and finetuning) where the tasks, not just the models, are classes themselves.
It abstracts over the HuggingFace transformers.Trainer
with one goal: reduce the entire model training process to a single
method call task.train(hyperparameters)
.
Let's say you want to train a RoBERTa-base model for dependency parsing (for which, by the way, there is no HuggingFace class). This is how you would do that in LaMoTO, supported by the magic of ArchIt:
from archit.instantiation.basemodels import RobertaBaseModel
from archit.instantiation.heads import DependencyParsingHeadConfig, BaseModelExtendedConfig
from lamoto.tasks import DP, getDefaultHyperparameters
# Define task hyperparameters.
hp = getDefaultHyperparameters()
hp.MODEL_CONFIG_OR_CHECKPOINT = "roberta-base"
hp.archit_basemodel_class = RobertaBaseModel
hp.archit_head_config = DependencyParsingHeadConfig(
head_dropout=0.33,
extended_model_config=BaseModelExtendedConfig(
layer_pooling=1
)
)
# Instantiate language modelling task as object, and train model.
task = DP()
task.train(hyperparameters=hp)
- Train models on >15 pre-training/fine-tuning tasks. See a list by importing
from lamoto.tasks
.- Model architectures come from ArchIt, which means that as long as you have a
BaseModel
wrapper for your language model backbone, you can train it on any task, regardless of whether you wrote code defining the backbone-with-head architecture required for that task. - Custom (i.e. given) architectures are also supported.
- Model architectures come from ArchIt, which means that as long as you have a
- Evaluate models with a superset of the metrics in HuggingFace's
evaluate
, with custom inference procedures (see e.g. strided pseudo-perplexity or bits-per-character). - Augment datasets before training or evaluating by somehow perturbing them.
- Supports TkTkT tokenisers.
- Weights-and-Biases integration.
If you don't want to edit the source code yourself, run
pip install "lamoto[github] @ git+https://github.com/bauwenst/LaMoTO"
and if you do, instead run
git clone https://github.com/bauwenst/LaMoTO
cd LaMoTO
pip install -e .[github]
To be able to use the Weights-and-Biases integration, make sure you first run wandb login
in a command-line terminal
on the system you want to run on.