Training dies on epoch 5

[daskandalis]

Using some variants of the following training parameters, training always dies on epoch 5. What could be causing this? Setting trainer.config.optimization.learning_rate_schedule.reduce_on_plateau = False does not help.

import os
import glob
import pathlib
import sleap

pkg = 'single_instance.json'
labels = 'label_package.pkg.slp'
output_dir = 'models'

num_epochs = 200
batch_size = 2 # debugging
batches_per_epoch = 50 # debugging

# UNet parameters
max_stride = 64
filters = 64
output_stride = 4
filter_rate = 1.5
middle_block = True

# Load config
cfg = sleap.load_config(pkg)

cfg.data.labels.training_labels = labels

trainer = sleap.nn.training.Trainer.from_config(cfg)

trainer.config.optimization.epochs = num_epochs
trainer.config.outputs.runs_folder = output_dir

trainer.config.optimization.batch_size = batch_size
trainer.config.optimization.batches_per_epoch = batches_per_epoch

trainer.config.model.max_stride = max_stride
trainer.config.model.output_stride = output_stride
trainer.config.model.heads.single_instance.output_stride = output_stride
trainer.config.model.filters = filters
trainer.config.model.filters_rate = filter_rate
trainer.config.model.middle_block = middle_block

trainer.train()

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[gitttt-1234]

Hi @daskandalis,

Could you please check if trainer.config.optimization.early_stopping.stop_training_on_plateau is set to False as well? If not, this could be one of the reasons. Could you try setting it to False or reducing the value for plateau_min_delta? Let us know if this doesn't work!

Thanks!

Divya

[gitttt-1234]

Hi @daskandalis,

Could you please paste your config here/ output log if any? Here's a google form link to upload your files.

Thanks,

Divya

[daskandalis]

As far as I can tell, this is a GPU memory issue. Why it is being killed on the 5th epoch isn't clear, but I'm noticing more often that the model can train for some number of epochs and then is suddenly killed.

[gitttt-1234]

Hi @daskandalis,

Could you try running this command sleap.disable_preallocation() before you start training? We run this here before the training starts to by prevent TensorFow from using up the entire memory.
Let us know if this works!

Thanks,

Divya

[daskandalis]

Unfortunately not, same result: killed at 5 epochs.

[gitttt-1234]

Hi @daskandalis,

It would be great if you could share your output logs and training config file here so that we could look deeper into this from our end.

Thanks,

Divya