Issues in training with single instance model and large frame sizes

[anarac97]

Hi, I'm using SLEAP to predict single animal poses. I'm importing two videos of one single fruit file in grayscale, recorded at 100fps, resolution of 900 x 900 and with ~ 10000 frames each. I chose and labeled 10 random frames of each video and started to train my model using basic hyperparameters. So far, I've ran over 10 training sessions and I'm facing one of the three situations every time I resume training: 1) no instances are predicted in the frames, 2) instances are predicted in all frames and metrics are not generated or 3) instances are predict, metrics are generated and they depict a real poor model performance (I've had situations where OKS mAP = 0), regardless of correcting of previous predictions. I've adjust the hyperparameters several times and, more recently, I've tried to use the training configurations of a previous model from other experiment that showed good metrics to run my training now. Nothing is working out.

I've been a SLEAP user for a while now and I've successfully managed to train a model in the past using very identical settings to the ones I'm using now (e.g., video settings of the images I'm importing to my project and training settings (i.e., skeleton, labeling, model hyperparameters)). Am I missing something? I've reached almost 300 labeled frames and I see no improvement at all...

Would be very grateful for you help!

Thanks,

Rita

Your personal set up

Versions:
OS: Windows-10-10.0.19041-SP0
SLEAP: 1.3.0
TensorFlow: 2.6.3
Numpy: 1.19.5
Python: 3.7.12

SLEAP installation method (listed here):

• Conda from package
  (Not using Mambaforge)

Screenshots

1. Interface with example of an unlabeled frame

2. Examples of the hyperparameters I've been using

3. Training sessions so far

[talmo]

Hi @anarac97,

Thanks for all the info!

Since you have a single fruit fly but a large frame size, the single instance model will definitely struggle. The reason is that most of the image is background, which means that the neural network is strongly encouraged to just learn to predict that there are no landmarks anywhere and it will still be correct the vast majority of the time.

I'd recommend trying a top-down model instead. In this model type, SLEAP will first detect the centroid of the fly in a scaled down version of the image, and then crop a box around it and predict the poses at high resolution. This will save a ton of computing time and will encourage the model to focus on just the relevant part of the image.

Even though top-down models were designed for data with multiple animals, you can constrain SLEAP to only predict one instance per frame by adjusting the Max Instances option in the training configuration window (it's on the first tab, below the model type selection).

I recommend that you give the top-down model a try with all default settings except for rotation augmentation, which you can set to +-180.

Give that a go and let us know how it goes! If that doesn't improve the results, we can iterate from there.

Cheers,

Talmo

[anarac97]

Hi Talmo,

Sorry for the late response, I could only handle this matter now. I followed your advice and went on retraining my model using a top-down approach and default hyperparameters (with the exception for rotation augmentation). Now, with ~ 550 labeled frames, I'm still not satisfied with the model's performance. With your recommended strategy, mAP for the centered instance model improved significantly at ~200 labeled frames, so I tuned the learning rate to get better results but my next training sessions were not successful. Any thoughts on this??

Cheers,

Rita

Recent models:

Hyperparameters in the first training sessions:

Hyperparameters in the last few sessions:

[eberrigan]

Hi @anarac97,

Did you try the Auto option for Crop Size?

Thanks!

Elizabeth

[anarac97]

Hi Elizabeth,

No, will try and let you know!

Thanks,

Rita

[anarac97]

Hi @eberrigan,

Ticking the auto option for crop size helped! The model performance has been increasing with further training. Thanks for your suggestion.

Best,

Rita