Fish tail consistently predicted as invisible

[noamco23-meet]

I've been training a model 5 or 6 times, each time labelling 50-100 frames but the model consistently predicts the bottom of the fish's tail as invisible, despite it being visible in every single frame that I've labelled.
Here is what I would like to be shown:

But in reality the model predicts:

I also sometimes get predictions like this:

Which claims not to be in the training data although I'm not sure what that means. I currently only have one instance throughout the video but the predictions sometimes splits the instance into multiple track so that's also a problem.

I don't know what would lead the model to think that the tail is invisible, particularly seeing that it does sort of guess an approximate location of where it is. Is the tail too dark? I'm using a topdown model, and other than the tailB mark, the model predicts most of the other features quite accurately. Any help is greatly appreciated.

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[talmo]

Hi @noamco23-meet,

Thanks for sharing all this info!

It would help to narrow it down if you can provide a couple of screenshots of the model configuration or the contents of the training_job_config.json file for the centered instance model.

If I had to guess, one of two things could be happening (or both):

1. The tailB node was marked as not visible in some of the ground truth frames and you might not have spotted it. This happens sometimes when the node is in the right place but marked as not visible (grey, italics text) and you forget to right-click it to mark it as visible. When this happens, the model learns to treat it as background "sometimes" and can produce pretty inconsistent behavior depending on the dataset.
2. The tailB might be falling outside of the bounding box of the crop. In some rare cases, this can happen due to an issue in how SLEAP attempts to automatically calculate the optimal bounding box size.
   • This calculation is influenced by which node you choose as the anchor node (and especially if you don't choose one) since it'll shift where the center of the crop will be placed. I'd recommend choosing tailT as the anchor node since it's very central in your skeleton configuration.
   • The calculation can also be affected by the training data. SLEAP computes the crop size based on the size of the instances in your labels, so if (1) above is an issue, it can throw off the calculation. It can also be off if it just happens that the frames that you labeled have animals that span a smaller area just by virtue of their pose in the specific frames. I'd recommend setting the crop size manually in your centered instance model. I'm not sure about your resolution, but eyeballing it, maybe 192 or 256?

Let us know if these work, if you want to give us a bit more info about the model, or if you have any other questions!

Cheers,

Talmo

[noamco23-meet]

Hi @talmo,
Thank you so much -- your suggestion to use tailT as an anchor worked for me (before that I had used bladderT). I was wondering if you could elaborate more on the crop size setting, I'm using a python script for my models as shown in the colab notebooks and I'm not sure how to implement it.

Thanks again,
Noam

[talmo]

Cool! How do you set up the training configuration?

If you're exporting the training package zip file from the GUI, you can change this setting near the top left of the centered instance tab before exporting.

You can also find the crop size in the JSON files that are exported and just edit it manually.

Let us know if you have any questions!

Talmo