SLEAP for ants

[jstnkttll331]

Hello,

I have been using the SLEAP neural net for ant identification for a few
years now, so first let me say thank you for making my research possible
and user friendly. I really appreciate all the hard work that's been
done, and am continually impressed by the program's features.

I wanted to reach out and request some pointers before fine tuning the
training package for our next data run. My use case is probably a little
unique, as rather than identifying a small number of complex skeletons,
I use SLEAP to identify 30-40 instances of a very simple skeleton in a
large frame, consisting of a head, thorax, and gaster of an ant. I am
using the resulting position and orientation maps to study the evolution
and dynamics of traffic patterns in the foraging column of an ant
colony. I've attached a link to a few examples of our preprocessed video
data, instances, and the current best trained models with the hopes that
you may be able to suggest better starting parameters (stride, receptive
field, augenmentation, etc) for the next round of iterative training
given the size of the ants, use case, etc. I know this is a lot to ask,
so maybe there is a good resource out there you can direct me to?

Colab has been working great.

files

Many thanks,

Justin Kittell
Ursell Lab
University of Oregon
371 Willamette Hall
Eugene, OR 97403-1274

[talmo]

Hi @jstnkttll331,

Thanks for reaching out! Looks like a cool application!

From taking a quick look I see two issues:

1. The thorax nodes don't seem to appear. This is strange -- did you have them set as visible in the training data? Did you import them from somewhere else instead of labeling within SLEAP?

2. The FPS of the video seems pretty low, resulting in pretty large jumps between frames when the animals are walking. The result is that we get a ton of new tracks pretty constantly. Honestly, even if the videos were recorded at a higher FPS, the ants can go in and out of the field of view, which will make tracking pretty tricky.

I'm guessing you just want short tracklets from which you can extract your dynamics, but let me know what kind of results you were looking for and we can work on recommendations that work for your use case.

In general, our Troubleshooting workflows guide has some tips on how to tune the models which might also help.

Cheers,

Talmo

[jstnkttll331]

Hi Talmo,

Thanks for the quick response. My research is centered around studying the relative positions and orientations of ant neighbors to understand how they organize their traffic, so tracking is actually unnecessary (for now). I just ignore the track information in data processing, and study ants from a single frame. I've attached a neat example plot, it shows a perspective-based alignment field (how aligned you and your neighbors are in any direction) for the 'average' ant in an older data set.

If I open the .slp file in the GUI and click "Add Instances from All Prediction on Current Frame" the thoraxes appear, but these features are almost always misplaced (sometimes dramatically). For now I've been working with just the head and gaster and approximating the thorax position as the midpoint. This is the 'issue' that drove me to reach out in the first place, as it wasn't happening with a previous imaging set up where the ants were slightly larger.

I'll read through workflows, they seem very helpful, thank you. Just curious, is bright-field or dark-field imaging preferred for SLEAP?

Best,

• Justin

[talmo]

Hi @jstnkttll331,

Notice that when you add the instances from predictions that the thorax nodes are small and have a grey background. This is because they are marked as "not visible". When this is the case, we initialize them in a random location since the actual position of the node doesn't matter as SLEAP will learn to ignore those.

This is not what you want -- you can right-click the nodes to mark them as "visible" and re-train the model (see this page in the tutorial). Make sure to go back over your previous labeled frames to check if you marked any other thorax nodes as not visible, toggle them to visible and position them correctly before retraining.

Let me know if you have any questions about that or if you run into any issues!

Talmo