Editing previously labelled frames

[rupertoverall]

We have labelled a bunch of frames in a project and now want to edit some of these in different ways to see if it improves the model. The idea is to use the remainder of the manually-labelled frames as a test set and see which alternative labelling strategy performs best.
Ideally, we would keep all of the frames in the project - but we want to tag only a subset as 'edited' and 'for inference'. Is there a way to do this without splitting the dataset up into sub-projects?
I see it is possible to train on only 'user labelled frames' but all of the frames in this project have been user labelled already. Maybe this flag can be reset?

P.S. As a possible extra complication, this particular project consists of many 'videos' each consisting of one frame (drawn from a variety of source videos). So a solution that allows flagging at the video level would also work here (but possibly not in future projects).

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[rupertoverall]

We have labelled a bunch of frames in a project and now want to edit some of these in different ways to see if it improves the model. The idea is to use the remainder of the manually-labelled frames as a test set and see which alternative labelling strategy performs best. Ideally, we would keep all of the frames in the project - but we want to tag only a subset as 'edited' and 'for inference'. Is there a way to do this without splitting the dataset up into sub-projects? I see it is possible to train on only 'user labelled frames' but all of the frames in this project have been user labelled already. Maybe this flag can be reset?

P.S. As a possible extra complication, this particular project consists of many 'videos' each consisting of one frame (drawn from a variety of source videos). So a solution that allows flagging at the video level would also work here (but possibly not in future projects).

After some tests, it seems what we need is the ability to restrict training to just a subset of the frames. It looks like the training config file might support this, with some promising-looking parameters in the 'labels' section:

        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [2,3,19,24,25,31,34,41,45,53,56,60,...],
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },

As you can see, I have tried passing a bunch of indices to 'training_inds'. This runs, but I can't verify that it is only using those frames/videos for training. I am guessing a bit here though as I have not managed to dig out the documentation for these config files. Can someone point me to where that is? Or has anyone successfully done training only using a subset of the training dataset?

[rupertoverall]

Thanks Divya, it sounds like I stumbled onto the right track then!

1. Where do I find the training logs? The only log-like file I find is 'training_log.csv', which just contains the loss history. Sorry for the naive question, but I'm not sure where to look.
2. The discussion you linked to mentioned the function Labels.make_training_splits, which looks close to what I wanted for cutting a training set out of our superset. I don't see how to use this to split on an explicit set of frames though. Something like this that takes a vector of instance indices would be the ticket. Does that exist?

[gitttt-1234]

Hi @rupertoverall,

You can find the logs in the terminal where you start the SLEAP GUI. Once you start training, the logs will show up in the terminal. For splitting the labels, I don't think we have a function to split the labels file based on explicit indices. The best option would be to specify the indices in the config file (training_inds) and train the model using CLI.

Let us know if you have any questions!

Thanks,

Divya

[rupertoverall]

I should have specified, I am running the training on a remote machine using the CLI. It would be great to see some log output to reassure me that I have set the config file up correctly. I think I am doing this right already, I am just not sure how to verify that.
Is there a command line parameter to generate verbose log files? Or are they already there somewhere (and I am just looking in the wrong place)?

[talmo]

Hi @rupertoverall,

We don't save the full terminal logs anywhere, but you can capture this from wherever you're calling sleap-train. For example, from a bash script:

#!/bin/bash

conda activate sleap
sleap-train model_config.json labels.pkg.slp > training_logs.txt

This will save the outputs to a file called training_logs.txt. You can get fancier and move it to the model directory by parsing the JSON file or specifying the paths ahead of time rather than using the auto-generated ones.

Let us know if you have any questions!

Cheers,

Talmo

[rupertoverall]

Thanks Talmo - the logs are really helpful. And they tell me I had the training split set up wrongly. I guess I needed "split_by_inds": true [facepalm].
I fixed the config and now I can verify INFO:sleap.nn.training:Creating training split from explicit indices (n = 165). as Divya indicated.
Now I can run the tests (and post the results in discussion #1936).

[rupertoverall]

For reference, I'll summarise what I needed to answer my original question:

In the training config file, the 'labels' section is set up as below, with split_by_inds set to true and training_inds set to an array of indices (I guess to instances although in my case each video has only one (labelled) frame, so videos, frames and instances all have the same indices for me).

        ...
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": true,
            "training_inds": [2,3,19,24,25,31,34,41,45,53,56,60,...],
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },...

By sending the console output to file as @talmo suggested (in the form sleap-train model_config.json labels.pkg.slp > training_logs.txt) the restricted training set can be confirmed by searching for the line starting INFO:sleap.nn.training:Creating training split from explicit indices as @gitttt-1234 suggested.

Thanks both of you!