Crash while training bottom-up with ID: Labels.skeleton

[lisadiez]

Hello,

We're trying to train a bottom-up w/ ID model and ran into the following error:
ValueError: Labels.skeleton can only be used when there is only a single skeleton saved in the labels. Use Labels.skeletons instead.

We train directly from the CLI using a python file, which you can find (along with the video and slp file) in this folder:
https://drive.google.com/drive/folders/1e_itkWMF3AoJrycY05-uB3AIXA_WVSqC?usp=sharing

We normally use Sleap v.1.1.1 - but I also tried it in v.1.2.9 and get the same error.

Thank you for your help!

[catubc]

This error comes from our manual merge of 2 .slp files using this code which is based on a function made by Talmo.

def merge_slp_files(fname_slp1, fname_slp2):
    
    #
    fname_hybrid_video = fname_slp1[:-4]+"_merged.avi"
    
    #
    merged_video = sleap.load_video(fname_hybrid_video)
    print(merged_video)

    #
    fname_hybrid_slp = fname_hybrid_video[:-4] + ".slp"

    #
    slp_files = [fname_slp1,
                 fname_slp2
                ]
    
    n_frames = 0
    all_frames = []
    reference_tracks = {}

    first_labels = None
    for slp_file in slp_files:
        
        # Load saved labels.
        labels = sleap.load_file(slp_file)#, match_to=first_labels)
        if first_labels is None:
            first_labels = labels

        new_frames = []
        for i, lf in enumerate(labels):

            # Update reference to merged video.
            lf.video = merged_video

            # Update frame index to the frame number within the merged video.
            lf.frame_idx = n_frames + i

            # # Update the track reference to use the reference tracks to prevent duplication.
            for instance in lf:
                if instance.track is not None:
                    if instance.track.name in reference_tracks:
                        instance.track = reference_tracks[instance.track.name]
                    else:
                        reference_tracks[instance.track.name] = instance.track

            # Append the labeled frame to the list of frames we're keeping from these labels.
            new_frames.append(lf)

        all_frames.extend(new_frames)
        n_frames += len(new_frames)

    merged_labels = sleap.Labels(all_frames)

    # save stuff
    merged_labels.save(fname_hybrid_slp)

    print("DONE...")

# 
merge_slp_files(fname_human_slp,
                fname_idswitch_slp)

[roomrys]

@catubc,

It seems the code above might be adding two different instances of (the same) Skeleton leading to multiple Skeletons in the merged labels. I've modified the code below and am moving to the SLEAP source to correct where needed in the source's merging code.

Thanks,
Liezl

[roomrys]

@lisadiez, @catubc,

We have seen this problem occur when the labels file references more than one instance of Skeleton. It could be that the two projects use seemingly identical Skeletons, but are actually two different instances/references in the code. Hence, we need to change all references of instance.skeleton to just a single skeleton.

We have a piece of code for loading a new skeleton into the project which will delete old skeletons:

sleap/sleap/gui/commands.py

Lines 1722 to 1867 in c4861e3

	class OpenSkeleton(EditCommand):
	topics = [UpdateTopic.skeleton]
	@staticmethod
	def load_skeleton(filename: str):
	if filename.endswith(".json"):
	new_skeleton = Skeleton.load_json(filename)
	elif filename.endswith((".h5", ".hdf5")):
	sk_list = Skeleton.load_all_hdf5(filename)
	new_skeleton = sk_list[0]
	return new_skeleton
	@staticmethod
	def compare_skeletons(
	skeleton: Skeleton, new_skeleton: Skeleton
	) -> Tuple[List[str], List[str]]:
	delete_nodes = []
	add_nodes = []
	if skeleton.node_names != new_skeleton.node_names:
	# Compare skeletons
	base_nodes = skeleton.node_names
	new_nodes = new_skeleton.node_names
	delete_nodes = [node for node in base_nodes if node not in new_nodes]
	add_nodes = [node for node in new_nodes if node not in base_nodes]
	return delete_nodes, add_nodes
	@staticmethod
	def delete_extra_skeletons(labels: Labels):
	if len(labels.skeletons) > 1:
	skeletons_used = list(
	set(
	[
	inst.skeleton
	for lf in labels.labeled_frames
	for inst in lf.instances
	]
	)
	)
	try:
	assert len(skeletons_used) == 1
	except AssertionError:
	raise ValueError("Too many skeletons used in project.")
	labels.skeletons = skeletons_used
	@staticmethod
	def ask(context: CommandContext, params: dict) -> bool:
	filters = ["JSON skeleton (*.json)", "HDF5 skeleton (*.h5 *.hdf5)"]
	filename, selected_filter = FileDialog.open(
	context.app, dir=None, caption="Open skeleton...", filter=";;".join(filters)
	)
	if len(filename) == 0:
	return False
	okay = True
	if len(context.labels.skeletons) > 0:
	# Ask user permission to merge skeletons
	okay = False
	skeleton: Skeleton = context.labels.skeleton # Assumes single skeleton
	# Load new skeleton and compare
	new_skeleton = OpenSkeleton.load_skeleton(filename)
	(delete_nodes, add_nodes) = OpenSkeleton.compare_skeletons(
	skeleton, new_skeleton
	)
	if (len(delete_nodes) > 0) or (len(add_nodes) > 0):
	# Warn about mismatching skeletons
	title = "Replace Skeleton"
	message = (
	"<p><b>Warning:</b> Pre-existing skeleton found."
	"<p>The following nodes will be <b>deleted</b> from all instances:"
	f"<br><em>From base labels</em>: {','.join(delete_nodes)}<br></p>"
	"<p>The following nodes will be <b>added</b> to all instances:<br>"
	f"<em>From new labels</em>: {','.join(add_nodes)}</p>"
	"<p>Nodes can be deleted or merged from the skeleton editor after "
	"merging labels.</p>"
	)
	query = QueryDialog(title=title, message=message)
	query.exec_()
	# Give the okay to add/delete nodes
	okay = bool(query.result())
	params["delete_nodes"] = delete_nodes
	params["add_nodes"] = add_nodes
	params["filename"] = filename
	return okay
	@staticmethod
	def do_action(context: CommandContext, params: dict):
	# Load new skeleton
	filename = params["filename"]
	new_skeleton = OpenSkeleton.load_skeleton(filename)
	# Case 1: No skeleton exists in project
	if len(context.labels.skeletons) == 0:
	context.state["skeleton"] = new_skeleton
	context.labels.skeletons.append(context.state["skeleton"])
	return
	# Case 2: Skeleton(s) already exist(s) in project
	# Delete extra skeletons in project
	OpenSkeleton.delete_extra_skeletons(context.labels)
	skeleton = context.labels.skeleton # Assume single skeleton
	if "delete_nodes" in params.keys():
	# We already compared skeletons in ask() method
	delete_nodes: List[str] = params["delete_nodes"]
	add_nodes: List[str] = params["add_nodes"]
	else:
	# Otherwise, load new skeleton and compare
	(delete_nodes, add_nodes) = OpenSkeleton.compare_skeletons(
	skeleton, new_skeleton
	)
	# Delete pre-existing symmetry
	for src, dst in skeleton.symmetries:
	skeleton.delete_symmetry(src, dst)
	# Delete nodes from skeleton that are not in new skeleton
	for node in delete_nodes:
	skeleton.delete_node(node)
	# Add nodes that only exist in the new skeleton
	for node in add_nodes:
	skeleton.add_node(node)
	# Add edges
	skeleton.clear_edges()
	for src, dest in new_skeleton.edges:
	skeleton.add_edge(src.name, dest.name)
	# Add new symmetry
	for src, dst in new_skeleton.symmetries:
	skeleton.add_symmetry(src.name, dst.name)
	# Set state of context
	context.state["skeleton"] = skeleton

Can you add the function:

def compare_skeletons(
    skeleton: Skeleton, new_skeleton: Skeleton
) -> Tuple[List[str], List[str]]:

    delete_nodes = []
    add_nodes = []
    if skeleton.node_names != new_skeleton.node_names:
        # Compare skeletons
        base_nodes = skeleton.node_names
        new_nodes = new_skeleton.node_names
        delete_nodes = [node for node in base_nodes if node not in new_nodes]
        add_nodes = [node for node in new_nodes if node not in base_nodes]

    return delete_nodes, add_nodes

and try:

def merge_slp_files(fname_slp1, fname_slp2):
    
    #
    fname_hybrid_video = fname_slp1[:-4]+"_merged.avi"
    
    #
    merged_video = sleap.load_video(fname_hybrid_video)
    print(merged_video)

    #
    fname_hybrid_slp = fname_hybrid_video[:-4] + ".slp"

    # Load labels
    slp_files = [fname_slp1,
                 fname_slp2
                ]
    first_labels = sleap.load_file(slp_files[0])
    slp_labels = [first_labels, sleap.load_file(slp_files[1])]#match_to=first_labels)]
    
    # Compare skeletons
    assert len(slp_labels[0].skeletons) <= 1  # Ensure each project only has a single skeleton
    assert len(slp_labels[1].skeletons) <= 1
    delete_nodes, add_nodes = compare_skeleton(slp_labels[0].skeleton, slp_labels[1].skeleton)
    assert len(delete_nodes) == 0  # Ensure project skeletons are identical
    assert len(add_nodes) == 0
    the_skeleton = slp_labels[0].skeleton or slp_labels[1].skeleton
    
    n_frames = 0
    all_frames = []
    reference_tracks = {}

    for labels in slp_labels:

        new_frames = []
        for i, lf in enumerate(labels):

            # Update reference to merged video.
            lf.video = merged_video

            # Update frame index to the frame number within the merged video.
            lf.frame_idx = n_frames + i

            # # Update the track reference to use the reference tracks to prevent duplication.
            for instance in lf:
                if instance.track is not None:
                    if instance.track.name in reference_tracks:
                        instance.track = reference_tracks[instance.track.name]
                    else:
                        reference_tracks[instance.track.name] = instance.track

                # Set the instance to reference the correct instance of `Skeleton`
                instance.skeleton = the_skeleton

            # Append the labeled frame to the list of frames we're keeping from these labels.
            new_frames.append(lf)

        all_frames.extend(new_frames)
        n_frames += len(new_frames)

    merged_labels = sleap.Labels(all_frames)

    # save stuff
    merged_labels.save(fname_hybrid_slp)

    print("DONE...")

# 
merge_slp_files(fname_human_slp,
                fname_idswitch_slp)

Thanks,
Liezl

[lisadiez]

Hi Liezl,

Thank you so much for the quick response.

I tried this but something weird is happening... I can't see the predictions in the GUI:

merging.mov

There's 1001 human-labeled frames (these look ok) and 537 predicted frames (but I can't see the predictions).

Any idea why this is happening?

Here's the colab I'm using: https://colab.research.google.com/drive/10AK6srXisOjokOSMWAcx1A2bZyu-1SXj?usp=sharing
And here's a link to the files: https://drive.google.com/drive/folders/11cfs4bMeckfpxZWdMo8T3iCefAl48XKU?usp=sharing
(hybrid.slp and hybrid_cropped.slp are the files we're merging, and hybrid_merged.slp is what comes out).

Thank you again!

[talmo]

Hi all,

Just jumping in here with a quick suggestion -- on this line:

slp_labels = [first_labels, sleap.load_file(slp_files[1])]#match_to=first_labels)]

Do you remember why we commented the #match_to=first_labels? Have you tried with that bit included? i.e.:

slp_labels = [first_labels, sleap.load_file(slp_files[1], match_to=first_labels)]]

The match_to should handle merging skeletons -- it just doesn't currently deal with duplicate tracks (see #1083).

Give that a spin and if it doesn't work just keep troubleshooting as per @roomrys's suggestions.

---

Another thing to try is to just force the labels to have a single skeleton:

merged_labels = sleap.Labels(all_frames)
merged_labels.skeletons = [the_skeleton]

assert all([inst.skeleton == the_skeleton for inst in merged_labels.instances()])

It's a bit concerning that this is happening in the first place since it means there's more than one skeleton in the instances, but the assert statement should catch that before you save.