Tests for Skeletonisation and Pruning

[ns-rse]

The maxgamill-sheffield/800-better-tracing needs tests to ensure all the skeletonisation and pruning steps run as expected.

Need to ensure that this branch is up-to-date with main (rebase) and tests will be developed on a branch from maxgamill-sheffield/800-better-tracing and periodically merged in as tests are completed.

[ns-rse]

Need to keep track of what needs doing as aside from tests for new code a bunch of existing tests also fail.

Failing Tests

Currently the following tests fail, note that some have already been skipped as pruning needs to be correct first.

=================================================================================================================================================== short test summary info ===================================================================================================================================================
SKIPPED [1] tests/test_utils.py:674: awaiting test development
SKIPPED [4] tests/tracing/test_dnatracing_single_grain.py:254: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:278: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:306: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:347: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:376: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:157: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:178: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:209: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_dnatracing_single_grain.py:230: Need to correctly prune arrays first.
SKIPPED [2] tests/tracing/test_pruning.py: two branches at the end does not have 'nibs' removed by the rm_nibs() function, instead a T-shaped junction remains. This is only removed by calling getSkeleton(method='zhang').get_skeleton() which is done under the prune_all_skeletons() method but this method is is to be remved since looping over all skeletons is outside of the scope/remit of the code and handled in process_scan().
SKIPPED [1] tests/tracing/test_pruning.py: unconditional skip
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 non-binary array; point as list] - Array is not binary.
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 1's; point on left edge] - Point on left edge of image.
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 1's; point on top edge] - Point on top edge of image.
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 1's; point on right edge] - Point on right edge of image.
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 1's; point on bottom edge] - Point on bottom edge of image.
XFAIL tests/tracing/test_pruning.py::test_local_area_sum[3x3 1's; point on corner] - Point on corner of image.
ERROR tests/test_plotting.py::test_set_label[end_to_end_distance-End to End Distance] - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_plot_violin - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_df_from_csv - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_set_label[grain_bound_len-Circumference] - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_plot_violin_multiple_directories - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_toposum_class - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_set_label[grain_curvature1-Smaller Curvature] - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_outfile - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_set_label_keyerror - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_set_label[contour_length-Contour Length] - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_plot_kde - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_plot_kde_multiple_directories - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/test_plotting.py::test_plot_hist - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain0-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain8-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image0-1.0-True-0.0-expected_spline_image0] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain9-True] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/test_plotting.py::test_plot_hist_multiple_directories - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image1-1.0-True-10.0-expected_spline_image1] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain10-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image2-10.0-True-0.0-expected_spline_image2] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain11-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image3-10.0-True-5.0-expected_spline_image3] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain12-True] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image4-5.0-False-5.0-expected_spline_image4] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/test_plotting.py::test_plot_hist_kde - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain13-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image5-4.0-False-5.0-expected_spline_image5] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain14-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image6-5.0-False-0.0-expected_spline_image6] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_get_splined_traces[trace_image7-5.0-False-5.0-expected_spline_image7] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain1-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain2-True] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/test_plotting.py::test_plot_hist_kde_multiple_directories - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain3-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain4-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain5-True] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain6-True] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
ERROR tests/tracing/test_dnatracing_methods.py::test_linear_or_circular[grain7-False] - TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain'
FAILED tests/test_plotting.py::test_melt_data - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
FAILED tests/test_grainstats_minicircle.py::test_grainstats_regression
FAILED tests/test_plotting.py::test_toposum - KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['grain_number']"
FAILED tests/test_grainstats_minicircle.py::test_cropped_image - assert (1, 1) == (21, 21)
FAILED tests/tracing/test_dnatracing_multigrain.py::test_prep_arrays[0-target_image0-target_mask0] - ValueError: too many values to unpack (expected 2)
FAILED tests/tracing/test_dnatracing_multigrain.py::test_prep_arrays[1-target_image1-target_mask1] - ValueError: too many values to unpack (expected 2)
FAILED tests/tracing/test_dnatracing_multigrain.py::test_image_trace_unequal_arrays - TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
FAILED tests/tracing/test_dnatracing_multigrain.py::test_trace_image[multigrain_topostats-topostats-1-statistics0-ordered_trace_start0-ordered_trace_end0] - TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
FAILED tests/test_plottingfuncs.py::test_plot_and_save_colorbar_and_axes - Failed: Error: Image files did not match.
FAILED tests/tracing/test_dnatracing_multigrain.py::test_trace_image[multigrain_zhang-zhang-1-statistics1-ordered_trace_start1-ordered_trace_end1] - TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
FAILED tests/test_plottingfuncs.py::test_mask_dilation - Failed: Error: Image files did not match.
FAILED tests/tracing/test_dnatracing_multigrain.py::test_trace_image[multigrain_lee-lee-1-statistics2-ordered_trace_start2-ordered_trace_end2] - TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
FAILED tests/tracing/test_dnatracing_multigrain.py::test_trace_image[multigrain_thin-thin-1-statistics3-ordered_trace_start3-ordered_trace_end3] - TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
FAILED tests/tracing/test_nodestats.py::test_connect_extended_nodes_nearest[theta_grain] - NameError: name 'do_sets_touch' is not defined
FAILED tests/tracing/test_nodestats.py::test_connect_extended_nodes_nearest[figure_8] - assert False
FAILED tests/test_plottingfuncs.py::test_plot_and_save_no_axes - Failed: Error: Image files did not match.
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[circular molecule, skeletonise topostats] - assert 183 == 154
FAILED tests/test_plottingfuncs.py::test_plot_and_save_no_axes_no_colorbar - Failed: Error: Image files did not match.
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[linear molecule, skeletonise zhang] - assert 166 == 122
FAILED tests/test_plottingfuncs.py::test_plot_and_save_no_colorbar - Failed: Error: Image files did not match.
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[circular molecule, skeletonise zhang] - assert 177 == 149
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[linear molecule, skeletonise topostats] - assert 177 == 91
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[linear molecule, skeletonise lee] - assert 197 == 130
FAILED tests/test_processing.py::test_run_dnatracing - TypeError: run_dnatracing() got an unexpected keyword argument 'grain_out_path'
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[circular molecule, skeletonise lee] - assert 185 == 151
FAILED tests/test_processing.py::test_process_scan_below - IndexError: list index out of range
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[linear molecule, skeletonise thin] - assert 142 == 118
FAILED tests/tracing/test_dnatracing_single_grain.py::test_get_disordered_trace[circular molecule, skeletonise thin] - assert 150 == 175
FAILED tests/test_plottingfuncs.py::test_plot_and_save_non_square_bounding_box - Failed: Error: Image files did not match.
FAILED tests/test_processing.py::test_process_scan_above - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_plottingfuncs.py::test_mask_cmap - Failed: Error: Image files did not match.
FAILED tests/test_processing.py::test_process_scan_both - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_processing.py::test_process_stages[Filtering, Grain, GrainStats and DNA Tracing enabled] - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_plottingfuncs.py::test_high_dpi - Failed: Error: Image files did not match.
FAILED tests/test_processing.py::test_save_cropped_grains[core-False] - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_processing.py::test_process_scan_no_grains - KeyError: "'green_black' is not a known colormap name"
FAILED tests/test_processing.py::test_process_scan_align_grainstats_dnatracing - IndexError: list index out of range
FAILED tests/test_run_topostats.py::test_run_topostats_process_all - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_processing.py::test_save_cropped_grains[all-True] - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_run_topostats.py::test_run_topostats_process_debug - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_processing.py::test_save_format[png] - NameError: name 'do_sets_touch' is not defined
FAILED tests/test_processing.py::test_save_format[tif] - NameError: name 'do_sets_touch' is not defined
========================================================================================================================= 42 failed, 558 passed, 24 skipped, 6 xfailed, 39 errors in 67.49s (0:01:07) =========================================================================================================================

Tasks to complete

• TypeError: dnaTrace.__init__() got an unexpected keyword argument 'grain' Finish of rationalising the use of image, mask and grain. These are used inconsistently at present, sometimes grain refers to the numpy array with heights, sometimes to the binary mask related to this. To which end completely removing reference to grain and favoring image (for arrays that have heights) and mask for binary arrays is less ambiguous.
• grain_number missing from dataframe (when running plotting tests).
• KeyError: "'green_black' is not a known colormap name"
• NameError: name 'do_sets_touch' is not defined
• TypeError: run_dnatracing() got an unexpected keyword argument 'grain_out_path'
• TypeError: trace_image() got an unexpected keyword argument 'skeletonisation_method'
• ValueError: too many values to unpack (expected 2)
• Why have the array sizes of disordered traces changed (this prevents a bunch of other tests from being run as we incrementally add on additional analysis steps so if this first step fails everything after will fail, which is a poorly designed bunch of unittests really so poor implementation on my behalf).

mol_is_circular v mol_is_circulars

In previous refactoring the processing of DNA tracing shifted from a whole image with loops in every single function to processing a single "grain" at a time (thus removing the loops).

Currently this has been reverted and the attribute mol_is_circular is now pluralised to mol_is_circulars because it is a list of booleans which apply to the disentangled molecules.

@MaxGamill-Sheffield and I have spoken about moving NodeStats processing out of dnaTrace as the end result of NodeStats are individual molecules each of which can then be processed by dnaTrace in the singular approach as was the case before this work was undertaken. If NodeStats only finds one single molecule it too will be processed.

I'm hesitant to make changes to this at the moment as the extraction of NodeStats being called needs to be done first, but this means a the tests/tracing/test_dnatracing_methods.py::linear_or_circular() test can not be corrected at present.

Lost of relevant tests

test_dnatracing_methods.py

Passing	Test	Error Message
	test_linear_or_circular
✅	test_crop_array
	test_get_splined_traces	AttributeError: type object 'genTracingFuncs' has no attribute 'countNeighbours'
✅	test_round_splined_traces
✅	test_remove_duplicate_consecutive_tuples

test_dnatracing_multigrain.py

Passing	Test	Error Message
	test_prep_arrays
	test_image_trace_unequal_arrays
	test_trace_mask
	test_trace_image

test_dnatracing_single_grain.py

Passing	Test	Error Message
✅	test_gaussian_filter
	test_get_disordered_trace	Array lengths do not match, i.e. a different trace is returned. These need to match otherwise most of the other tests of dnaTrace methods will fail.
	test_linear_or_circular
	test_get_ordered_traces
	test_get_ordered_trace_heights
	test_ordered_get_trace_cumulative_distances
	test_coord_dist
	test_get_fitted_traces
	test_get_splined_traces
	test_measure_contour_length
	test_measure_end_to_end_distance
✅	test_crop_array
✅	test_pad_bounding_box
✅	test_grain_anchor
	test_trace_grain

[ns-rse]

Changes in get_disordered_trace()

This lies at the heart of why many tests within tests/tracing/test_dnatracing_single_grain.py fail.

After a sense of deja vu I've realised I've started investigating this already and have comments in #848 that are relevant, but that clearly didn't fix everything.

• Pruning not as expected
• max_branch_length changing (part 1)
• different masks being passed in
• .... and subsequent comments.

Summary

The gaussian_filter() method was replaced with smooth_mask() but the disordered traces returned by get_disordered_trace() are still different even though we have updated the test to use the later.

Currently things look like this and it is perhaps most disconcerting that there are quite big changes in the disordered traces from the topostats method. Initially I thought this might be related to using heights in the skeletonisation but realised that that is a step subsequent to this so that can not be the explanation.

Skeletonisation	main	maxgamill-sheffield/800-better-tracing
lee
thin
topostats
zhang
lee
thin
topostats
zhang

[ns-rse]

Check whether gaussian_filter or binary_dilation is being used prior to skeletonisation as this could be one of the underlying causes of the differences.