[Bug]: wrong magnitude

[AdriSP2]

Checklist

• Re-run analysis with topostats process --core 1.
• Describe the bug.
• Include the configuration file.
• Copy of the output.
• The exact command that failed. This is what you typed at the command line, including any options.
• TopoStats version, this is reported by topostats --version
• Operating System and Python Version

Describe the bug

When processing some Highspeed images, for some reason the magnitude of the height is about 10 times higher than the real one.
For instance, while manually analyzing some images you get a height of around 1.5-1.8nm the output is giving around 11-8nm being 8 if I increase the above threshold for grains as maximum as possible for the mask to fit.
Could it be somehow that the channel is being read wrong? Using another set of images it worked fine

Copy of the output

[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [318] [230523-01-0048_nigericina_61-1017_1015] : Coloured regions
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [416] [230523-01-0048_nigericina_61-1017_1015] : Extracted bounding boxes (above)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [171] [230523-01-0048_nigericina_61-1017_1015] : Grains found for direction above : 2
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [186] [230523-01-0048_nigericina_61-1017_1015] : Plotting Grain Finding Images
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [188] [230523-01-0048_nigericina_61-1017_1015] : Plotting above Grain Finding Images
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting mask_grains image
[Wed, 02 Oct 2024 18:08:55] [WARNING ] [topostats] [io.py] [1110] Item 'grain_trace_data' is None. Skipping.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [io.py] [1138] Cannot save key 'grain_trace_data' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting labelled_regions_01 image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting tidied_border image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting removed_noise image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting removed_small_objects image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting labelled_regions_02 image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1015] : Plotting coloured_regions image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [plottingfuncs.py] [307] [230523-01-0048_nigericina_61-1017_1013_above_masked] : Image saved to : output\processed\230523-01-0048_nigericina_61-1017_1013_above_masked.png | DPI: 100
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [280] [230523-01-0048_nigericina_61-1017_1013] : *** Grain Statistics ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1013] : Processing grain: 0
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [plottingfuncs.py] [307] [230523-01-0048_nigericina_61-1017_1014_above_masked] : Image saved to : output\processed\230523-01-0048_nigericina_61-1017_1014_above_masked.png | DPI: 100
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [280] [230523-01-0048_nigericina_61-1017_1014] : *** Grain Statistics ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1014] : Processing grain: 0
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [plottingfuncs.py] [307] [230523-01-0048_nigericina_61-1017_1016] : Image saved to : output\processed\230523-01-0048_nigericina_61-1017_1016.png | DPI: 100
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [162] [230523-01-0048_nigericina_61-1017_1016] : *** Grain Finding ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [358] [230523-01-0048_nigericina_61-1017_1016] : Thresholding method (grains) : absolute
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [367] [230523-01-0048_nigericina_61-1017_1016] : Finding above grains, threshold: (5)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [utils.py] [165] [230523-01-0048_nigericina_61-1017_1016] : Masking (above) Threshold: 5
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [169] [230523-01-0048_nigericina_61-1017_1016] : Labelling Regions
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [147] [230523-01-0048_nigericina_61-1017_1016] : Tidying borders
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [386] [230523-01-0048_nigericina_61-1017_1016] : Removing noise (above)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [287] [230523-01-0048_nigericina_61-1017_1016] : Area thresholding grains | Thresholds: L: 1.12,U: 5625.00 px^2, L: 0.50, U: 2500.00 nm^2.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [392] [230523-01-0048_nigericina_61-1017_1016] : Removing small / large grains (above)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [287] [230523-01-0048_nigericina_61-1017_1016] : Area thresholding grains | Thresholds: L: 2.25,U: 6750.00 px^2, L: 1.00, U: 3000.00 nm^2.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [169] [230523-01-0048_nigericina_61-1017_1016] : Labelling Regions
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [411] [230523-01-0048_nigericina_61-1017_1016] : Region properties calculated (above)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [318] [230523-01-0048_nigericina_61-1017_1016] : Coloured regions
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grains.py] [416] [230523-01-0048_nigericina_61-1017_1016] : Extracted bounding boxes (above)
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [171] [230523-01-0048_nigericina_61-1017_1016] : Grains found for direction above : 1
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [186] [230523-01-0048_nigericina_61-1017_1016] : Plotting Grain Finding Images
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [188] [230523-01-0048_nigericina_61-1017_1016] : Plotting above Grain Finding Images
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting mask_grains image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting labelled_regions_01 image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting tidied_border image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting removed_noise image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting removed_small_objects image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting labelled_regions_02 image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [194] [230523-01-0048_nigericina_61-1017_1016] : Plotting coloured_regions image
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1013] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [480] [230523-01-0048_nigericina_61-1017_1013] Calculation of DNA Tracing disabled, returning grainstats data frame.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [647] [230523-01-0048_nigericina_61-1017_1013] : *** Image Statistics ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [io.py] [1189] [230523-01-0048_nigericina_61-1017_1013] : Saving image to .topostats file
[Wed, 02 Oct 2024 18:08:55] [WARNING ] [topostats] [io.py] [1110] Item 'grain_trace_data' is None. Skipping.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [io.py] [1138] Cannot save key 'grain_trace_data' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1014] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1014] : Processing grain: 1
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [plottingfuncs.py] [307] [230523-01-0048_nigericina_61-1017_1015_above_masked] : Image saved to : output\processed\230523-01-0048_nigericina_61-1017_1015_above_masked.png | DPI: 100
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [280] [230523-01-0048_nigericina_61-1017_1015] : *** Grain Statistics ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1015] : Processing grain: 0
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1014] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [480] [230523-01-0048_nigericina_61-1017_1014] Calculation of DNA Tracing disabled, returning grainstats data frame.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [processing.py] [647] [230523-01-0048_nigericina_61-1017_1014] : *** Image Statistics ***
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [io.py] [1189] [230523-01-0048_nigericina_61-1017_1014] : Saving image to .topostats file
[Wed, 02 Oct 2024 18:08:55] [WARNING ] [topostats] [io.py] [1110] Item 'grain_trace_data' is None. Skipping.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [io.py] [1138] Cannot save key 'grain_trace_data' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1015] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:55] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1015] : Processing grain: 1
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [plottingfuncs.py] [307] [230523-01-0048_nigericina_61-1017_1016_above_masked] : Image saved to : output\processed\230523-01-0048_nigericina_61-1017_1016_above_masked.png | DPI: 100
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [processing.py] [280] [230523-01-0048_nigericina_61-1017_1016] : *** Grain Statistics ***
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [grainstats.py] [220] [230523-01-0048_nigericina_61-1017_1016] : Processing grain: 0
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1015] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [processing.py] [480] [230523-01-0048_nigericina_61-1017_1015] Calculation of DNA Tracing disabled, returning grainstats data frame.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [processing.py] [647] [230523-01-0048_nigericina_61-1017_1015] : *** Image Statistics ***
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [io.py] [1189] [230523-01-0048_nigericina_61-1017_1015] : Saving image to .topostats file
[Wed, 02 Oct 2024 18:08:56] [WARNING ] [topostats] [io.py] [1110] Item 'grain_trace_data' is None. Skipping.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [io.py] [1138] Cannot save key 'grain_trace_data' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [grainstats.py] [311] [230523-01-0048_nigericina_61-1017_1016] : Height profiles extracted.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [processing.py] [480] [230523-01-0048_nigericina_61-1017_1016] Calculation of DNA Tracing disabled, returning grainstats data frame.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [processing.py] [647] [230523-01-0048_nigericina_61-1017_1016] : *** Image Statistics ***
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [io.py] [1189] [230523-01-0048_nigericina_61-1017_1016] : Saving image to .topostats file
[Wed, 02 Oct 2024 18:08:56] [WARNING ] [topostats] [io.py] [1110] Item 'grain_trace_data' is None. Skipping.
[Wed, 02 Oct 2024 18:08:56] [INFO    ] [topostats] [io.py] [1138] Cannot save key 'grain_trace_data' to HDF5. Item type: <class 'NoneType'>. Skipping. One of data, shape or dtype must be specified
![230523-01-0048_nigericina_61-1017_0_above_masked](https://github.com/user-attachments/assets/a12df6e1-02c1-4ac0-8e2b-98cadc10f57e)

Include the configuration file

# Configuration from TopoStats run completed : 2024-10-02 18:09:00
# For more information on configuration and how to use it:
# https://afm-spm.github.io/TopoStats/main/configuration.html
base_dir: C:\Users\Adrià\Desktop\TopoStats
cores: 5
dnatracing: {min_skeleton_size: 10, pad_width: 1, run: false, skeletonisation_method: topostats,
  spline_circular_smoothing: 0.0, spline_linear_smoothing: 5.0, spline_step_size: 7e-09}
file_ext: .asd
filter:
  gaussian_mode: nearest
  gaussian_size: 1.15
  otsu_threshold_multiplier: 1.0
  remove_scars: {max_scar_width: 4, min_scar_length: 16, removal_iterations: 2, run: false,
    threshold_high: 0.666, threshold_low: 0.25}
  row_alignment_quantile: 0.5
  run: true
  threshold_absolute: {above: 1.0, below: -1.0}
  threshold_method: std_dev
  threshold_std_dev: {above: 1.2, below: 10.0}
grains:
  absolute_area_threshold:
    above: [1, 3000]
    below: [null, null]
  direction: above
  otsu_threshold_multiplier: 1.0
  remove_edge_intersecting_grains: true
  run: true
  smallest_grain_size_nm2: 0.5
  threshold_absolute: {above: 5, below: 15}
  threshold_method: absolute
  threshold_std_dev: {above: 1.0, below: 10.0}
grainstats: {cropped_size: 40.0, edge_detection_method: binary_erosion, extract_height_profile: true,
  run: true}
loading: {channel: TP}
log_level: info
output_dir: output
plotting:
  axes: true
  cmap: null
  colorbar: true
  histogram_log_axis: false
  image_set: core
  mask_cmap: blu
  num_ticks: [null, null]
  pixel_interpolation: null
  run: true
  savefig_dpi: null
  savefig_format: null
  style: topostats.mplstyle
  zrange: [null, null]
summary_stats: {config: null, run: true}

To Reproduce

https://drive.google.com/drive/folders/1qzD15wpRilwI6A57jkYdb2s6XaGluK8j?usp=sharing

TopoStats Version

2.2.1

Python Version

3.10.13

Operating System

Windows

Python Packages

AFMReader==0.0.1
archspec @ file:///croot/archspec_1709217642129/work
asttokens==2.4.1
boltons @ file:///C:/b/abs_707eo7c09t/croot/boltons_1677628723117/work
Brotli @ file:///C:/Windows/Temp/abs_63l7912z0e/croots/recipe/brotli-split_1659616056886/work
certifi @ file:///C:/b/abs_35d7n66oz9/croot/certifi_1707229248467/work/certifi
cffi @ file:///C:/b/abs_924gv1kxzj/croot/cffi_1700254355075/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
cheap-repr==0.5.1
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
conda @ file:///C:/b/abs_1e6dlkntna/croot/conda_1710772093015/work
conda-content-trust @ file:///C:/b/abs_e3bcpyv7sw/croot/conda-content-trust_1693490654398/work
conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1702997573971/work/src
conda-package-handling @ file:///C:/b/abs_b9wp3lr1gn/croot/conda-package-handling_1691008700066/work
conda_package_streaming @ file:///C:/b/abs_6c28n38aaj/croot/conda-package-streaming_1690988019210/work
contourpy==1.2.1
cryptography @ file:///C:/b/abs_e8cnom_zw_/croot/cryptography_1702071486468/work
cycler==0.12.1
decorator==5.1.1
distro @ file:///C:/b/abs_a3uni_yez3/croot/distro_1701455052240/work
exceptiongroup==1.2.2
executing==2.0.1
fonttools==4.53.1
h5py==3.11.0
idna @ file:///C:/b/abs_bdhbebrioa/croot/idna_1666125572046/work
igor2==0.5.8
imageio==2.34.2
ipython==8.26.0
jedi==0.19.1
joblib==1.4.2
jsonpatch @ file:///tmp/build/80754af9/jsonpatch_1615747632069/work
jsonpointer==2.1
kiwisolver==1.4.5
lazy_loader==0.4
libmambapy @ file:///D:/bld/mamba-split_1703178789789/work/libmambapy
loguru==0.7.2
matplotlib==3.9.1
matplotlib-inline==0.1.7
menuinst @ file:///C:/b/abs_e8p75b4m9q/croot/menuinst_1702390332729/work
networkx==3.3
numpy==1.26.4
packaging @ file:///C:/b/abs_28t5mcoltc/croot/packaging_1693575224052/work
pandas==2.2.2
parso==0.8.4
pillow==10.4.0
platformdirs @ file:///C:/b/abs_b6z_yqw_ii/croot/platformdirs_1692205479426/work
pluggy @ file:///C:/ci/pluggy_1648042746254/work
prompt_toolkit==3.0.47
psutil==5.9.8
pure-eval==0.2.2
pycosat @ file:///C:/b/abs_31zywn1be3/croot/pycosat_1696537126223/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pyfiglet==1.0.2
Pygments==2.18.0
pyOpenSSL @ file:///C:/b/abs_08f38zyck4/croot/pyopenssl_1690225407403/work
pyparsing==3.1.2
PySocks @ file:///C:/ci_310/pysocks_1642089375450/work
pyspm==0.6.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests @ file:///C:/b/abs_316c2inijk/croot/requests_1690400295842/work
ruamel.yaml @ file:///C:/b/abs_30ee5qbthd/croot/ruamel.yaml_1666304562000/work
ruamel.yaml.clib @ file:///C:/b/abs_aarblxbilo/croot/ruamel.yaml.clib_1666302270884/work
schema==0.7.7
scikit-image==0.24.0
scikit-learn==1.5.1
scipy==1.14.0
seaborn==0.13.2
six==1.16.0
snoop==0.4.3
stack-data==0.6.3
threadpoolctl==3.5.0
tifffile==2024.7.2
topostats==2.2.1
tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work
traitlets==5.14.3
truststore @ file:///C:/b/abs_55z7b3r045/croot/truststore_1695245455435/work
typing_extensions==4.12.2
tzdata==2024.1
urllib3 @ file:///C:/b/abs_9cmlsrm3ys/croot/urllib3_1698257595508/work
wcwidth==0.2.13
win-inet-pton @ file:///C:/ci_310/win_inet_pton_1642658466512/work
win32-setctime==1.1.0
zstandard==0.19.0

[ns-rse]

Thanks for the 🐞 report @AdriSP2

The first thing to check is whether this problem persists in the most recent version of TopoStats which is v2.2.1.

Ideally this should be done in a clean virtual environment with the latest versions of all packages installed. This can be done after creating and activating your virtual environment using...

pip install --no-cache-dir topostats==2.2.1

I notice also that you've indicated that you are running GNU/Linux but some of the package paths that are listed by pip freeze point to files that look like they are on a Windows system as they include C:/ are you running Windows Subsystem for Linux perhaps?

[AdriSP2]

Hi, I got an issue opening the tiket and it looks like the dropdown menues went to the default position
Topostats version 2.2.1
Python 3.10.13
OS: Windows

let me see if i can edit it from the top so it does not create confusion