Add parsing for JEOL #39
Comments
mkuehbach
added a commit
that referenced
this issue
Aug 26, 2024
* Start working on TIFF parsing for JEOL * Implementing functionality * Image parsing for JEOL working * Filename fix * Add a few metadata items, working * Parser for TESCAN TIFF, fixes #26 (#45) * Initial implementation of working TESCAN parser * Parser for Zeiss TIFF parser, fixes #31 (#46) * Initial version of the implementation for the Zeiss parser, activation of TESCAN metadata, testing remains * Fixed bug that more than one ureg was used across different parsers, fixed bug that mapping to a target quantity was ignored for case_three_str * Prepare nionswift * Parser for Nion Co. nd binary and HDF5 files to NeXus, fixes #27 (#53) * Implementation generic plotting and initial metadata mapping table * Added interfacing to data and mdata processing functions, initial plots working, need to update pynxtools upstream, need to test metadata * Suggestions from Lukas * Minor linting * Tested plot generation functional including default plots for multi-signal example from B. Haas and pytchography dataset from Anton, Sherjeel et al. * Initial test of writing metadata, several already written but three issues i) some metadata not in expected places due to unclear strategy when certain information end up where in nion metadata, ii) some concepts like corrector_cs not resolved and thus not autoannotated and thus will likely create problems in nomad, iii) only hotfixing currently the case that information written to template is not checked for whether it exists already and thus also logical situations like I know already what this detector is can not be handled automatically but only by overwriting * Further fixes for Nion metadata parsing * Reconfigured location of aberrations * Fixed variable typo * Fixing several so far incorrectly resolved metadata concepts * Fixed except for one dwell_time for event 1,3,4,12 for multisignal B. Haas datasets all issues * Parser for Gatan DigitalMicrograph DM3/DM4, fixes #12 (#54) * Starting code for DigitalMicrograph * Test and lint with several python versions, fixes #41 * Fix whitespace * Working initial version of generic plotting for Gatan with default plot resolving, next steps, test all cases, activate metadata * Working initial working of a Gatan parser, metadata are mapped but substantial discussion with the community is required * Parser for Hitachi TIFF, fixes #44 (#55) * Reorganized locations of tfs concepts * Working Hitachi parser * Enabling HDR sidecar metadata file for TESCAN (#56) * Preparing for TESCAN sidecar file, next steps: i) complete this, then cleaning, and final tests * Fixed incorrect routing Zeiss, mypy issue still remains * Fixed mypy, possibly a mypy bug, just switching order changed it? * Rm unused variables (#57) * First round of unnecessary variables removed, first round of unnecessary complicated variable names simplified * Equalized all call lines for individual parsers, removed deprecated term subparser * Further reduction of code bloat from unnecessary small functions but called with the similar configuration * Refactoring, suggestions from lukaspiel * Preparing docs update (#59) * Preparing docs update * Tested automated mapping of as a suggestion how to implement #321 of www.github.com/FAIRmat-NFDI/pynxtools * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> * Further testing and modernization to match NXem v3, fixes #28 (#60) * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]>
mkuehbach
added a commit
that referenced
this issue
Aug 26, 2024
* Initial implementation point electronic DISS parser * Fixed mapping_functors_pint mypy issue but here a more elegant and performant solution should be found! * Added case_six * Switching from >=0.2.1 to unstable EM/microstructure v3 definitions feature tracking branch * Updating dev-requirements * Updated definitions * Fix #28 for conventions * Various bugfixes, information flow logic related * Fixed naming convention mismatch between nxdefs-em v2 and v3 because of which default plot locations were not found, fixed bug that pint.units were written as python objects rather than strings, first working version of minimal point electronic DISS EBIC parser * Parser for JEOL TIFF, fixes #39 (#43) * Start working on TIFF parsing for JEOL * Implementing functionality * Image parsing for JEOL working * Filename fix * Add a few metadata items, working * Parser for TESCAN TIFF, fixes #26 (#45) * Initial implementation of working TESCAN parser * Parser for Zeiss TIFF parser, fixes #31 (#46) * Initial version of the implementation for the Zeiss parser, activation of TESCAN metadata, testing remains * Fixed bug that more than one ureg was used across different parsers, fixed bug that mapping to a target quantity was ignored for case_three_str * Prepare nionswift * Parser for Nion Co. nd binary and HDF5 files to NeXus, fixes #27 (#53) * Implementation generic plotting and initial metadata mapping table * Added interfacing to data and mdata processing functions, initial plots working, need to update pynxtools upstream, need to test metadata * Suggestions from Lukas * Minor linting * Tested plot generation functional including default plots for multi-signal example from B. Haas and pytchography dataset from Anton, Sherjeel et al. * Initial test of writing metadata, several already written but three issues i) some metadata not in expected places due to unclear strategy when certain information end up where in nion metadata, ii) some concepts like corrector_cs not resolved and thus not autoannotated and thus will likely create problems in nomad, iii) only hotfixing currently the case that information written to template is not checked for whether it exists already and thus also logical situations like I know already what this detector is can not be handled automatically but only by overwriting * Further fixes for Nion metadata parsing * Reconfigured location of aberrations * Fixed variable typo * Fixing several so far incorrectly resolved metadata concepts * Fixed except for one dwell_time for event 1,3,4,12 for multisignal B. Haas datasets all issues * Parser for Gatan DigitalMicrograph DM3/DM4, fixes #12 (#54) * Starting code for DigitalMicrograph * Test and lint with several python versions, fixes #41 * Fix whitespace * Working initial version of generic plotting for Gatan with default plot resolving, next steps, test all cases, activate metadata * Working initial working of a Gatan parser, metadata are mapped but substantial discussion with the community is required * Parser for Hitachi TIFF, fixes #44 (#55) * Reorganized locations of tfs concepts * Working Hitachi parser * Enabling HDR sidecar metadata file for TESCAN (#56) * Preparing for TESCAN sidecar file, next steps: i) complete this, then cleaning, and final tests * Fixed incorrect routing Zeiss, mypy issue still remains * Fixed mypy, possibly a mypy bug, just switching order changed it? * Rm unused variables (#57) * First round of unnecessary variables removed, first round of unnecessary complicated variable names simplified * Equalized all call lines for individual parsers, removed deprecated term subparser * Further reduction of code bloat from unnecessary small functions but called with the similar configuration * Refactoring, suggestions from lukaspiel * Preparing docs update (#59) * Preparing docs update * Tested automated mapping of as a suggestion how to implement #321 of www.github.com/FAIRmat-NFDI/pynxtools * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> * Further testing and modernization to match NXem v3, fixes #28 (#60) * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]>
mkuehbach
added a commit
that referenced
this issue
Sep 11, 2024
* Added description of the new concept mapping * More documentation * Editing * Cleaned concept for using pint and case analysis * Formatting * Added list to enable concatenations of concept values like x, y, z into an array * One more example * Polishing * Initial implementation pint-based conversions and cases * Bugfixes * Further contractions of the notation * Parser for point electronic DISS TIFF, fixes #13 (#40) * Initial implementation point electronic DISS parser * Fixed mapping_functors_pint mypy issue but here a more elegant and performant solution should be found! * Added case_six * Switching from >=0.2.1 to unstable EM/microstructure v3 definitions feature tracking branch * Updating dev-requirements * Updated definitions * Fix #28 for conventions * Various bugfixes, information flow logic related * Fixed naming convention mismatch between nxdefs-em v2 and v3 because of which default plot locations were not found, fixed bug that pint.units were written as python objects rather than strings, first working version of minimal point electronic DISS EBIC parser * Parser for JEOL TIFF, fixes #39 (#43) * Start working on TIFF parsing for JEOL * Implementing functionality * Image parsing for JEOL working * Filename fix * Add a few metadata items, working * Parser for TESCAN TIFF, fixes #26 (#45) * Initial implementation of working TESCAN parser * Parser for Zeiss TIFF parser, fixes #31 (#46) * Initial version of the implementation for the Zeiss parser, activation of TESCAN metadata, testing remains * Fixed bug that more than one ureg was used across different parsers, fixed bug that mapping to a target quantity was ignored for case_three_str * Prepare nionswift * Parser for Nion Co. nd binary and HDF5 files to NeXus, fixes #27 (#53) * Implementation generic plotting and initial metadata mapping table * Added interfacing to data and mdata processing functions, initial plots working, need to update pynxtools upstream, need to test metadata * Suggestions from Lukas * Minor linting * Tested plot generation functional including default plots for multi-signal example from B. Haas and pytchography dataset from Anton, Sherjeel et al. * Initial test of writing metadata, several already written but three issues i) some metadata not in expected places due to unclear strategy when certain information end up where in nion metadata, ii) some concepts like corrector_cs not resolved and thus not autoannotated and thus will likely create problems in nomad, iii) only hotfixing currently the case that information written to template is not checked for whether it exists already and thus also logical situations like I know already what this detector is can not be handled automatically but only by overwriting * Further fixes for Nion metadata parsing * Reconfigured location of aberrations * Fixed variable typo * Fixing several so far incorrectly resolved metadata concepts * Fixed except for one dwell_time for event 1,3,4,12 for multisignal B. Haas datasets all issues * Parser for Gatan DigitalMicrograph DM3/DM4, fixes #12 (#54) * Starting code for DigitalMicrograph * Test and lint with several python versions, fixes #41 * Fix whitespace * Working initial version of generic plotting for Gatan with default plot resolving, next steps, test all cases, activate metadata * Working initial working of a Gatan parser, metadata are mapped but substantial discussion with the community is required * Parser for Hitachi TIFF, fixes #44 (#55) * Reorganized locations of tfs concepts * Working Hitachi parser * Enabling HDR sidecar metadata file for TESCAN (#56) * Preparing for TESCAN sidecar file, next steps: i) complete this, then cleaning, and final tests * Fixed incorrect routing Zeiss, mypy issue still remains * Fixed mypy, possibly a mypy bug, just switching order changed it? * Rm unused variables (#57) * First round of unnecessary variables removed, first round of unnecessary complicated variable names simplified * Equalized all call lines for individual parsers, removed deprecated term subparser * Further reduction of code bloat from unnecessary small functions but called with the similar configuration * Refactoring, suggestions from lukaspiel * Preparing docs update (#59) * Preparing docs update * Tested automated mapping of as a suggestion how to implement #321 of www.github.com/FAIRmat-NFDI/pynxtools * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> * Further testing and modernization to match NXem v3, fixes #28 (#60) * conventions parser using v3 and tested * eln and oasis parser using v3 and tested * Minor formatting and adding of conventions and eln/oasis cfg parser in docs * axon parser using v3 and tested * gatan parser using v3 and tested * hitachi parser using v3 and tested * jeol parser using v3 and tested * nion parser using v3 and tested * point electronic parser using v3 and tested * tescan parser using v3 and tested * zeiss parser using v3 and tested but some more concepts should be mapped * tfs/fei tiff parser using v3, testing remains * Some changes on the Velox parser but not complete (refactoring for images and testing) remains * Refactored naming of mapping tables to aid automated lexical sorting during linting * Fix bug with case_three_list handle when concatenating a sequence of pint quantities with units to an array pint quantity for a scaled that same multiplicative unit * tfs parser using v3 and tested * velox refactored and tested, mapping table names shortened, but further testing for velox is required * apex parser using v3 and tested * Suggestions from @lukaspiel for all em PRs in this chain (#61) * Suggestions from @lukaspiel for #60 * Reviewer comments @lukaspiel #57 * Review comments @lukaspiel #55 * Remaining changes of @lukaspiel * Nion kV to V --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> * Update Readme, starting refactoring of EBSD-focused parsers to enable… (#64) * Update Readme, starting refactoring of EBSD-focused parsers to enable making them more general * Started breaking refactoring of EBSD parsers * Initial refactoring for ebsd functionalities of former pyxem parser * Starting initial refactoring of eds functionalities of former pyxem parser * Fixed generic regridding 1d, 2d, fixed H5OINA overview plot, next step fix IPF maps * Reactivated and fixed IPF plotting, next apex * Working on the refactoring of apex * Fixed apex EBSD * Reflecting update of nxdefs * Eventually breaking changes using latest fairmat defs and pynxtools master dev branch * Eventually breaking changes which removes np.column_stack in ebsd.py * Refactoring dream3d * finished code refactoring of remaining hfive parsers, next up: i) testing kdtree, ii) testing all * Initial test of apex and oxford successful after refactoring including with new definitions, next steps i) add eds, ii) testing and removing mismatch to appdef * Continuing on the refactoring of apex going slower than expected * Fixed examples Laehnemann using new schema * Removed deprecated add_specific_metadata code, unified reporting when a parser does not find content * Viteshs example (#65) * Initial tests for implementing #51 * Fixed dangling units * Linting * Completed and tested Viteshs example --------- Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]> * Working EBSD ZIP parser for a special project (#66) * Working EBSD ZIP parser for a special project * Downgraded typing or notation to union for it to work for py3.8 and py3.9 * Reactivated parsers that were temporarily deactivated for testing purposes --------- Co-authored-by: mkuehbach <[email protected]> * Cleaned docs (#67) Co-authored-by: mkuehbach <[email protected]> --------- Co-authored-by: mkuehbach <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: