diff --git a/applications/NXarpes.nxdl.xml b/applications/NXarpes.nxdl.xml index e792d30ad..3c3d822f4 100644 --- a/applications/NXarpes.nxdl.xml +++ b/applications/NXarpes.nxdl.xml @@ -34,8 +34,8 @@ This definition is a legacy support for older NXarpes experiments. There is, however, a newer definition to collect data & metadata - for general photoemission experiments, called :ref:NXmpes, - as well as a specialization for ARPES experiments, called :ref:NXmpes_arpes." + for general photoemission experiments, called :ref:`NXmpes`:, + as well as a specialization for ARPES experiments, called :ref:`NXmpes_arpes`:" diff --git a/dev_tools/docs/nxdl_index.py b/dev_tools/docs/nxdl_index.py index 21cd999a9..c1ca3dd21 100644 --- a/dev_tools/docs/nxdl_index.py +++ b/dev_tools/docs/nxdl_index.py @@ -113,6 +113,18 @@ def get_nxclass_description(nxdl_file: Path, namespaces) -> str: *might* be used in an instance of that class. Consider the base classes as a set of *components* that are used to construct a data file. +Some contributions are grouped together: + :ref:`Optical Spectroscopy ` + + :ref:`Multi-dimensional Photoemission Spectroscopy ` + + :ref:`Atomprobe Microscopy ` + + :ref:`Electron Microscopy ` + + +and others are simply listed here: + """, # - - - - - - - - - - - - - - - - - - - - - - - - - - - - "applications": """ @@ -141,6 +153,18 @@ def get_nxclass_description(nxdl_file: Path, namespaces) -> str: In application definitions involving raw data, write the raw data in the :ref:`NXinstrument` tree and then link to it from the location(s) defined in the relevant application definition. + Some contributions are grouped together: + :ref:`Optical Spectroscopy ` + + :ref:`Multi-dimensional Photoemission Spectroscopy ` + + :ref:`Atomprobe Microscopy ` + + :ref:`Electron Microscopy ` + + +and others are simply listed here: + """, # - - - - - - - - - - - - - - - - - - - - - - - - - - - - "contributed_definitions": """ diff --git a/manual/source/classes/applications/apm-structure.rst b/manual/source/classes/applications/apm-structure.rst new file mode 100644 index 000000000..dd6c45662 --- /dev/null +++ b/manual/source/classes/applications/apm-structure.rst @@ -0,0 +1,284 @@ +.. _Apm-Structure-APP: + +===================== +Atom-probe tomography +===================== + +.. index:: + IntroductionApm + ApmAppDef + ApmBC + StatusQuoApm + ApmParaprobeAppDef + ApmGermanNfdi + +.. _IntroductionApm-APP: + +Introduction +############ + +Set of data schemas to describe the acquisition, i.e. measurement side, the extraction of hits from detector raw data, +steps to compute mass-to-charge state ratios from uncorrected time of flight data, the reconstruction, and the ranging, i.e. identification of peaks in the mass-to-charge-state ratio histogram to detect (molecular) ions. +The data schemas can be useful to generate data artifacts also for field-ion microscopy experiments. + +.. _ApmAppDef-APP: + +Application Definition +###################### + + :ref:`NXapm`: + A general application definition with many detailed places for leaving metadata and computational steps described which are commonly used when reporting the measurement of atom probe data including also detector hit data, as well as how to proceed with reconstructing atom positions from these data, and how to store details about definitions made, which describe how mass-to-charge-state ratio values are mapped to iontypes in a process called ranging. The structure of the schema has been designed to also document a simulation of an atom probe + experiment. Having a combined schema for the measurement and the simulation is beneficial to document that + there are many similarities between the measurement and a computer simulation of it. + +.. _ApmBC-APP: + +Base Classes +############ + +The following base classes are proposed to support modularizing the storage of pieces of information: + + :ref:`NXchamber`: + A base class to describe a component of the instrument which houses other components. + A chamber may offer a controlled atmosphere to execute an experiment and/or offer functionalities + for storing and loading specimens. + + :ref:`NXcoordinate_system_set`, :ref:`NXcoordinate_system`: + Base classes to describe different coordinate systems used and/or to be harmonized + or transformed into one another when interpreting the dataset. + + :ref:`NXion`: (about to become replaced by :ref:`NXatom_set`) + A base class to describe molecular ions with an adjustable number of atoms/isotopes building each ion. + For the usage in atom probe research the maximum number of atoms supported building a molecular ion + is currently set to a maximum of 32. Suggestions made in reference `DOI: 10.1017/S1431927621012241 `_ are used to map isotope to hash values with + which all possible nuclides (stable, radioactive, or synthetically generated ones) can be described. + + :ref:`NXfabrication`: + A base class to bundle manufacturer/technology-partner-specific details about + a component or device of an instrument. + + :ref:`NXpeak`: (about to become complemented by NXpeak_fitting) + A base class to describe peaks mathematically to detail how peaks in + mass-to-charge-state ratio histograms (aka mass spectra) are defined and + labelled as iontypes. + + :ref:`NXpump`: + A base class to describe details about pump(s) used as components of an instrument. + + :ref:`NXpulser_apm`: + A base class to describe the high-voltage and/or laser pulsing capabilities. + + :ref:`NXreflectron`: + A base class to describe a kinetic-energy-sensitive filtering device + for time-of-flight (ToF) mass spectrometry. + + :ref:`NXstage_lab`: + A base class to describe the specimen fixture including the cryo-head. + Nowadays, stages of microscopes represent small-scale laboratory platforms. + Therefore, there is a need to define the characteristics of such stages in more detail, + especially in light of in-situ experiments. Many similarities exists between a stage + in an electron microscope and one in an atom probe instrument. Both offer fixture + functionalities and additional components for applying e.g. stimuli on the specimen. + +Microscopy experiments, not only taking into account those performed on commercial instruments, offer users to apply a set of +data processing steps. Some of them are frequently applied on-the-fly. For now we represent these steps with specifically named +instances of the :ref:`NXprocess` base class. + +Several base classes were defined to document processing of atom probe data with established algorithms: + + :ref:`NXapm_hit_finding`: + A base class to describe hit finding algorithm. + + :ref:`NXapm_volt_and_bowl`: + A base class to describe the voltage-and-bowl correction. + + :ref:`NXapm_charge_state_analysis`: + A base class to document the resolving of the charge_state. + + :ref:`NXapm_reconstruction`: + A base class to document the tomographic reconstruction algorithm. + + :ref:`NXapm_ranging`: + A base class to document the ranging process. + + :ref:`NXapm_msr`, :ref:`NXapm_sim`: + Respective base classes that serve as templates to compose the :ref:`NXapm` application definition from. + +These base classes are examples that substantiate that data processing steps are essential to transform atom probe measurements or simulations into knowledge. Therefore, these steps should be documented +to enable reproducible research, if possible even numerical reproducibility of the results, +and learn how pieces of information are connected. In what follows, an example is presented how an +open-source community software can be modified to use descriptions of these computational steps. + +A detailed inspection of spatial and other type of filters frequently used in analysis of atom probe +data revealed that it is better to define atom-probe-agnostic reusable concepts for filters: + + :ref:`NXspatial_filter`: + A base class proposing how a point cloud can be spatially filtered in a specific yet general manner. + This base class takes advantage of :ref:`NXcg_ellipsoid_set`, :ref:`NXcg_cylinder_set`, + and :ref:`NXcg_hexahedron_set` to cater for commonly used geometric primitives in atom probe. + The primitives are used for defining the shape and extent of a region of interest (ROI). + + :ref:`NXsubsampling_filter`: + A base class for a filter that can also be used for specifying how entries + like ions can be filtered via sub-sampling. + + :ref:`NXmatch_filter`: + A base class for a filter that can also be used for specifying how entries + like ions can be filtered based on their type or other descriptors like hit multiplicity. + +The respective research software here is the `paraprobe-toolbox `_ +The software is developed by `M. Kühbach et al. `_. +For atom probe research the proposal can also serve as a blue print how computational steps of other software +tool including commercial ones could be developed further to benefit from NeXus. + +.. _IntroductionApmParaprobe-APP: + +apmtools +######## + +The paraprobe-toolbox is an example of an open-source parallelized software for analyzing +point cloud data, for assessing meshes in 3D continuum space, and for studying the effects of +parameterization on descriptors of micro- and nanoscale structural features (crystal defects) +within materials when characterized and studied with atom probe. + +The need for a thorough documentation of the tools in not only the paraprobe-toolbox +was motivated by several needs: + +First, users of software would like to better understand and also be able to study for themselves +which individual parameters and settings for each tool exist and how configuring these +affects analyses quantitatively. This stresses the aspect how to improve documentation. + +Second, scientific software like paraprobe-toolbox implement numerical/algorithmical +(computational) workflows whereby data coming from multiple input sources +(like previous analysis results) are processed and carried through more involved analyses +within several steps inside the tool. The tool then creates output as files. This +provenance and workflow should be documented. + +Individual tools of paraprobe-toolbox are developed in C/C++ and/or Python. +Provenance tracking is useful as it is one component and requirement for making +workflows exactly numerically reproducible and thus to enable reproducibility (the "R" +of the FAIR principles of data stewardship). + +For tools of the paraprobe-toolbox each workflow step is a pair or triple of sub-steps: +1. The creation of a configuration file. +2. The actual analysis using the Python/or C/C++ tools. +3. The optional analyses/visualization of the results based on data in NeXus/HDF5 files generated by each tool. + +.. _StatusQuoApm-APP: + +What has been achieved so far? +############################## + +This proposal summarizes work of members of the FAIRmat project, which is part of the `German +National Research Data Infrastructure `_. The here detailed +proposal documents how all tools of the paraprobe-toolbox were modified to generate +only well-defined configuration files as accepted input and yield specifically formatted output +files according to the following NeXus application definitions. + +Data and metadata between the tools are exchanged with NeXus/HDF5 files. This means that data +inside HDF5 binary containers are named, formatted, and hierarchically structured according +to application definitions. + +For example the application definition NXapm_paraprobe_config_surfacer specifies +how a configuration file for the paraprobe-surfacer tool should be formatted +and which parameters it contains including optionality and cardinality constraints. + +Thereby, each config file uses a controlled vocabulary of terms. Furthermore, +the config files store a SHA256 checksum for each input file. This implements a full +provenance tracking on the input files along the workflow. + +As an example, a user may first range their reconstruction and then compute spatial +correlation functions. The config file for the ranging tool stores the files +which hold the reconstructed ion position and ranging definitions. +The ranging tool generates a results file with the labels of each molecular ion. +This results file is formatted according to the tool-specific `results` +application definition. The generated results file and the reconstruction is +imported by the spatial statistics tool which again keeps track of all files +and reports its results in a spatial statistics tool results file. + +This design makes it possible to rigorously trace which numerical results were achieved +with specific inputs and settings using specifically-versioned tools. Noteworthy +this includes Y-junction on a graph which is where multiple input sources are +combined to generate new results. + +We are convinced that defining, documenting, using, and sharing application definitions +is useful and future-proof strategy for software development and data analyses as it enables +automated provenance tracking which happens silently in the background. + +Base classes have been defined to group common pieces of information for each tool of the +toolbox. For each tool we define a pair of base classes. One for the configuration (input) side +and one for the results (output) side: + + :ref:`NXapm_paraprobe_tool_config`, :ref:`NXapm_paraprobe_tool_results`, :ref:`NXapm_paraprobe_tool_common`: + Configuration, results, and common parts respectively useful for the application definitions for tools of the paraprobe-toolbox. + +.. _ApmParaprobeAppDef-APP: + +Application Definitions +####################### + +NXapm_paraprobe application definitions are in fact pairs of application definitions. +One for the configuration (input) side and one for the results (output) side. For each +tool one such pair is proposed: + + :ref:`NXapm_paraprobe_transcoder_config`, :ref:`NXapm_paraprobe_transcoder_results`: + Configuration and the results respectively of the paraprobe-transcoder tool. + Load POS, ePOS, APSuite APT, RRNG, RNG, and NeXus NXapm files. + Store reconstructed positions, ions, and charge states. + + :ref:`NXapm_paraprobe_ranger_config`, :ref:`NXapm_paraprobe_ranger_results`: + Configuration and results respectively of the paraprobe-ranger tool. + Apply ranging definitions and explore possible molecular ions. + Store applied ranging definitions and combinatorial analyses of possible iontypes. + + :ref:`NXapm_paraprobe_selector_config`, :ref:`NXapm_paraprobe_selector_results`: + Configuration and results respectively of the paraprobe-selector tool. + Defining complex spatial regions-of-interest to filter reconstructed datasets. + Store which points are inside or on the boundary of complex spatial regions-of-interest. + + :ref:`NXapm_paraprobe_surfacer_config`, :ref:`NXapm_paraprobe_surfacer_results`: + Configuration and results respectively of the paraprobe-surfacer tool. + Create a model for the edge of a point cloud via convex hulls, alpha shapes, or alpha-wrappings. + Store triangulated surface meshes of models for the edge of a dataset. + + :ref:`NXapm_paraprobe_distancer_config`, :ref:`NXapm_paraprobe_distancer_results`: + Configuration and results respectively of the paraprobe-distancer tool. + Compute and store analytical distances between ions to a set of triangles. + + :ref:`NXapm_paraprobe_tessellator_config`, :ref:`NXapm_paraprobe_tessellator_results`: + Configuration and results respectively of the paraprobe-tessellator tool. + Compute and store Voronoi cells and properties of these for all ions in a dataset. + + :ref:`NXapm_paraprobe_spatstat_config`, :ref:`NXapm_paraprobe_spatstat_results`: + Configuration and results respectively of the paraprobe-spatstat tool. + Compute spatial statistics on the entire or selected regions of the reconstructed dataset. + + :ref:`NXapm_paraprobe_clusterer_config`, :ref:`NXapm_paraprobe_clusterer_results`: + Configuration and results resepctively of the paraprobe-clusterer tool. + Compute cluster analyses with established machine learning algorithms using CPU or GPUs. + + :ref:`NXapm_paraprobe_nanochem_config`, :ref:`NXapm_paraprobe_nanochem_results`: + Configuration and results resepctively of the paraprobe-nanochem tool. + Compute delocalization, iso-surfaces, analyze 3D objects, composition profiles, and mesh interfaces. + + :ref:`NXapm_paraprobe_intersector_config`, :ref:`NXapm_paraprobe_intersector_results`: + Configuration and results resepctively of the paraprobe-intersector tool. + Analyze volumetric intersections and proximity of 3D objects discretized as triangulated surface meshes + in continuum space to study the effect the parameterization of surface extraction algorithms on the resulting shape, + spatial arrangement, and colocation of 3D objects via graph-based techniques. + +.. _ApmGermanNfdi-APP: + +Joint work German NFDI consortia NFDI-MatWerk and FAIRmat +####################################################################### + +Members of the NFDI-MatWerk and the FAIRmat consortium of the German National Research Data Infrastructure +are working together within the Infrastructure Use Case IUC09 of the NFDI-MatWerk project to work on examples +how software tools in both consortia become better documented and interoperable to use. Within this project, +we have also added the `CompositionSpace tool that has been developed at the Max-Planck-Institut für Eisenforschung +GmbH in Düsseldorf `_ by A. Saxena et al. + +Specifically, within the IUC09 we refactored the code base behind the publication `A. Saxena et al. `_ to better document its configuration, here as an example implemented like for the above-mentioned paraprobe-toolbox using NeXus: + + :ref:`NXapm_compositionspace_config`, :ref:`NXapm_compositionspace_results`: + Configuration and the results respectively of the CompositionSpace tool. diff --git a/manual/source/classes/applications/container/ComplexContainerBeampath.png b/manual/source/classes/applications/container/ComplexContainerBeampath.png new file mode 100644 index 000000000..597cee834 Binary files /dev/null and b/manual/source/classes/applications/container/ComplexContainerBeampath.png differ diff --git a/manual/source/classes/applications/container/ComplexExampleContainer.png b/manual/source/classes/applications/container/ComplexExampleContainer.png new file mode 100644 index 000000000..4ffdd862d Binary files /dev/null and b/manual/source/classes/applications/container/ComplexExampleContainer.png differ diff --git a/manual/source/classes/applications/em-structure.rst b/manual/source/classes/applications/em-structure.rst new file mode 100644 index 000000000..782e27489 --- /dev/null +++ b/manual/source/classes/applications/em-structure.rst @@ -0,0 +1,164 @@ +.. _Em-Structure-APP: + +=================== +Electron microscopy +=================== + +.. index:: + IntroductionEm + EmAppDef + EmBC + EmAnalysisClasses + +.. _IntroductionEm-APP: + +Introduction +############ + +A set of data schemas is proposed to describe components of an electron microscope and its eventually available focused-ion beam functionalities. +The data schemas were designed from the perspective of how electron microscopes are used by colleagues in the materials-science-branch of electron microscopy. +We realize that the biology-/bio-materials/omics-branch of electron microscopy is eventually in an already more mature state of discussion with respect +to data management practices. In what follows, the focus is on the usage of electron microscopy in condensed-matter physics, chemical physics of solids, +and materials engineering applications. As many of the components of electron microscopes used in the bio-materials communities are the same or at least many +components are very similar, it is likely that the here presented schema definitions can also inspire discussion and exchange with the bio-materials community. +Partner consortia in the German National Research Data Infrastructure are here e.g. NFDI-BioImage, NFDI-Microbiota, NFDI4Health, and e.g. NFDI-Neuro. + +Electron microscopes are functionally very customizable tools: Examples include multi-signal/-modal analyses which are frequently realized as on-the-fly computational analyses, regularly switching between GUI-based instrument control, computational steps, and more and more using high-throughput stream-based processing. Also artificial intelligence methods are increasingly used and are becoming more closely interconnected with classical modes of controlling the instrument and perform data processing. A challenge in electron microscopy is that these steps are often executed within commercial integrated control and analysis software. This makes it difficult to keep track of workflows in a technology-partner-agnostic, i.e. interdisciplinary manner. + +.. _EmAppDef-APP: + +Application Definitions +####################### + +We acknowledge that it can be difficult to agree on a single application definition which is generally enough applicable yet not unnecessarily complex and useful for applications across a variety of instruments, technology partners, and instrument use cases. In what follows, the proposal conceptualizes first the basic components of an electron microscope and the usual workflow of how an electron microscope is used for collecting data with detectors via probing radiation-specimen-matter interaction mechanisms. + +In summary, scientists place a specimen/sample into the microscope, calibrate the instrument, take measurements, and may perform experiments, prepare their specimens with a focused ion beam, calibrate again, and take other measurements, before their session on the instrument ends. In between virtually all of these steps data are collected and stream in from different detectors probing different physical mechanisms of the interaction between electrons or other types of radiation with the specimen. + +A microscope session ends with the scientist removing the specimen from the instrument or parking it so that the next user can start a session. Occasionally, service technicians perform calibrations and maintenance which also can be described as a session on the microscope. We have provided base classes to describe these steps and events and an application definition for electron microscopy: + + :ref:`NXem`: + An application definition which explores the possibilities of electron microscopes. + + +.. _EmBC-APP: + +Base Classes +############ + +The following base classes are proposed to support modularizing the storage of pieces of information related to electron microscopy research: + + :ref:`NXem_msr`, :ref:`NXem_sim`: + Base classes to distinguish descriptions relevant for an experiment that is performed with a real microscope or a computer simulation of + electron matter interaction. Through these base classes NeXus supports to serialize details of a measurement and a related computer simulation + into one data artifact. + + :ref:`NXidentifier`, :ref:`NXserialized`: + Base classes to support storage of metadata whereby the source of information stored in a NeXus data artifact or class instances can be + documented especially when one does not store all relevant information using NeXus but one would like to refer to a specific other resource + where these pieces of information are stored. + + :ref:`NXebeam_column`: + A base class serving the possibility to group the components relevant for generating + and shaping the electron beam. + + :ref:`NXibeam_column`: + A base class serving the possibility to group the components relevant for generating + and shaping an ion beam of an instrument to offer focused-ion beam (milling) capabilities. + + :ref:`NXcomponent`: + A base class to describe components aka devices to building an instrument like a microscope irrespective whether that is a real one or a simulated one. + + :ref:`NXlens_em`: + A base class to detail an electro-magnetic lens. In practice, an electron microscope has many such lenses. It is possible to specify as many lenses as necessary to represent eventually each single lens of the microscope and thus describe how the lenses are affecting the electron beam. This can offer opportunities for developers of software tools which strive to model the instrument e.g. to create digital twins of the instrument. We understand there is still a way to go with this to arrive there though. Consequently, we suggest to focus first on which details should be collected for a lens as a component so that developers of application definitions can take immediate advantage of this work. + + :ref:`NXdeflector`: + A base class to describe a component to deflect a beam of charged particles. + + :ref:`NXchamber`: + A base class to describe the chamber as a part of the microscope or storage unit + for transferring specimens in between or within an instrument. + + :ref:`NXpump`: + A base class to describe details about pump(s) as components of an electron microscope. + + :ref:`NXfabrication`: + A base class to bundle manufacturer/technology-partner-specific details about a component or device of an instrument. + + :ref:`NXcoordinate_system_set`, :ref:`NXcoordinate_system`, :ref:`NXtransformations`: + Base classes to describe different coordinate systems used and/or to be harmonized + or transformed into one another and respective transformations. + + :ref:`NXcorrector_cs`, :ref:`NXaberration`: + Base classes to describe procedures and values for the calibration of aberrations based on + conventions of different companies active in the field of aberration correction including a base class + to describe details about corrective lens or compound lens devices which reduce + (spherical) aberrations of an electron beam. + + :ref:`NXscanbox_em`: + A base class to represent the component of an electron microscope which realizes a controlled deflection + (and eventually shift, blanking, and/or descanning) of the electron beam to illuminate the specimen in a controlled manner + This base class can be used to document the scan pattern. The base class focuses mostly on the concept idea that there + is a component in a microscope which controls eventually multiple other components such as beam deflectors to achieve deflection + and thus a controlled scanning of the beam over the sample/specimen surface. + + :ref:`NXstage_lab`: + A base class to describe the stage/specimen holder which offers place for the documentation of the small-scale laboratory functionalities + which modern stages of electron microscopes typically offer. + + :ref:`NXevent_data_em`: + A base class representing a container to hold time-stamped and microscope-state-annotated + data during a session at an electron microscope. + + :ref:`NXevent_data_em_set`: + A base class to group all :ref:`NXevent_data_em` instances. + + :ref:`NXimage_set`: + Base classes for storing acquisition details for individual images or stacks of images. + + :ref:`NXspectrum_set`: + A base class and specializations comparable to :ref:`NXimage_set` but for storing spectra. + + :ref:`NXinteraction_vol_em`: + A base class to describe details about e.g. the assumed or simulated volume of interaction of the electrons with the specimen. + + :ref:`NXion`: + A base class to describe molecular ions with an adjustable number of atoms/isotopes building each ion. Right now the maximum number of atoms supported building a molecular ion is 32. Suggestions made in reference `DOI: 10.1017/S1431927621012241 `_ are used to map isotope to hash values with which all possible isotopes can be described. + + :ref:`NXoptical_system_em`: + A base class to store for now qualitative and quantitative values of frequent interest + which are affected by the interplay of the components and state of an electron microscope. + Examples are the semiconvergence angle or the depth of field and depth of focus, the magnification, or the camera length. + + :ref:`NXpeak`: + A base class to describe peaks mathematically. + + :ref:`NXcircuit`: + A base class to describe a logical unit of at least one integrated circuit. + + +.. _EmAnalysisClasses-APP: + +We provide specific base classes which granularize frequently collected or analyzed quantities in specific application fields of electron microscopy to deal +with the situation that there are cases were logical connections between generated data artifacts mainly exist for the fact that the data artifacts were +collected during a workflow of electron microscopy research (e.g. taking measurements and then performing method-specific analyses generating new data and conclusions). +We see a value in granularizing out these pieces of information into own classes. In fact, one limitation of application definitions in NeXus, exactly as it applies for serialization +of information also more generally, is currently that they define a set of constraints on their graph of controlled concepts and terms. + +If we take for example diffraction experiments performed with an electron microscope, it is usually the case that (diffraction) patterns are collected in the session at the microscope. +However, all scientifically relevant conclusions are typically drawn later, i.e. through post-processing the collected diffraction (raw) data. These numerical and algorithmic steps +define computational workflows were data from an instance of an application definition such as NXem are used as input but many additional concepts, constraints, and assumptions +are applied without that these demand necessarily changes in the constraints on fields or groups of NXem. If we were to modify NXem for these cases, +NXem would combinatorially diverge as every different combination of required constraints demands having an own but almost similar application definition. +For this reason, method-specific base classes are used which granularize out how specific pieces of information are processed further to eventually enable their +storage (i.e. serialization) using NeXus. + +More consolidation through the use of NXsubentry classes should be considered in the future. For now we use an approach whereby base classes are combined to reuse vocabulary and a hierarchical organization of pieces of information with specific constraints which are relevant only for specific usage of such data by specific tools used by an eventually smaller circle of users. + + :ref:`NXem_method`, :ref:`NXem_ebsd`, :ref:`NXem_eds`, :ref:`NXem_eels`, :ref:`NXem_img`, :ref:`NXem_correlation`: + Base classes with method-specific details especially when it comes to reporting post-processed data within electron microscopy. + + :ref:`NXcrystal_structure`: + A base class to store crystalline phase/structure used for a simulation of diffraction pattern and comparison of these pattern against patterns to support indexing. + + :ref:`NXroi`: + A base class to granularize information collected and relevant for the characterization of a region-of-interest. diff --git a/manual/source/classes/applications/mpes-structure.rst b/manual/source/classes/applications/mpes-structure.rst new file mode 100644 index 000000000..c92a28334 --- /dev/null +++ b/manual/source/classes/applications/mpes-structure.rst @@ -0,0 +1,42 @@ +.. _Mpes-Structure: + +======================================= +Photoemission & core-level spectroscopy +======================================= + +.. index:: + IntroductionMpes + MpesAppDef + MpesBC + MpesCommonBC + MpesExtendedBC + + +.. _IntroductionMpes-APP: + +Introduction +############ + +Set of data storage objects to describe multidimensional photoemission (MPES) experiments including x-ray photoelectron spectroscopy (XPS), ultraviolet photoelectron spectroscopy (UPS), +hard x-ray photoelectron spectroscopy (HAXPES), angle-resolved photoemission spectroscopy (ARPES), two-photon photoemission (2PPE) +and photoemission electron microscopy (PEEM). Also includes descriptors for advanced specializations, such as spin-resolution, time resolution, +near-ambient pressure conditions, dichroism etc. + +.. _MpesAppDef-APP: + +Application Definitions +####################### + +:ref:`NXmpes`: + A general application definition with minimalistic metadata requirements, apt to describe all photoemission experiments. + +:ref:`NXmpes_arpes`: + An application definition for angle-resolved photoemission spectroscopy (ARPES) experiments. + +:ref:`NXxps`: + An application definition for X-ray/ultraviolet photoelectron spectroscopy (XPS/UPS) measurements. + +:ref:`NXarpes`: + An application definition for angle-resolved photoemission spectroscopy (ARPES) experiments. This definition is a legacy + support for older NXarpes experiments. For newer experiments, the user is advised to use :ref:`NXmpes_arpes`:.” + diff --git a/manual/source/classes/applications/optical-spectroscopy-structure.rst b/manual/source/classes/applications/optical-spectroscopy-structure.rst new file mode 100644 index 000000000..cf610fc21 --- /dev/null +++ b/manual/source/classes/applications/optical-spectroscopy-structure.rst @@ -0,0 +1,199 @@ +.. _Optical-Spectroscopy-Structure-BC: + +==================== +Optical Spectroscopy +==================== + +.. index:: + Ellipsometry + Raman + DispersiveMaterial + + +.. _Ellipsometry-BC: + +Ellipsometry +############ + +Ellipsometry is an optical characterization method to describe optical properties of interfaces and thickness of films. +The measurements are based on determining how the polarization state of light changes upon transmission and reflection. +Interpretation is based on Fresnel equations and numerical models of the optical properties of the materials. + +In the application definition, we provide a minimum set of description elements allowing for a reproducible recording of ellipsometry measurements. + +.. _Raman-BC: + +Raman +############ + +Raman spectroscopy is a characterization method to analyze vibrational properties for liquids, gases, or solids. +The measurements is based on the inelastic light scattering due to the material's vibrations. +Interpretation can be done based on peaks, which represent the phonon properties (intensity, center, width). + +The application definition contains a minimum of descriptive elements required to understand Raman spectroscopy measurements. + + +Application Definitions +----------------------- + + :ref:`NXoptical_spectroscopy`: + A generic application definition for spectroscopy measurements. This includes in particular ellipsometry and Raman spectroscopy measurements, but also other techniques such as photoluminescence, transmission, and reflection measurements. The requirements are: (i) an incident photon beam, (ii) a detector to measure scattered/emitted photons, and (iii) a sample. + + :ref:`NXellipsometry`: + An application definition for ellipsometry measurements, including complex systems up to variable angle spectroscopic ellipsometry. + + :ref:`NXraman`: + An application definition for Raman spectroscopy measurements. + + +Base Classes +------------ + +This is the set of base classes for describing an optical experiment. + + :ref:`NXbeam_device` + Beam devices are used to relate a beam, which has always at least one origin + and at least one destination. + + By referencing the beam devices with each other, a beam path can be + constructed. This can be used for vizualization or beam propery modeling + along the beam path. + + :ref:`NXbeam` + Beam properties such as intensity, polarization, wavelength or direction. + + :ref:`NXdetector` + A detector for signal detection. + + :ref:`NXsource` + A light source such as laser, lamp or LED. + + :ref:`NXmonochromator` + A monochromator is often used to energetically disperse the scattered or emitted light. + + :ref:`NXlens_opt` + Description of an optical lens. + + :ref:`NXwaveplate` + A waveplate or retarder. + + :ref:`NXsensor` + Specify external parameters that have influenced the sample such as + varied parameters e.g. temperature, pressure, pH value, beam intensity, etc. + + + +.. _DispersiveMaterial-BC: + +Dispersive Material +################### + +A dispersive material is a description for the optical dispersion of materials. +This description may be used to store optical model data from an ellipsometric analysis +(or any other technique) or to build a database of optical constants for optical properties of materials. + +Application Definition +---------------------- + + :ref:`NXdispersive_material`: + An application definition to describe the dispersive properties of a material. + The material may be isotropic, uniaxial or biaxial. Hence, it may contain up + to three dispersive functions or tables. + + + +Base Classes +------------ + +There is a set of base classes for describing a dispersion. + + :ref:`NXdispersion` + This is an umbrella base class for a group of dispersion functions to describe the material. + For a simple dispersion it may contain only on NXdispersion_function or NXdispersion_table entry. + If it contains multiple entries the actual dispersion is the sum of all dispersion functions and tables. + This allows for, e.g. splitting real and imaginary parts and describing them seperately or + adding a dielectric background (e.g. Sellmeier model) to an oscillator model (e.g. Lorentz). + + :ref:`NXdispersion_function` + This dispersion is described by a function and its single and repeated parameter values. + It follows a formula of the form ``eps = eps_inf + sum[A * lambda ** 2 / (lambda ** 2 - B ** 2)]`` + (Sellmeier formula). See the formula grammar below for an ebnf grammar for this form. + + :ref:`NXdispersion_single_parameter` + This denotes a parameter which is used outside the summed part of a dispersion function, + e.g. ``eps_inf`` in the formula example above. + + :ref:`NXdispersion_repeated_parameter` + This denotes arrays of repeated parameters which are used to build a sum of parameter values, e.g. + ``A`` and ``B`` are repeated parameters in the formula above. + + :ref:`NXdispersion_table` + This describes a tabular dispersion where the dielectric function is an array versus wavelength or energy. + +Formula Grammar +--------------- + +Below you find a grammar to which the formula should adhere and which can be used to parse and +evaluate the dispersion function. The terms ``single_param_name`` and ``param_name`` should be +filled with the respective single and repeated params from the stored data. +The grammer is written in the `EBNF `_ dialect +of `Lark `_, which is a parsing toolkit for python. +It is easily translatable to general EBNF and other parser generator dialects. +`Here `_ is a reference implementation in Rust/Python with a +`grammar `_ +written in `lalrpop `_. + +.. code-block:: + + ?assignment: "eps" "=" kkr_expression -> eps + | "n" "=" kkr_expression -> n + + ?kkr_expression: expression + | "" "+" "1j" "*" term -> kkr_term + + ?expression: term + | expression "+" term -> add + | expression "-" term -> sub + + ?term: factor + | term "*" factor -> mul + | term "/" factor -> div + + ?factor: power + | power "**" power -> power + + + ?power: "(" expression ")" + | FUNC "(" expression ")" -> func + | "sum" "[" repeated_expression "]" -> sum_expr + | NAME -> single_param_name + | SIGNED_NUMBER -> number + | BUILTIN -> builtin + + ?repeated_expression: repeated_term + | repeated_expression "+" repeated_term -> add + | repeated_expression "-" repeated_term -> sub + + + ?repeated_term: repeated_factor + | repeated_term "*" repeated_factor -> mul + | repeated_term "/" repeated_factor -> div + + ?repeated_factor: repeated_power + | repeated_power "**" repeated_power -> power + + ?repeated_power: "(" repeated_expression ")" + | FUNC "(" repeated_expression ")" -> func + | SIGNED_NUMBER -> number + | NAME -> param_name + | BUILTIN -> builtin + + FUNC.1: "sin" | "cos" | "tan" | "sqrt" | "dawsn" | "ln" | "log" | "heaviside" + BUILTIN.1: "1j" | "pi" | "eps_0" | "hbar" | "h" | "c" + + %import common.CNAME -> NAME + %import common.SIGNED_NUMBER + %import common.WS_INLINE + + %ignore WS_INLINE + diff --git a/manual/source/classes/base_classes/apm-structure.rst b/manual/source/classes/base_classes/apm-structure.rst new file mode 100644 index 000000000..347008864 --- /dev/null +++ b/manual/source/classes/base_classes/apm-structure.rst @@ -0,0 +1,284 @@ +.. _Apm-Structure-BC: + +===================== +Atom-probe tomography +===================== + +.. index:: + IntroductionApm + ApmAppDef + ApmBC + StatusQuoApm + ApmParaprobeAppDef + ApmGermanNfdi + +.. _IntroductionApm: + +Introduction +############ + +Set of data schemas to describe the acquisition, i.e. measurement side, the extraction of hits from detector raw data, +steps to compute mass-to-charge state ratios from uncorrected time of flight data, the reconstruction, and the ranging, i.e. identification of peaks in the mass-to-charge-state ratio histogram to detect (molecular) ions. +The data schemas can be useful to generate data artifacts also for field-ion microscopy experiments. + +.. _ApmAppDef: + +Application Definition +###################### + + :ref:`NXapm`: + A general application definition with many detailed places for leaving metadata and computational steps described which are commonly used when reporting the measurement of atom probe data including also detector hit data, as well as how to proceed with reconstructing atom positions from these data, and how to store details about definitions made, which describe how mass-to-charge-state ratio values are mapped to iontypes in a process called ranging. The structure of the schema has been designed to also document a simulation of an atom probe + experiment. Having a combined schema for the measurement and the simulation is beneficial to document that + there are many similarities between the measurement and a computer simulation of it. + +.. _ApmBC: + +Base Classes +############ + +The following base classes are proposed to support modularizing the storage of pieces of information: + + :ref:`NXchamber`: + A base class to describe a component of the instrument which houses other components. + A chamber may offer a controlled atmosphere to execute an experiment and/or offer functionalities + for storing and loading specimens. + + :ref:`NXcoordinate_system_set`, :ref:`NXcoordinate_system`: + Base classes to describe different coordinate systems used and/or to be harmonized + or transformed into one another when interpreting the dataset. + + :ref:`NXion`: (about to become replaced by :ref:`NXatom_set`) + A base class to describe molecular ions with an adjustable number of atoms/isotopes building each ion. + For the usage in atom probe research the maximum number of atoms supported building a molecular ion + is currently set to a maximum of 32. Suggestions made in reference `DOI: 10.1017/S1431927621012241 `_ are used to map isotope to hash values with + which all possible nuclides (stable, radioactive, or synthetically generated ones) can be described. + + :ref:`NXfabrication`: + A base class to bundle manufacturer/technology-partner-specific details about + a component or device of an instrument. + + :ref:`NXpeak`: (about to become complemented by NXpeak_fitting) + A base class to describe peaks mathematically to detail how peaks in + mass-to-charge-state ratio histograms (aka mass spectra) are defined and + labelled as iontypes. + + :ref:`NXpump`: + A base class to describe details about pump(s) used as components of an instrument. + + :ref:`NXpulser_apm`: + A base class to describe the high-voltage and/or laser pulsing capabilities. + + :ref:`NXreflectron`: + A base class to describe a kinetic-energy-sensitive filtering device + for time-of-flight (ToF) mass spectrometry. + + :ref:`NXstage_lab`: + A base class to describe the specimen fixture including the cryo-head. + Nowadays, stages of microscopes represent small-scale laboratory platforms. + Therefore, there is a need to define the characteristics of such stages in more detail, + especially in light of in-situ experiments. Many similarities exists between a stage + in an electron microscope and one in an atom probe instrument. Both offer fixture + functionalities and additional components for applying e.g. stimuli on the specimen. + +Microscopy experiments, not only taking into account those performed on commercial instruments, offer users to apply a set of +data processing steps. Some of them are frequently applied on-the-fly. For now we represent these steps with specifically named +instances of the :ref:`NXprocess` base class. + +Several base classes were defined to document processing of atom probe data with established algorithms: + + :ref:`NXapm_hit_finding`: + A base class to describe hit finding algorithm. + + :ref:`NXapm_volt_and_bowl`: + A base class to describe the voltage-and-bowl correction. + + :ref:`NXapm_charge_state_analysis`: + A base class to document the resolving of the charge_state. + + :ref:`NXapm_reconstruction`: + A base class to document the tomographic reconstruction algorithm. + + :ref:`NXapm_ranging`: + A base class to document the ranging process. + + :ref:`NXapm_msr`, :ref:`NXapm_sim`: + Respective base classes that serve as templates to compose the :ref:`NXapm` application definition from. + +These base classes are examples that substantiate that data processing steps are essential to transform atom probe measurements or simulations into knowledge. Therefore, these steps should be documented +to enable reproducible research, if possible even numerical reproducibility of the results, +and learn how pieces of information are connected. In what follows, an example is presented how an +open-source community software can be modified to use descriptions of these computational steps. + +A detailed inspection of spatial and other type of filters frequently used in analysis of atom probe +data revealed that it is better to define atom-probe-agnostic reusable concepts for filters: + + :ref:`NXspatial_filter`: + A base class proposing how a point cloud can be spatially filtered in a specific yet general manner. + This base class takes advantage of :ref:`NXcg_ellipsoid_set`, :ref:`NXcg_cylinder_set`, + and :ref:`NXcg_hexahedron_set` to cater for commonly used geometric primitives in atom probe. + The primitives are used for defining the shape and extent of a region of interest (ROI). + + :ref:`NXsubsampling_filter`: + A base class for a filter that can also be used for specifying how entries + like ions can be filtered via sub-sampling. + + :ref:`NXmatch_filter`: + A base class for a filter that can also be used for specifying how entries + like ions can be filtered based on their type or other descriptors like hit multiplicity. + +The respective research software here is the `paraprobe-toolbox `_ +The software is developed by `M. Kühbach et al. `_. +For atom probe research the proposal can also serve as a blue print how computational steps of other software +tool including commercial ones could be developed further to benefit from NeXus. + +.. _IntroductionApmParaprobe: + +apmtools +######## + +The paraprobe-toolbox is an example of an open-source parallelized software for analyzing +point cloud data, for assessing meshes in 3D continuum space, and for studying the effects of +parameterization on descriptors of micro- and nanoscale structural features (crystal defects) +within materials when characterized and studied with atom probe. + +The need for a thorough documentation of the tools in not only the paraprobe-toolbox +was motivated by several needs: + +First, users of software would like to better understand and also be able to study for themselves +which individual parameters and settings for each tool exist and how configuring these +affects analyses quantitatively. This stresses the aspect how to improve documentation. + +Second, scientific software like paraprobe-toolbox implement numerical/algorithmical +(computational) workflows whereby data coming from multiple input sources +(like previous analysis results) are processed and carried through more involved analyses +within several steps inside the tool. The tool then creates output as files. This +provenance and workflow should be documented. + +Individual tools of paraprobe-toolbox are developed in C/C++ and/or Python. +Provenance tracking is useful as it is one component and requirement for making +workflows exactly numerically reproducible and thus to enable reproducibility (the "R" +of the FAIR principles of data stewardship). + +For tools of the paraprobe-toolbox each workflow step is a pair or triple of sub-steps: +1. The creation of a configuration file. +2. The actual analysis using the Python/or C/C++ tools. +3. The optional analyses/visualization of the results based on data in NeXus/HDF5 files generated by each tool. + +.. _StatusQuoApm: + +What has been achieved so far? +############################## + +This proposal summarizes work of members of the FAIRmat project, which is part of the `German +National Research Data Infrastructure `_. The here detailed +proposal documents how all tools of the paraprobe-toolbox were modified to generate +only well-defined configuration files as accepted input and yield specifically formatted output +files according to the following NeXus application definitions. + +Data and metadata between the tools are exchanged with NeXus/HDF5 files. This means that data +inside HDF5 binary containers are named, formatted, and hierarchically structured according +to application definitions. + +For example the application definition NXapm_paraprobe_config_surfacer specifies +how a configuration file for the paraprobe-surfacer tool should be formatted +and which parameters it contains including optionality and cardinality constraints. + +Thereby, each config file uses a controlled vocabulary of terms. Furthermore, +the config files store a SHA256 checksum for each input file. This implements a full +provenance tracking on the input files along the workflow. + +As an example, a user may first range their reconstruction and then compute spatial +correlation functions. The config file for the ranging tool stores the files +which hold the reconstructed ion position and ranging definitions. +The ranging tool generates a results file with the labels of each molecular ion. +This results file is formatted according to the tool-specific `results` +application definition. The generated results file and the reconstruction is +imported by the spatial statistics tool which again keeps track of all files +and reports its results in a spatial statistics tool results file. + +This design makes it possible to rigorously trace which numerical results were achieved +with specific inputs and settings using specifically-versioned tools. Noteworthy +this includes Y-junction on a graph which is where multiple input sources are +combined to generate new results. + +We are convinced that defining, documenting, using, and sharing application definitions +is useful and future-proof strategy for software development and data analyses as it enables +automated provenance tracking which happens silently in the background. + +Base classes have been defined to group common pieces of information for each tool of the +toolbox. For each tool we define a pair of base classes. One for the configuration (input) side +and one for the results (output) side: + + :ref:`NXapm_paraprobe_tool_config`, :ref:`NXapm_paraprobe_tool_results`, :ref:`NXapm_paraprobe_tool_common`: + Configuration, results, and common parts respectively useful for the application definitions for tools of the paraprobe-toolbox. + +.. _ApmParaprobeAppDef: + +Application Definitions +####################### + +NXapm_paraprobe application definitions are in fact pairs of application definitions. +One for the configuration (input) side and one for the results (output) side. For each +tool one such pair is proposed: + + :ref:`NXapm_paraprobe_transcoder_config`, :ref:`NXapm_paraprobe_transcoder_results`: + Configuration and the results respectively of the paraprobe-transcoder tool. + Load POS, ePOS, APSuite APT, RRNG, RNG, and NeXus NXapm files. + Store reconstructed positions, ions, and charge states. + + :ref:`NXapm_paraprobe_ranger_config`, :ref:`NXapm_paraprobe_ranger_results`: + Configuration and results respectively of the paraprobe-ranger tool. + Apply ranging definitions and explore possible molecular ions. + Store applied ranging definitions and combinatorial analyses of possible iontypes. + + :ref:`NXapm_paraprobe_selector_config`, :ref:`NXapm_paraprobe_selector_results`: + Configuration and results respectively of the paraprobe-selector tool. + Defining complex spatial regions-of-interest to filter reconstructed datasets. + Store which points are inside or on the boundary of complex spatial regions-of-interest. + + :ref:`NXapm_paraprobe_surfacer_config`, :ref:`NXapm_paraprobe_surfacer_results`: + Configuration and results respectively of the paraprobe-surfacer tool. + Create a model for the edge of a point cloud via convex hulls, alpha shapes, or alpha-wrappings. + Store triangulated surface meshes of models for the edge of a dataset. + + :ref:`NXapm_paraprobe_distancer_config`, :ref:`NXapm_paraprobe_distancer_results`: + Configuration and results respectively of the paraprobe-distancer tool. + Compute and store analytical distances between ions to a set of triangles. + + :ref:`NXapm_paraprobe_tessellator_config`, :ref:`NXapm_paraprobe_tessellator_results`: + Configuration and results respectively of the paraprobe-tessellator tool. + Compute and store Voronoi cells and properties of these for all ions in a dataset. + + :ref:`NXapm_paraprobe_spatstat_config`, :ref:`NXapm_paraprobe_spatstat_results`: + Configuration and results respectively of the paraprobe-spatstat tool. + Compute spatial statistics on the entire or selected regions of the reconstructed dataset. + + :ref:`NXapm_paraprobe_clusterer_config`, :ref:`NXapm_paraprobe_clusterer_results`: + Configuration and results resepctively of the paraprobe-clusterer tool. + Compute cluster analyses with established machine learning algorithms using CPU or GPUs. + + :ref:`NXapm_paraprobe_nanochem_config`, :ref:`NXapm_paraprobe_nanochem_results`: + Configuration and results resepctively of the paraprobe-nanochem tool. + Compute delocalization, iso-surfaces, analyze 3D objects, composition profiles, and mesh interfaces. + + :ref:`NXapm_paraprobe_intersector_config`, :ref:`NXapm_paraprobe_intersector_results`: + Configuration and results resepctively of the paraprobe-intersector tool. + Analyze volumetric intersections and proximity of 3D objects discretized as triangulated surface meshes + in continuum space to study the effect the parameterization of surface extraction algorithms on the resulting shape, + spatial arrangement, and colocation of 3D objects via graph-based techniques. + +.. _ApmGermanNfdi: + +Joint work German NFDI consortia NFDI-MatWerk and FAIRmat +####################################################################### + +Members of the NFDI-MatWerk and the FAIRmat consortium of the German National Research Data Infrastructure +are working together within the Infrastructure Use Case IUC09 of the NFDI-MatWerk project to work on examples +how software tools in both consortia become better documented and interoperable to use. Within this project, +we have also added the `CompositionSpace tool that has been developed at the Max-Planck-Institut für Eisenforschung +GmbH in Düsseldorf `_ by A. Saxena et al. + +Specifically, within the IUC09 we refactored the code base behind the publication `A. Saxena et al. `_ to better document its configuration, here as an example implemented like for the above-mentioned paraprobe-toolbox using NeXus: + + :ref:`NXapm_compositionspace_config`, :ref:`NXapm_compositionspace_results`: + Configuration and the results respectively of the CompositionSpace tool. diff --git a/manual/source/classes/base_classes/container/ComplexContainerBeampath.png b/manual/source/classes/base_classes/container/ComplexContainerBeampath.png new file mode 100644 index 000000000..597cee834 Binary files /dev/null and b/manual/source/classes/base_classes/container/ComplexContainerBeampath.png differ diff --git a/manual/source/classes/base_classes/container/ComplexExampleContainer.png b/manual/source/classes/base_classes/container/ComplexExampleContainer.png new file mode 100644 index 000000000..4ffdd862d Binary files /dev/null and b/manual/source/classes/base_classes/container/ComplexExampleContainer.png differ diff --git a/manual/source/classes/base_classes/em-structure.rst b/manual/source/classes/base_classes/em-structure.rst new file mode 100644 index 000000000..5985ad8d7 --- /dev/null +++ b/manual/source/classes/base_classes/em-structure.rst @@ -0,0 +1,164 @@ +.. _Em-Structure-BC: + +=================== +Electron microscopy +=================== + +.. index:: + IntroductionEm + EmAppDef + EmBC + EmAnalysisClasses + +.. _IntroductionEm: + +Introduction +############ + +A set of data schemas is proposed to describe components of an electron microscope and its eventually available focused-ion beam functionalities. +The data schemas were designed from the perspective of how electron microscopes are used by colleagues in the materials-science-branch of electron microscopy. +We realize that the biology-/bio-materials/omics-branch of electron microscopy is eventually in an already more mature state of discussion with respect +to data management practices. In what follows, the focus is on the usage of electron microscopy in condensed-matter physics, chemical physics of solids, +and materials engineering applications. As many of the components of electron microscopes used in the bio-materials communities are the same or at least many +components are very similar, it is likely that the here presented schema definitions can also inspire discussion and exchange with the bio-materials community. +Partner consortia in the German National Research Data Infrastructure are here e.g. NFDI-BioImage, NFDI-Microbiota, NFDI4Health, and e.g. NFDI-Neuro. + +Electron microscopes are functionally very customizable tools: Examples include multi-signal/-modal analyses which are frequently realized as on-the-fly computational analyses, regularly switching between GUI-based instrument control, computational steps, and more and more using high-throughput stream-based processing. Also artificial intelligence methods are increasingly used and are becoming more closely interconnected with classical modes of controlling the instrument and perform data processing. A challenge in electron microscopy is that these steps are often executed within commercial integrated control and analysis software. This makes it difficult to keep track of workflows in a technology-partner-agnostic, i.e. interdisciplinary manner. + +.. _EmAppDef: + +Application Definitions +####################### + +We acknowledge that it can be difficult to agree on a single application definition which is generally enough applicable yet not unnecessarily complex and useful for applications across a variety of instruments, technology partners, and instrument use cases. In what follows, the proposal conceptualizes first the basic components of an electron microscope and the usual workflow of how an electron microscope is used for collecting data with detectors via probing radiation-specimen-matter interaction mechanisms. + +In summary, scientists place a specimen/sample into the microscope, calibrate the instrument, take measurements, and may perform experiments, prepare their specimens with a focused ion beam, calibrate again, and take other measurements, before their session on the instrument ends. In between virtually all of these steps data are collected and stream in from different detectors probing different physical mechanisms of the interaction between electrons or other types of radiation with the specimen. + +A microscope session ends with the scientist removing the specimen from the instrument or parking it so that the next user can start a session. Occasionally, service technicians perform calibrations and maintenance which also can be described as a session on the microscope. We have provided base classes to describe these steps and events and an application definition for electron microscopy: + + :ref:`NXem`: + An application definition which explores the possibilities of electron microscopes. + + +.. _EmBC: + +Base Classes +############ + +The following base classes are proposed to support modularizing the storage of pieces of information related to electron microscopy research: + + :ref:`NXem_msr`, :ref:`NXem_sim`: + Base classes to distinguish descriptions relevant for an experiment that is performed with a real microscope or a computer simulation of + electron matter interaction. Through these base classes NeXus supports to serialize details of a measurement and a related computer simulation + into one data artifact. + + :ref:`NXidentifier`, :ref:`NXserialized`: + Base classes to support storage of metadata whereby the source of information stored in a NeXus data artifact or class instances can be + documented especially when one does not store all relevant information using NeXus but one would like to refer to a specific other resource + where these pieces of information are stored. + + :ref:`NXebeam_column`: + A base class serving the possibility to group the components relevant for generating + and shaping the electron beam. + + :ref:`NXibeam_column`: + A base class serving the possibility to group the components relevant for generating + and shaping an ion beam of an instrument to offer focused-ion beam (milling) capabilities. + + :ref:`NXcomponent`: + A base class to describe components aka devices to building an instrument like a microscope irrespective whether that is a real one or a simulated one. + + :ref:`NXlens_em`: + A base class to detail an electro-magnetic lens. In practice, an electron microscope has many such lenses. It is possible to specify as many lenses as necessary to represent eventually each single lens of the microscope and thus describe how the lenses are affecting the electron beam. This can offer opportunities for developers of software tools which strive to model the instrument e.g. to create digital twins of the instrument. We understand there is still a way to go with this to arrive there though. Consequently, we suggest to focus first on which details should be collected for a lens as a component so that developers of application definitions can take immediate advantage of this work. + + :ref:`NXdeflector`: + A base class to describe a component to deflect a beam of charged particles. + + :ref:`NXchamber`: + A base class to describe the chamber as a part of the microscope or storage unit + for transferring specimens in between or within an instrument. + + :ref:`NXpump`: + A base class to describe details about pump(s) as components of an electron microscope. + + :ref:`NXfabrication`: + A base class to bundle manufacturer/technology-partner-specific details about a component or device of an instrument. + + :ref:`NXcoordinate_system_set`, :ref:`NXcoordinate_system`, :ref:`NXtransformations`: + Base classes to describe different coordinate systems used and/or to be harmonized + or transformed into one another and respective transformations. + + :ref:`NXcorrector_cs`, :ref:`NXaberration`: + Base classes to describe procedures and values for the calibration of aberrations based on + conventions of different companies active in the field of aberration correction including a base class + to describe details about corrective lens or compound lens devices which reduce + (spherical) aberrations of an electron beam. + + :ref:`NXscanbox_em`: + A base class to represent the component of an electron microscope which realizes a controlled deflection + (and eventually shift, blanking, and/or descanning) of the electron beam to illuminate the specimen in a controlled manner + This base class can be used to document the scan pattern. The base class focuses mostly on the concept idea that there + is a component in a microscope which controls eventually multiple other components such as beam deflectors to achieve deflection + and thus a controlled scanning of the beam over the sample/specimen surface. + + :ref:`NXstage_lab`: + A base class to describe the stage/specimen holder which offers place for the documentation of the small-scale laboratory functionalities + which modern stages of electron microscopes typically offer. + + :ref:`NXevent_data_em`: + A base class representing a container to hold time-stamped and microscope-state-annotated + data during a session at an electron microscope. + + :ref:`NXevent_data_em_set`: + A base class to group all :ref:`NXevent_data_em` instances. + + :ref:`NXimage_set`: + Base classes for storing acquisition details for individual images or stacks of images. + + :ref:`NXspectrum_set`: + A base class and specializations comparable to :ref:`NXimage_set` but for storing spectra. + + :ref:`NXinteraction_vol_em`: + A base class to describe details about e.g. the assumed or simulated volume of interaction of the electrons with the specimen. + + :ref:`NXion`: + A base class to describe molecular ions with an adjustable number of atoms/isotopes building each ion. Right now the maximum number of atoms supported building a molecular ion is 32. Suggestions made in reference `DOI: 10.1017/S1431927621012241 `_ are used to map isotope to hash values with which all possible isotopes can be described. + + :ref:`NXoptical_system_em`: + A base class to store for now qualitative and quantitative values of frequent interest + which are affected by the interplay of the components and state of an electron microscope. + Examples are the semiconvergence angle or the depth of field and depth of focus, the magnification, or the camera length. + + :ref:`NXpeak`: + A base class to describe peaks mathematically. + + :ref:`NXcircuit`: + A base class to describe a logical unit of at least one integrated circuit. + + +.. _EmAnalysisClasses: + +We provide specific base classes which granularize frequently collected or analyzed quantities in specific application fields of electron microscopy to deal +with the situation that there are cases were logical connections between generated data artifacts mainly exist for the fact that the data artifacts were +collected during a workflow of electron microscopy research (e.g. taking measurements and then performing method-specific analyses generating new data and conclusions). +We see a value in granularizing out these pieces of information into own classes. In fact, one limitation of application definitions in NeXus, exactly as it applies for serialization +of information also more generally, is currently that they define a set of constraints on their graph of controlled concepts and terms. + +If we take for example diffraction experiments performed with an electron microscope, it is usually the case that (diffraction) patterns are collected in the session at the microscope. +However, all scientifically relevant conclusions are typically drawn later, i.e. through post-processing the collected diffraction (raw) data. These numerical and algorithmic steps +define computational workflows were data from an instance of an application definition such as NXem are used as input but many additional concepts, constraints, and assumptions +are applied without that these demand necessarily changes in the constraints on fields or groups of NXem. If we were to modify NXem for these cases, +NXem would combinatorially diverge as every different combination of required constraints demands having an own but almost similar application definition. +For this reason, method-specific base classes are used which granularize out how specific pieces of information are processed further to eventually enable their +storage (i.e. serialization) using NeXus. + +More consolidation through the use of NXsubentry classes should be considered in the future. For now we use an approach whereby base classes are combined to reuse vocabulary and a hierarchical organization of pieces of information with specific constraints which are relevant only for specific usage of such data by specific tools used by an eventually smaller circle of users. + + :ref:`NXem_method`, :ref:`NXem_ebsd`, :ref:`NXem_eds`, :ref:`NXem_eels`, :ref:`NXem_img`, :ref:`NXem_correlation`: + Base classes with method-specific details especially when it comes to reporting post-processed data within electron microscopy. + + :ref:`NXcrystal_structure`: + A base class to store crystalline phase/structure used for a simulation of diffraction pattern and comparison of these pattern against patterns to support indexing. + + :ref:`NXroi`: + A base class to granularize information collected and relevant for the characterization of a region-of-interest. diff --git a/manual/source/classes/base_classes/mpes-structure.rst b/manual/source/classes/base_classes/mpes-structure.rst new file mode 100644 index 000000000..5486afb88 --- /dev/null +++ b/manual/source/classes/base_classes/mpes-structure.rst @@ -0,0 +1,77 @@ +.. _Mpes-Structure-BC: + +======================================= +Photoemission & core-level spectroscopy +======================================= + +.. index:: + IntroductionMpes + MpesAppDef + MpesBC + MpesCommonBC + MpesExtendedBC + + +.. _IntroductionMpes: + +Introduction +############ + +Set of data storage objects to describe multidimensional photoemission (MPES) experiments including x-ray photoelectron spectroscopy (XPS), ultraviolet photoelectron spectroscopy (UPS), +hard x-ray photoelectron spectroscopy (HAXPES), angle-resolved photoemission spectroscopy (ARPES), two-photon photoemission (2PPE) +and photoemission electron microscopy (PEEM). Also includes descriptors for advanced specializations, such as spin-resolution, time resolution, +near-ambient pressure conditions, dichroism etc. + +.. _MpesBC: + +Base Classes +############ + +:ref:`NXelectronanalyser`: + A base class to describe electron kinetic energy analizers. Contains the collective characteristics of the instrument such as energy resolution, and includes the following subclasses: + + :ref:`NXcollectioncolumn`: + Base class to describe the set of electronic lenses in the electron collection column (standard, PEEM, momentum-microscope, etc.). + + :ref:`NXenergydispersion`: + Base class to describe the energy dispersion sytem (hemispherical, time-of-flight, etc.). + + :ref:`NXspindispersion`: + Base class to describe the set of electronic lenses in the electron collection column. + +:ref:`NXmanipulator`: + A base class to describe the complex manipulators used in photoemission experiments, often with > 4 degrees of freedom, + cryogenic cooling and other advanced features. + +Three base classes to describe data processing, which can be used as subclasses of :ref:`NXprocess` if describing post-processing or as subclasses of :ref:`NXdetector` if describing live, electronics level processing: + + :ref:`NXcalibration`: + A base class to describe the 1D calibration of an axis, with a function mapping a raw data scale to a calibrated scale with the same number of points. + + :ref:`NXdistortion`: + A base class to describe the 2D distortion correction of an axis, with a matrix mapping a raw data image to a undistorted image. + + :ref:`NXregistration`: + A base class to describe the rigid transformations that are applied to an image. May be redundant as they can be described with :ref:`NXtransformations`. + + :ref:`NXprocess_mpes`: + A base class specializing :ref:`NXprocess`, describing events of data processing, reconstruction, or analysis for photoemission-related data. + +.. _MpesCommonBC: + +Common Base Classes +################### + +There are related base classes that are common to other techniques: + + :ref:`NXlens_em`: + A class to describe all types of lenses. Includes electrostatic lenses for electron energy analysers. + + :ref:`NXdeflector` + A class to describe all kinds of deflectors, including electrostatic and magnetostatic deflectors for electron energy analysers. + + :ref:`NXresolution`: + Describes the resolution of a physical quantity, e.g. the resolution of the MPES spectrometer. + + :ref:`NXfit`, :ref:`NXpeak`, :ref:`NXfit_background`, :ref:`NXfit_function`, :ref:`NXfit_parameter`: + Base classes for describing a fit procedure, e.g. a peak fitting in energy space in XPS. \ No newline at end of file diff --git a/manual/source/classes/base_classes/optical-spectroscopy-structure.rst b/manual/source/classes/base_classes/optical-spectroscopy-structure.rst new file mode 100644 index 000000000..cf610fc21 --- /dev/null +++ b/manual/source/classes/base_classes/optical-spectroscopy-structure.rst @@ -0,0 +1,199 @@ +.. _Optical-Spectroscopy-Structure-BC: + +==================== +Optical Spectroscopy +==================== + +.. index:: + Ellipsometry + Raman + DispersiveMaterial + + +.. _Ellipsometry-BC: + +Ellipsometry +############ + +Ellipsometry is an optical characterization method to describe optical properties of interfaces and thickness of films. +The measurements are based on determining how the polarization state of light changes upon transmission and reflection. +Interpretation is based on Fresnel equations and numerical models of the optical properties of the materials. + +In the application definition, we provide a minimum set of description elements allowing for a reproducible recording of ellipsometry measurements. + +.. _Raman-BC: + +Raman +############ + +Raman spectroscopy is a characterization method to analyze vibrational properties for liquids, gases, or solids. +The measurements is based on the inelastic light scattering due to the material's vibrations. +Interpretation can be done based on peaks, which represent the phonon properties (intensity, center, width). + +The application definition contains a minimum of descriptive elements required to understand Raman spectroscopy measurements. + + +Application Definitions +----------------------- + + :ref:`NXoptical_spectroscopy`: + A generic application definition for spectroscopy measurements. This includes in particular ellipsometry and Raman spectroscopy measurements, but also other techniques such as photoluminescence, transmission, and reflection measurements. The requirements are: (i) an incident photon beam, (ii) a detector to measure scattered/emitted photons, and (iii) a sample. + + :ref:`NXellipsometry`: + An application definition for ellipsometry measurements, including complex systems up to variable angle spectroscopic ellipsometry. + + :ref:`NXraman`: + An application definition for Raman spectroscopy measurements. + + +Base Classes +------------ + +This is the set of base classes for describing an optical experiment. + + :ref:`NXbeam_device` + Beam devices are used to relate a beam, which has always at least one origin + and at least one destination. + + By referencing the beam devices with each other, a beam path can be + constructed. This can be used for vizualization or beam propery modeling + along the beam path. + + :ref:`NXbeam` + Beam properties such as intensity, polarization, wavelength or direction. + + :ref:`NXdetector` + A detector for signal detection. + + :ref:`NXsource` + A light source such as laser, lamp or LED. + + :ref:`NXmonochromator` + A monochromator is often used to energetically disperse the scattered or emitted light. + + :ref:`NXlens_opt` + Description of an optical lens. + + :ref:`NXwaveplate` + A waveplate or retarder. + + :ref:`NXsensor` + Specify external parameters that have influenced the sample such as + varied parameters e.g. temperature, pressure, pH value, beam intensity, etc. + + + +.. _DispersiveMaterial-BC: + +Dispersive Material +################### + +A dispersive material is a description for the optical dispersion of materials. +This description may be used to store optical model data from an ellipsometric analysis +(or any other technique) or to build a database of optical constants for optical properties of materials. + +Application Definition +---------------------- + + :ref:`NXdispersive_material`: + An application definition to describe the dispersive properties of a material. + The material may be isotropic, uniaxial or biaxial. Hence, it may contain up + to three dispersive functions or tables. + + + +Base Classes +------------ + +There is a set of base classes for describing a dispersion. + + :ref:`NXdispersion` + This is an umbrella base class for a group of dispersion functions to describe the material. + For a simple dispersion it may contain only on NXdispersion_function or NXdispersion_table entry. + If it contains multiple entries the actual dispersion is the sum of all dispersion functions and tables. + This allows for, e.g. splitting real and imaginary parts and describing them seperately or + adding a dielectric background (e.g. Sellmeier model) to an oscillator model (e.g. Lorentz). + + :ref:`NXdispersion_function` + This dispersion is described by a function and its single and repeated parameter values. + It follows a formula of the form ``eps = eps_inf + sum[A * lambda ** 2 / (lambda ** 2 - B ** 2)]`` + (Sellmeier formula). See the formula grammar below for an ebnf grammar for this form. + + :ref:`NXdispersion_single_parameter` + This denotes a parameter which is used outside the summed part of a dispersion function, + e.g. ``eps_inf`` in the formula example above. + + :ref:`NXdispersion_repeated_parameter` + This denotes arrays of repeated parameters which are used to build a sum of parameter values, e.g. + ``A`` and ``B`` are repeated parameters in the formula above. + + :ref:`NXdispersion_table` + This describes a tabular dispersion where the dielectric function is an array versus wavelength or energy. + +Formula Grammar +--------------- + +Below you find a grammar to which the formula should adhere and which can be used to parse and +evaluate the dispersion function. The terms ``single_param_name`` and ``param_name`` should be +filled with the respective single and repeated params from the stored data. +The grammer is written in the `EBNF `_ dialect +of `Lark `_, which is a parsing toolkit for python. +It is easily translatable to general EBNF and other parser generator dialects. +`Here `_ is a reference implementation in Rust/Python with a +`grammar `_ +written in `lalrpop `_. + +.. code-block:: + + ?assignment: "eps" "=" kkr_expression -> eps + | "n" "=" kkr_expression -> n + + ?kkr_expression: expression + | "" "+" "1j" "*" term -> kkr_term + + ?expression: term + | expression "+" term -> add + | expression "-" term -> sub + + ?term: factor + | term "*" factor -> mul + | term "/" factor -> div + + ?factor: power + | power "**" power -> power + + + ?power: "(" expression ")" + | FUNC "(" expression ")" -> func + | "sum" "[" repeated_expression "]" -> sum_expr + | NAME -> single_param_name + | SIGNED_NUMBER -> number + | BUILTIN -> builtin + + ?repeated_expression: repeated_term + | repeated_expression "+" repeated_term -> add + | repeated_expression "-" repeated_term -> sub + + + ?repeated_term: repeated_factor + | repeated_term "*" repeated_factor -> mul + | repeated_term "/" repeated_factor -> div + + ?repeated_factor: repeated_power + | repeated_power "**" repeated_power -> power + + ?repeated_power: "(" repeated_expression ")" + | FUNC "(" repeated_expression ")" -> func + | SIGNED_NUMBER -> number + | NAME -> param_name + | BUILTIN -> builtin + + FUNC.1: "sin" | "cos" | "tan" | "sqrt" | "dawsn" | "ln" | "log" | "heaviside" + BUILTIN.1: "1j" | "pi" | "eps_0" | "hbar" | "h" | "c" + + %import common.CNAME -> NAME + %import common.SIGNED_NUMBER + %import common.WS_INLINE + + %ignore WS_INLINE +