Documentation for our repos NOMAD-Measurments and NOMAD-Material-Processing and AreaA-data_modelling_and_schemas

[budschi]

I brainstormed on what we discussed lately in different meetings regarding the differen levels of schema in NOMAD and how users should get started with NOMAD for experimental data. The goal is to help new users get started with NOMAD and show them best practices and also help ourselves to be consistent. At the end we have to update the READMEs of our Area A controlled repos and also contribute to the NOMAD documentation. This could be part of that. Please add your thoughts and comments.

• Using NOMAD for experimental data:
  • NOMAD entries are schema based --> uploaded data gets processed into NOMAD when a matching data schema exists
  • there a three different levels of schema
    • base sections
      • General definitions, relation between different EntryData (entity-activity-model)
      • NOMAD tools get developed on base sections (search app, workflow, visualizations, etc.)
      • Not for specific field
      • Maintained by FAIRmat
      • No EntryData
    • standard plugins
      • inherit from base sections (!?)
      • Not for specific lab/instrument
      • nomad-measurements, nomad-material-processing
      • Community (CHMO, AutoPeroSol, NeXus etc.)
    • user schema
      • should inherit from base sections (Applies one or more base sections)
      • could inherit from standard plugins
      • can contribute/partly move up to standard plugin --> work of FAIRmat team (establish community standards)
      • Specific to lab/instrument
      • Developed by users
      • Creates instances by inheriting EntryData
  • user have 4 options to use NOMAD, e.g. for a measurement:
    • Built-in ELN Measurment
      • a ELN that is built on the basesections.Measurment : brings in the functionality of the basesections, but measurement-specific data is not stored in a structured way (plain text field)
    • Custom YAML Schema
      • they should be built on basesections.Measurment or a Standard plugin and then are customized by extending with the specific quantities or sections that the user needs to sore all (meta)data of a measurement
    • NOMAD Plugin with Normalizer
      • they should be built on basesections.Measurment or a Standard plugin based and can be customize (same as Custom YAML Schema)
      • enable automatization like raw file readers, NOMAD parsers (just drop raw file in NOMAD and they get processed), calculations/analysis within schemas
    • standardized NeXus Reader
      • make use of community-based NeXus Reader and NeXus app defs if available
  • typical NOMAD implementation workflow:
    • user does a data revision of his existing experimental data and creates a experimental data workflow
    • user can start out with NOMAD and use the built-in ELNs to get used to and benefit from the NOMAD basesections data model
    • user checks if there is anything useful for him in the Standard plugins
    • user builds a Custom Yaml Schema based on basesections or if possible Standard plugin (--> see examples in https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas)
    • if power of plugin is required the Custom Yaml Schema gets extended into a NOMAD plugin (e.g. PLD based on PVD in https://github.com/FAIRmat-NFDI/nomad-material-processing )
  • best practice Custom yamls schema
    • inherit from basesections
    • if possible inherit from Standard plugins
    • share with FAIRmat --> helps community building and standardization
    • reward/benefit:
      • community standards evolve
      • access to tools that are built on the basesections and standard plugins
  • best practice NOMAD plugin
    • same as above
    • use Hampus tool github.com/hampusnasstrom/metainfo-yaml2py --> plugin gets python package etc.
    • for plugins with file readers,
      • schema.py, reader.py, parser.py --> modularize to make the plugin easy to maintain or exchange parts by community contributions (e.g. Nexus reader)
    • FAIRmat team includes general schema structure in FAIRmat controlled standard plugins --> standardization and communnity building (see PVD example)

[budschi]

@hampusnasstrom

[budschi]

Documentation in NOMAD:

• different roles of users:
  

• start with Built-In schemas. Main message no schema writing, make use of basesections and get interlinked data in NOMAD, get concept of NOMAD ui/ux (upload, explore)

• custom yaml schema: reuse our tutorial 8 as much as possible and extend with standard/community plugin when part of NOMAD

• plugin: comes a bit later

• things from hackathon

things to clarify:

• general picture/slide of Markus --> use introduction to schemas in NOMAD

• should our NOMAD-xxx plugins become part of NOMAD?

  • yes, both.
  • how? Update cycles, cross dependencies
  • coolect all questions and organize meeting with Markus S.

[budschi]

Deadline: finish Built-in and Custom-Schema Tutorial until Users meeting in November (13.11.2023)