YANG Parsing Issues (v2.0.2)

The following expounds on the YANG parsing problems inherent within TDM which make its information less than perfectly reliable.

Upstream published YANG missing models

The published YANG models do not always include all of the necessary models to fully compile the schema. https://github.com/YangModels/yang/issues/375

Upstream YANG NETCONF Hello schemas are broken per platform.

A critical issue for YANG parsing is that the YangModels/yang repository does not actually have all the YANG modules it needs to for the full YANG schema. This breaks full schema parsing. An example is the following attempting to use the provided NETCONF Hello:

{
    "xe/1691/cbr-netconf-capability.xml": [
        "module 'Cisco-IOS-XE-arp@2017-11-07' specified in hello not found.",
        "module 'Cisco-IOS-XE-bgp@2018-06-26' specified in hello not found.",
        "module 'Cisco-IOS-XE-crypto@2018-05-21' specified in hello not found.",
        "module 'Cisco-IOS-XE-eigrp@2017-09-21' specified in hello not found.",
        "module 'Cisco-IOS-XE-isis@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-mpls@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-multicast@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-nat@2018-05-29' specified in hello not found.",
        "module 'Cisco-IOS-XE-ntp@2018-01-05' specified in hello not found.",
        "module 'Cisco-IOS-XE-ospf@2018-05-21' specified in hello not found.",
        "module 'Cisco-IOS-XE-ospfv3@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-policy@2018-05-14' specified in hello not found.",
        "module 'Cisco-IOS-XE-rip@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-sla@2018-04-11' specified in hello not found.",
        "module 'Cisco-IOS-XE-snmp@2017-11-27' specified in hello not found.",
        "module 'Cisco-IOS-XE-vrrp@2018-04-11' specified in hello not found."
    ]
}

These modules/revisions being missing or simply improperly exposed breaks correct parsing. This is a fundamental exposure issue which we currently attempt to resolve with stopgap forks in cisco-ie/yang@fix-ietf-types-cisco (expanded beyond simply fixing IETF types). We're simply pulling the modules from wherever they exist in other directories to fix, but upstream desires them to be verified from the release in question. This would require going through each previous release/product image and extracting the models.

This is also not simply a Hello message issue - sometimes fundamental types are not published with the models as well - for instance

TDM parsing currently ignores NETCONF Hello messages and parses every YANG module in a directory with whatever latest revision exists.

Exposed NETCONF Hello messages have defaults pyang doesn't have.

Internal tooling generating NETCONF Hello messages adds default revisions which aren't necessarily there, breaking pyang tooling. Either the modules should be modified to have the revisions listed, or pyang needs to handle parsing defaults somehow. https://github.com/YangModels/yang/issues/727

Nebulous usage of pyang internals.

TDM currently uses pyang internals instead of any plugins. pyang is difficult to use as a library, and our usage is indeed imperfect. This has extremely important implications - such as whether we are implicitly parsing deviations for an entire OS/release which might only apply to certain less-than-well-supported platforms. If we are deviating critical XPaths which should be available via IOS XE on Catalyst 9300 but aren't yet on ASR1k, we desperately need to NOT parse those deviations as it removes it from the entire OS/Release tree otherwise.

This deviations issue is also a huge implication for the TDM data model (as in ArangoDB) as it requires a OS & Release & Device to DataPath relationship, or OS & Release & Device & DataModel to DataPath relationship to track what's actually available given potential per-platform deviations to the modules. Or even OS & Release & Device & DataModel&DataModel... given the deviations are their own modules - many potential representations.

Not parsing per-platform.

TDM parsing currently simply does OS/Release, and does not parse individual platforms. The way we parse the platforms on a per OS/Release basis is from the platform-metadata.json or capabilit*.xml NETCONF Hello messages. This will effect the YANG schema due to deviations, etc.

pyang plugin flatten does not parse machine_id format.

pyang -f flatten currently does not have an option to flatten XPaths to the mangled format of module:prefix:node which machine_id currently expects. This is not a real XPath per se, and whether it will be accepted upstream to pyang is uncertain. We might need to add it to our fork of pyang at cisco-ie/pyang.

e.g. machine_id of /Cisco-IOS-XR-ptp-oper:ptp-oper:ptp/Cisco-IOS-XR-ptp-oper:ptp-oper:interface-packet-counters/Cisco-IOS-XR-ptp-oper:ptp-oper:interface-packet-counter is not a colloquial XPath - only /ptp-oper:ptp/... or /Cisco-IOS-XR-ptp-oper:ptp/... would usually be presented not both semi-coloned.