Validating Radar data files with versioning in CUE

[MikkelHJuul]

ODIM H5 Validator

On my job I was tasked to create software that could validate the internal tree/file structure of hdf5 files following the ODIM H5 specification. The specification describe how metadata is structured inside the given file. So this is basically the perfect fit for CUE.

The project consist of a mapping routine written in python, that map the internal tree to json, the specification written in CUE and a wrapping bash-script.

The CUE implementation turned out to be far from trivial. I started describing the, at that time, newest version: v2.3 of the specification. and then came the other versions, it promptly became clear to me, that describing each version of the specification would become a pain to maintain when new versions arrived.

Versioning

So I ended up building some basic constructs:

_vs: ["V2_0", "V2_1",....]
#supportedVersions: or(_vs)

#VersionEnum: close({
	name:         string
	description?: string
	versions:     [...#supportedVersions] | *_vs
})

#VersionObject: close({
	keys:         _
	description?: string
	versions:     [...#supportedVersions] | *_vs
	deprecated?: [...#supportedVersions]
})

which served as a way to cherry-pick the data based on the objects availability through time (version)
this way structs can be built in runtime using something like

someObj: {
   for o in _someObj if list.Contains(o.versions, _v) {
     o.keys
  }
}
someEnum: or( [ for e in _someEnum if list.Contains(e.versions, _v) { e.name } ] )

now of course this wasn't simple enough for the needs. The most complicated part of the specification is a group called...

The Where group

consider a tree

where?: #TopWhere
[Name=#DataSetName]: where?: #DatasetWhere
[Name=#DataSetName]: [Name=#DataName]: where?: #DataWhere

where the definitions #TopWhere, #DatasetWhere and #DataWhere are composed of mostly the same attributes, that are allowed at different levels of the tree (top, dataset or data) and belong to different types (polar data, vertical data, ...). Ending out with a definition where e.g. #TopWhat: {...loc=top/group=vertical/version=_v..} | { ...loc=top/group=polar/version=v... }

So I had to expand the #VersionObject

_locs: ["top", "dataset", "data"] 
#allowedLocations:                or(_locs)

#VersionObject: close({
	keys:         _
	description?: string
	versions:     [...#supportedVersions] | *_vs
	deprecated?: [...#supportedVersions] 
	locations:                           [...#allowedLocations] | *_locs
	groups: [...string]
})

this way I aggregate and construct the definitions like so:

_whereAggr: {
	for w in _whereObjects if list.Contains(w.versions, _v) for l in w.locations for g in w.groups {
		"\(l)": "\(g)": w.keys
	}
}

#TopWhere:     or([for w in _whereAggr["top"] {close({w})}])
#DatasetWhere: or([for w in _whereAggr["dataset"] {close({w})}])
#DataWhere:    or([for w in _whereAggr["data"] {close({w})}])

Limitation

unfortunately, and somewhat unsatifactory, I ran into a limitation in which cue cannot vet the #TopWhere as one of the groups is a direct subgroup of the other.

forcing me to change:

_whereAggr: "top": "polar": {
   lat: float64
   lon: float 64 
   height: float64
}
_whereAggr: "top": "vertical": {
   lat: float64
   lon: float 64 
   height: float64
   levels: uint
   interval: uint
   minheight: float64
   maxheight: float64
}

to

_whereAggr: "top": "vertical": {
   lat: float64
   lon: float 64 
   height: float64
   levels?: uint
   interval?: uint
   minheight?: float64
   maxheight?: float64
}

All in all I am really glad to have chosen CUE. This would have been a non-starter in JSON schema, and very tedious in any other language. Keep up the good work!

[myitcv]

@MikkelHJuul thanks very much for sharing this.

Just to note, that in:

#VersionEnum: close({
	name:         string
	description?: string
	versions:     [...#supportedVersions] | *_vs
})

the use of the close() builtin is superfluous. From the spec (emphasis my own):

Referencing a definition will recursively close it. That is, a referenced definition will not unify with a struct that would add a field anywhere within the definition that it does not already define or explicitly allow with a pattern constraint or .... Embeddings allow bypassing this check.

[MikkelHJuul]

Yeah I know, first I thought it was needed, I did gloss over it but left it in because I thougth it added to the statement a feeling of being explicitly closed. The same can be said for #VersionObject: close({...})

I will remove it