Current syntactic and semantic redundancies

[cueckoo]

Originally opened by @extemporalgenome in cuelang/cue#699

CUE appears to have a few syntactic forms and concepts that may have had clearer distinctions in the past, but now are somewhat interchangeable.

Equivalence of hidden identifiers

The following appear to all be equivalent in practice, even if there are subtleties or nuances in their particular semantics (examples borrowed from #687):

let byte = "[0-9a-x]{2}"
_byte:     "[0-9a-x]{2}"
_#byte:    "[0-9a-x]{2}"

#hexIP: =~"\(byte).\(_byte).\(_#byte).\(byte)"

These common properties appear to apply to all three "byte" declarations:

• They can be referenced within the current scope and nested scopes.
• They can be used in string interpolations.
• They are not exposed in any formal output/contract.
• They cannot be referenced across package boundaries.

Further, given CUE's proximity to and initial influence from the Go language, let declarations may be the most surprising: although _Byte and _#Byte are intuited as being hidden due to the underscore prefix, let Byte could be mistaken for being visible across package boundaries, even though it is not.

De-shadowing

In the tutorial lesson on aliases, we have: https://cuelang.org/docs/tutorials/tour/references/aliases/

let A = a  // A is an alias for a
a: {
    d: 3
}
b: {
    a: {
        // A provides access to the outer "a" which would
        // otherwise be hidden by the inner one.
        c: A.d
    }
}

This can also be accomplished equivalently with a hidden field:

_a: a
a: {
    d: 3
}
b: {
    a: {
        // _a provides access to the outer "a" which would
        // otherwise be hidden by the inner one.
        c: _a.d
    }
}

Hidden fields vs hidden definitions

Normal definitions (#) are likely to be restricted so that they may not be declared within structs, per #543. In contrast, hidden definitions (_#), like hidden fields, will not be affected by that change, and can continue to be declared within structs.

Normal (exposed) definitions are primarily distinguished from exposed fields in that definitions are not normally output in JSON or other data formats, but can generate schemas, such as in OpenAPI output: this is a meaningful, compelling design.

However, since hidden definitions are not exported as schemas, the only distinction between hidden fields and hidden definitions appear to be their default behaviors regarding struct openness: definitions, including hidden definitions, are closed by default, but can use ... to opt-into being open, while fields, including hidden fields, are open by default, but can use the close builtin to opt-into being closed.

For non-struct constraints, there doesn't appear to be any functional distinction between hidden definitions and hidden fields.

Therefore, aside from the slight syntactical convenience of using a nested hidden definition instead of a nested hidden field when a closed struct is desired, there does not appear to be much benefit to the existence of hidden definitions (we could remove _# from the language and have one less thing in the language to teach). If the convenience of closed nested struct schemas would be sorely missed, we could introduce syntax like --- to close a struct, and in exchange we'd be free to remove the somewhat awkward close builtin (awkward because there are so few builtin functions in CUE, and its use requires parentheses that often wrap multiple lines).

Aliases for referring to non-identifiers

The spec demonstrates that non-let aliases can be used to reference fields that otherwise could not be referenced (https://cuelang.org/docs/references/spec/#aliases):

foo: X  // 4
X="not an identifier": 4

For a reader who doesn't know about this feature of the language, it's particularly difficult to interpret (I ended up asking about it in #687 because I couldn't figure out from the documentation alone what it actually meant).

In this case it introduces a field called "not an identifier" but also gives the right-hand-side an alternate name that can be used elsewhere. Presumably "not an identifier": X=4 would also be equivalent since the field name itself doesn't appear to be captured by the alias, but introducing an alias in value-position is specifically disallowed.

Anyways, this form of alias does not appear to introduce any capability that a hidden field couldn't accomplish equally well, since the following produces the same result:

_x: 4
"not an identifier": _x
foo: _x

or for that matter...

foo: 4
"not an identifier": foo

Since CUE is commutative, there is of course no difference (that I'm aware of) between introducing a concrete value in foo or "not an identifier", though introducing it in foo is much more convenient. Use of _x might be favored in cases where fields foo and "not an identifier" should be de-coupled.

Implications of the above

Hidden definitions (_#), let declarations, and non-template aliases seem to be features that could be removed from the language without losing any functionality, and would simplify the learning and reading of the language as a result. [Y=string] template aliases would certainly still be needed in some form (though potentially [_y:string] could have worked equivalently well if = were not already an established, elegant syntax for this).

let declarations and non-template aliases are of particular note, since both are syntactically quite different from other parts of the language: as far as I can tell, let declarations are the only keyword-prefixed way of introducing an identifier, or more generally the only word-space-word grammar production outside of a preamble or comprehension. Aliases are also relatively unusual in CUE in that they use =, where almost everything else uses :.

I have not spent much time thinking about the use of let declarations within comprehensions, but they do not seem to be critical there either.

[cueckoo]

Original reply by @mpvl in cuelang/cue#699 (comment)

Thanks for the feedback. I'll try to explain the raison d'etre of the various constructs. I'll be curious to hear if that explains the differences sufficiently or if there is still an uneasy feeling of redundancy or suggestions for simplifications, beyond the ones confirmed below.

Aliases

Unlike fields, which always have a :, an alias, never introduces something new and always just introduces a different name to refer to an existing element, whether it be a value or field. Moreover, the only other place where = is used is in let. More on let below. But for now, basically, any use of = never introduces something that influences unification.
Another way to view it, for X = Y for any let or alias, one can always get the same result by substituting expression Y for every reference to X, where Y is assumed to be a valid reference.

Aliases serve various functions that can otherwise not be accomplished. One important one is being able to refer to generated fields, such as:

foo: [X=string]: [Y=string]: name: "\(X)-\(Y)"

or

foo: {
    X="\(expr)": baz
    bar: X.name
}

The construct X=foo: bar is allowed as a consequence of these introductions. It would be weird to allow X="\(y)": z, but not X="y": z. At that point, it would also be strange to not allow X=y: z. More on this in the de-shadowing section.

In general, it is good to not have arbitrary semantic restrictions on what seemingly is allowed by syntax. Most language proposals we have made or are planning to make are mostly just a generalization of which productions are allowed in which place.

Granted, and as you remarked, CUE is currently a bit inconsistent in this regard. For instance,

foo: X={}

and

foo: {
    X=_
}

should be allowed. Both variants solve some important issues that are currently unsolvable. But if nothing else, it would be consistent to allow it.

The reason this doesn't exist yet is partly a function of it just hanging on the to-do list and partly waiting on phasing out past mistakes/usages: what is now let used to be written as just X=expr. This violated the rule that an alias always binds to an existing element. Also, with embedded scalars this old semantics really had an unexpected meaning. We started phasing this out a while back and if I'm not mistaken, the current CUE no longer supports it. It seems proper to wait at least beyond v0.3, though, before reappropriating this construct.

Detailed semantics of aliases

Even if we did have value aliases, they would not be a substitute for field aliases. One semantic subtlety arises from the way graph unification works and for the duality that arises from viewing types and values as the same thing. Suppose we have value aliases and then consider

X=#List: Y={
     a: null | X // same as using #List directly
     b: null | Y
     head: int
}

Using X or Y does not mean the same thing. For instance:

z: #List & {
     head: 1
     a: head: 2 // allowed 
     b: head: 2 // error 1 & 2
}

Clearly, the correct one to use for defining a linked list type is X. But there are cases, especially when referring to a field as data instead of a type context, then Y would be better.

We believe it is possible to come up with some good guidelines for users to figure out which to use when.

The main reason value aliases don't exist yet is because its syntax conflicts with the old meaning of aliases and we need some transition. But also, one would need to consider to allow either a: X=b or a: { X=b }, or both. The first one often looks nicer and intuitive. But the second form solves a case that the first doesn't (reference to root). Even though these are syntactically somewhat different, to a user they may "feel" like the same thing. So it probably makes sense to allow both.

Note that it is not possible to write X="not an identifier": 4 as "not an identifier": X=4: : introduces a new scope X, so it would not be visible at the same level. To allow this anyway would be quite a large language change and it goes against the principle of lexical scoping, which generally adds to readability and maintainability.

Also note that in general, one cannot rewrite

#Foo: {
  foo: X  // 4
  X="not an identifier": int
}

as

#Foo: {
  foo: int
  "not an identifier": foo
}

or

#Foo: {
  _x: int
  "not an identifier": _x
  foo: _x
}

The former would change the meaning of the original, especially when bar is non-concrete value/ constraint.

The latter introduces a new field (_x). Even though this field is not visible in the output, it is part of the struct and interacts with other declarations of _x if merged with foo.

It is also in general not possible to write

#Foo: {
    let X = int
    "not an identifier": X
    foo: X
}

because

a: #Foo: "not an identifier": 2

would yield a: { "not an identifier": 2, foo: 2 } for the original and a: { "not an identifier": 2, foo: int } for its replacement.

As a general rule of thumb, one should always use aliases to work around unaccessible fields.

Disallow aliases that don't add functionality?

I know that above I claimed that it is good to not have arbitrary semantic restrictions on what seemingly is allowed by syntax. I know this contradicts what you are suggesting above. Admittedly this is a trade-off. I remember the discussion in this matter with type aliases in Go. There is something to say to disallow certain uses where people can really shoot themselves in the foot or even avoiding different ways to do the same thing. But this has to be weighed against the possible frustration users may face by doing so.

For instance, consider a user is given a schema with fields of the form someRediculouslyLargeTypName. Being familiar with the form X="foo-bar": int, she now is cleverly considering the use of X=someRediculouslyLargeTypeName: int to avoid repetition. Being surprised by this being (hypothetically) forbidden, she tries X="someRediculouslyLargeTypeName": int. You can imagine the consternation if this is forbidden as well, and even more so if X="foo-bar": int is allowed.

Another reason to not gratuitously block use cases is that a language construct often ends up being used for good uses that are unforeseen by its creator. Blocking constructs may impair this natural evolution.

In this case, there doesn't seem to be a real harm in allowing it, except for being able to solve a few things in different ways, but that doesn't seem to weigh up to the potential frustration and other disadvantages.

De-shadowing

Shadowing can be solved in several ways in CUE, as you noticed. CUE doesn't really have a construct that is just for de-shadowing. The current techniques for de-shadowing are all a consequence of constructs that were designed for another purpose: aliases were introduced initially to refer to computed and other fields that otherwise cannot be referenced. By not restricting their use they can be used for de-shadowing.

But also, quoted fields cannot be referenced. This is a result of finding the simplest rule to explain the inability reference "\(foo)": bar or "foo-bar": baz: rather than explaining when a field in quotes binds to an identifier, we simply just never bind it to an identifier at all.

A third way to de-shadow would be the use of value references (don't exist yet, as explained above), which probably would introduce the most generic solution.

I agree that there should be better documentation explaining recommended techniques for de-shadowing.

Here again some of the design decisions were guided by the ability to do automatic rewrites. One technique for CUE-ifying other sources is to start with everything quotes fields, use references only for builtins and then let cue fmt strip quotes when it doesn't change the semantics. This allows rather complex modifications purely at the AST level. Granted, the tooling around CUE generation (like astutil.Sanatize) has given some alternatives, but this is still an appealingly straightforward technique for automation.

letdeclaration

A let declaration defines an expression for use in another expression. It neither introduces a binding to an existing element (like aliases) nor does it introduce something that can influence unification (like fields): it is never used as a constraint. This is a common meaning for let across programming and even configuration languages

Historically, in CUE, let was foreseen as being very useful in comprehensions. When we realized that the old semantics for X=expr was "unfortunate", we also saw let as a no brainer solution to being able to automatically rewrite that usage. cue fmt and cue fix have been rewriting this construct for a while now. This is why let was implemented as a declaration before it was allowed in comprehensions itself.

The fact that let stands out syntactically different from aliases and fields is intentional. As said, the let construct used to be written as X=foo. This was confusing, as unlike other aliases, it would not bind to something that already existed. So it seemed more like it would be binding to an embedded value, rather than something new. Using let solved this, by marking it clearly as something different from an alias or field. On top of that, it is a common meaning for let in other languages.

Hidden identifiers versus let declarations

So the main advantage for having a let is to allow introducing a reusable expression locally without having to worry about breaking anything. Consider this:

#Foo: {
    _byte:     "[0-9a-x]{2}"
    ip: =~"\(_byte).\(_byte).\(_byte).\_(byte)"
}

How do we know that somewhere else in our configuration someone is not doing something like
#Foo: _byte: "something else" or

foos: [string]: #Foo
foos: bar: {
     _byte: 128 // error as 128 & "[0-9a-x]{2}" doesn't match.
    ip: "\(_byte).\(_byte).\(_byte).\_(byte)"
}

Luckily, as _byte is a hidden field, we only have to look within the scope of our package. But in large installations, where different files within the same package could even be maintained by different people, this can get quite burdensome.

Also, what is hard for humans is often also hard for machines. Suppose we write a tool that automatically rewrites a configuration to hoists repeated strings from a larger string. With the aid of let such automation can be done solely based on the AST of the CUE file and correctness is easily determined. If we only had hidden fields such rewrite would have to potentially rely on a full evaluation, making things much more complicated and harder to prove. Granted, tooling could help with this, but why make things complicated if you don't have to?

Automated rewrites are considered to be an important part of CUE and many of the design decisions have been made to facilitate this. But conversely, like in this example, what makes things easier for machines often also makes it easier for humans, so this is often a good thing.

Aliases versus letdeclarations

I hope that with the above to explanations of aliases and let declarations it should be clear that these serve very different purposes and that there is little to no overlap there.

Hidden identifiers vs let declarations.

The biggest difference between the two is that a hidden field remains after unification (it is a field, after all) where a let doesn't. On top of that, it only lives in the current scope. So:

a: { let X = 2 }
b: a.X // error

gives an error, whereas

a: { _X: 2 }
b: a._X

works. As a general rule, a = never introduces something that can influence the outcome of a unification by itself, whereas the use : does.

The let syntax originated as a concept as a clause for comprehensions in CUE, to allow the assignment of a temporary variable. Experience with GCL taught us that it is frequently handy to hoist expressions. Being able to do so locally without having to inspect other files in a package is quite useful.

Hidden fields versus hidden definitions versus definitions

The main distinction between hidden fields and non-hidden fields is that hidden fields are scoped at the package level whereas non-hidden fields are globally scoped.

The main distinction between definitions and regular field is that definitions recursively closed and are idiomatically used used to indicate the "top" of a certain type, to be further instantiated, whereas regular structs are idiomatically used to make values more concrete.

Hidden definitions were mainly introduced to "complete the matrix". They do serve a purpose, however, if one wants to add a definition within a type imported from a different package. This is especially pertinent when #543 would take effect. But even without, hidden definitions can be used to add a definition to an imported type without having to worry about breakage if the author of the imported package changes things around.

Now, granted, here things get a bit redundant. Instead of writing _#foo: y, one could write _foo: #foo: y and use _foo.#foo as a reference, instead of _#foo. This would be equivalent.

Another possibility is that we introduce a predeclared identifier #closed: {} that one could embed to recursive closed as if there were a definition. This is only a very recent possibility, though, and _# predates this possibility. Note that #closed here is your equivalent of ---, but without the need for additional syntax. Then one could write

_foo: {
    #closed
}

instead of _#foo. One big problem the use of #foo solved was to move definitions into a different namespace. This is likely less needed for hidden definitions and arguably not much of a loss in this case.

On though about the various identifier prefixes early in the design is that one could nicely document all the idiom and syntax of identifier prefixes in a single table, including _, #, _#, $, etc. (the latter is just idiomatic use for indicating a meta field, but has no special meaning to CUE). That only works, granted, if one then actually writes that documentation. :)

But it is not clear to me what is best. If we keep _#, we may do without #closed. If we go with #closed it is another thing to explain that doesn't fit in this table and can itself open another can of worms. If we nix both, then we need to document the _foo: #foo: pattern somewhere. Probably the best course of action is to write the foreseen documentation first and see if that generally clarifies things.

We do plan to deprecate the closed builtin. Firstly, note that closed only closes one level and as such differs from definitions. A planned extension to the language is to allow ...T in structs, which simplifies JSON schema compatibility and is another generalization in the language (it is already allowed for lists). With this, one could simply write closed({}) as {..._|_} or {...error(msg)} (we are considering to phase out _|_, proposal forthcoming).

[cueckoo]

Original reply by @extemporalgenome in cuelang/cue#699 (comment)

Thank you for the clarifications!

I realize my explorations were too shallow, thus I did not encounter the practical binding differences between aliases and hidden fields; and because I was using just a single file, the file-scoping advantages of aliases compared to hidden fields.

[cueckoo]

Original reply by @myitcv in cuelang/cue#699 (comment)

Related to the good discussion in this thread, I've created a tracking issue for "Effective CUE".