User-defined functions?

[cueckoo]

Originally opened by @zombiezen in cuelang/cue#484

Is there a convenient way of creating user-defined functions or something similar? Under boilerplate removal, the pattern constraint is used to imply a default constraint, but sometimes I want to build a value based on some set of input parameters (not just a single string). The best I can come up with now is:

#udf: {
  #foo: string
  out: "\(#foo)bar"
}

"quux": (#udf & {#foo: "foo"}).out

[cueckoo]

Original reply by @zombiezen in cuelang/cue#484 (comment)

From @myitcv in Slack: this is the correct way to go for now, but the new evaluator will make this better by allowing non-structs to be emitted. Combined with type cast semantics (like those I proposed in #454), this will make these sorts of limited functions much easier to work with.

[cueckoo]

Original reply by @mpvl in cuelang/cue#484 (comment)

We are also contemplating a syntactic sugar for the arguments, allowing the above to be written as

#udf: {
  #0: string
  
  "\(#0)bar"
}

"quux": #udf("foo")

#0 would just be a regular field otherwise. This approach avoids some of the pitfalls of having functions in a configuration language.

Note that the new evaluator now explicitly disallows reentrancy. This was already disallowed by the spec, but not previously enforced.

[cueckoo]

Original reply by @mpvl in cuelang/cue#484 (comment)

For completeness, we are also investigating a native bridge to some regular language, perhaps only to be done in the tooling layer.

The idea is that a configuration language is not the best place to do computation and that this is best left to an established general programming language. Rather than enhancing CUE with the ability of computation, we use CUE's commutativity to mix in computed results. The strict separation of computation and configuration, combined with CUE's commutativity property, allows computation to be ignored leaving a non-concrete pure CUE configuration that can be refactored and analyzed with tools like trim and anti-unification. Many of the possible analysis tools would break down if we start to blur the lines between computation and configuration.

Either way, the idea is still to allow simple "macros" to allow dealing with simple cases within CUE itself.

[cueckoo]

Original reply by @myitcv in cuelang/cue#484 (comment)

For those who view this answer, there was further discussion about the syntactic sugar for functions in cuelang/cue#69 (comment)

[cueckoo]

Original reply by @PierreR in cuelang/cue#484 (comment)

Talking about native bridge is webAssembly in the picture to propose an interface for various programming languages ?

[myitcv]

Created an issue for the better syntactic support for user-defined "functions" so that we can reference it more precisely than an aswer in a discussion: #1480

[cueckoo]

Original reply by @myitcv in cuelang/cue#484 (comment)

The current best answer is:

foo: {#a: int, #b: int, #a + #b } // struct with embedded scalar
call: foo&{_, #a: 1, #b: 2}         // 3

(a change from the previously marked answer because we now have full support for embedded scalars)

There is a proposal to support a syntactic sugar based on this:

foo: { #a: int, #b: int, #a + #b } // struct with embedded scalar
call: foo(#a: 1, #b: 2) // 3

[cueckoo]

Original reply by @codewithcheese in cuelang/cue#484 (comment)

@myitcv with cue eval --concrete the above returns foo: non-concrete value int in operand to +: ./demo/func.cue:25:25

[cueckoo]

Original reply by @myitcv in cuelang/cue#484 (comment)

@codewithcheese please can you raise an issue if you're seeing an issue with this? Thanks

[cueckoo]

Original reply by @morlay in cuelang/cue#484 (comment)

@myitcv your case will be throw the errors. (v0.3.0-beta.7)

@codewithcheese function should be a hidden one _ or # prefix

_foo: {#a: int, #b: int, #a + #b }
call: _foo & {_, #a: 1, #b: 2} 

[cueckoo]

Original reply by @myitcv in cuelang/cue#484 (comment)

@morlay

@myitcv your case will be throw the errors. (v0.3.0-beta.7)

Apologies, I'm not sure what you're referring to here. Are you suggesting there's an issue with beta.7? If so, please can you give an example of what is failing?

[cueckoo]

Original reply by @vikstrous2 in cuelang/cue#484 (comment)

Taking this idea a step further and adding a conditional:

cond: true
foo: {
  #a: int
  #b: int
  if cond  { output: #a + #b }
  if !cond { output: 4 }
} // struct with embedded scalar
call: (foo&{#a: 1, #b: 2}).output         // 3

The conditional version doesn't work with the embedding scalars approach.

[cueckoo]

Original reply by @myitcv in cuelang/cue#484 (comment)

It should work with embedded scalars. So I just raised cuelang/cue#729. Thanks for pointing this out.

[steeling]

@myitcv is it possible to define functions in go, then import them into cue? I see the builtin-libs use internal.Package to generate the cue code. It would be pretty sweet to allow me to write a custom function in go, and use reflect to generate the related cue code.

In the interim I'd be satisfied with even just annotating code in go, or if you made the Package construct external. Not sure if this would break any assumptions on the cue runtime, but this could open up making network calls, or other interesting use cases.

[steeling]

Thanks for the detailed reply! You make some great points. I'll add some counter arguments to some of them below :)

@mpvl is the creator and expert in such topics. I defer any absolutisms.

personal opinions...

• CUE is an open-source data validation language and inference engine with its roots in logic programming. [...] the language is not a general-purpose programming language [...] (https://cuelang.org/docs/about/) this is an important framing reference
• I think the hermetic nature is good and is there is for a reason for it, based on experience. (again deferring to @mpvl) I understand some limited and unknown amount of the theory and philosophy. Still, the foundations and philosophy are what attract me to CUE.
• I'm not sure how one would detect non-hermetic code, this seems like it belongs in the same class as determining if a program will terminate.

Detecting the right subset of syscalls would be one way.

• I've learned the customer is not always right. Steve Jobs is famous for this w.r.t. the iPhone touchscreen, but there are many more cases. Just because users ask for something doesn't mean it should be added. We need to flesh out what it is they want to achieve and show them how to do so given the features and constraints. This is the main goal of my https://cuetorials.com and also something I am still learning to do. Documentation is the key here. While we've tried to capture common patterns in GitHub Discussions, they need to be further refined and captured on the websites.

While I agree the customer is always right, I still adhere to the mantra of providing the happy path, but also providing tools to forge other paths. The cool use cases come from power users who want to do novel things, and that can really help grow any system.

• ETL has been a use case thought about during CUE development. I think much of what you want to do can already be done. The E & L interact with the outside world and will need to be handled in the non-hermetic areas. T is really the interesting part that pure CUE addresses quite nicely.

There are a few issues here. (I'll plug #1232 which is frustrating to deal with), but also namely database generated values are difficult to deal with in cue. ie: setting foreign keys on other tables with auto generated id's.

• I believe I was the first person to ask for user defined functions. I have changed my opinion on this, initially based on @shykes's fragmented ecosystem argument and later in that it breaks the hermetic guarantee. The second reason is more resilient. The majority of those conversations is likely lost to slack history, though I may have some in some backups I made [and haven't looked at since ;]
• CUE is still young. It has bugs and todos
• Knowing how data and configuration are a part of every program, and that CUE is a language for this, but also that programs have domain specifics. some border or tradeoff point is needed. CUE, or any language, cannot be everything to everyone.
• I suspect that code gen will face this same "everyone / everything" issue. Consider that CUE lacks many of the concepts found in OOP. How would we generate types in these languages? Also notably, there isn't a native way to express relationships. This would apply to your DB example, but also the language types they marshal into. I believe the answers are tied to Attributes vs Required (opinionated) Struct #1229 among others. There are ecosystem considerations as well. Assuming the definitions or schemas are community created, we could see multiple modules for DB. It seems unlikely that a single schema could apply to all, as there are opposing concepts OOP vs the Go model.
• Note that CUE the language is logical and hermetic. The tooling layer is implemented in the cue binary as a wrapper around the language.
• I don't personally use the tooling layer. I find it too verbose for scripting and continue to use bash, python, and Make. For more complicated or performance oriented needs, I use the API and build a Go program.

With some amount of jest and seriousness, maybe we should have a CUE2 label?

jsonnet has functions. It enhance the language with functions that are written in a syntax of CUE. ie: adding keywords func and return would clean up that syntax.

One last point: Above I mentioned what I wanted to do would be possible in a binary. After a bit more investigation I realized that's not true. At least not simply by leveraging the cue go packages. I'd essentially have to fork and rewrite CUE on my own :/

Happy to provide more concrete examples on what exactly I found I can't do with CUE

[verdverm]

CUE is intentionally turing incomplete, with a very limited form of recursion and thus cannot represent functions. Sounds like you are looking for a general purpose language, which CUE is not.

Happy to provide more concrete examples on what exactly I found I can't do with CUE

Alternatively, could you provide an input/output example of what you want to achieve irrespective of language? Opening a new discussion would be helpful, maybe something around working with database relations? Outlining the original problem in detail will be helpful. Then we can help show how to do this with CUE & (tool | Go).

[steeling]

CUE is intentionally turing incomplete, with a very limited form of recursion and thus cannot represent functions. Sounds like you are looking for a general purpose language, which CUE is not.

To be fair,

func AsSet(in string) [string] {
  var: seen: {
		for v in #In {
			"\(v)": _
		}
  return list.SortStrings([ for k, _ in var.seen {k}])
}

AsSet(v.#SomeStrings)

Would simply by syntactic sugar for:

_#AsSet: {
	#In: [...string]
	_#Seen: {
		for v in #In {
			"\(v)": _
		}
	}

	// NB: I'd prefer to just use an embedded scalar here (as opposed to a named field), but CUE
	// version 0.4.0 panics in some cases when consuming the resulting value.
	out: list.SortStrings([ for k, _ in _#Seen {k}])
}

(_#AsSet & {#In: v.#SomeStrings}).out

That doesn't impact what the language is capable of, it's just syntactic sugar.

Happy to provide more concrete examples on what exactly I found I can't do with CUE

Alternatively, could you provide an input/output example of what you want to achieve irrespective of language? Opening a new discussion would be helpful, maybe something around working with database relations? Outlining the original problem in detail will be helpful. Then we can help show how to do this with CUE & (tool | Go).

Continuuing in #1233, thanks!

[verdverm]

@steeling not sure how the "to be fair..." relates to the turing incompleteness you quoted. The syntactic sugar for "transforms" or "functions" is unrelated to my comment and written by another user.

My comment was intentioned to highlight some of CUE's core principles, and that are unlikely to align with the features you are looking for in the language. Introducing proper functions and non-hermetic operations would be a fundamental change in CUE's philosophy, which is in part based on the problems in having functions (turing completeness) and hermetic operations (inability to reproduce because the outside world changed).

[myitcv]

@seh just to confirm the syntactic sugar being proposed for "function calls" is described in #69 (comment).

@steeling similarly, just want to ensure you are aware of the proposed syntax I linked to @seh above. Also just to make sure I'm understanding things correctly, are you clear that the tooling layer (cue cmd as @verdverm linked above) does not work for you here? I'll take a look at #1233 to see if that clarifies things for me.