Debugging slow/huge Cue inputs

[cueckoo]

Originally opened by @nyarly in cuelang/cue#1074

I've come up against the issue of CUE code taking a long time and lots of memory to process a number of times, and I still haven't really gotten a feel for how to fix this. Most recently, after 20 minutes my cue export got OOMkilled. It's unclear to me what exactly causes the time and space of CUE processing to run away like this, or how to find those causes.

For reference, this is about 100kloc of CUE, that results in about 300kloc of Kubernetes YAML. There's a fair amount of literal inclusion there, from importing upstream raw config - I'd guess about 80%! (MISSING)

That said, the stalling out seems to happen as I refactor to better represent intent. My goal right now is to be able to compose the features required by a particular cluster out of struct-based configurations. Eventually I might also want to remove Kustomize from our workflow, but right now it's serving a valuable purpose patching resources and unifying namespaces. I'm making strides towards mergeable feature configurations, only to be held up by extremely lengthy builds.

As new releases of the tool drop, they've made noticeable differences in build speed. Which is greatly appreciated! My temperament wants to solve its own problems, though. Even if that means contributing PRs up to the toolchain.

(I'm on track to get our code into the private unity program, and I'm excited to see how that pans out.)

[cueckoo]

Original reply by @verdverm in cuelang/cue#1074 (comment)

I have found manually "unrolling" to improve speeds drastically. This has been largely about inlining definitions, but of course means losing reusable components. There are a number of issues tagged with performance and some known optimizations still to be implemented.

The first thing I might recommend w.r.t. PRs is getting to know the evaluator. There are a lot of comments and todos in the codebase as well as background reading that is helpful. The evaluator is a complicated thing and I'd guess @mpvl is the only one who really understands it's implementation right now. One thing that has helped me is running various simple cases with expanded output. The majority of the interesting code is in the internal directory.

[cueckoo]

Original reply by @nyarly in cuelang/cue#1074 (comment)

Unrolling brings up an idea I've been considering for a while: staged CUE compilation. As a way of speeding things up, I'm considering breaking up my project into packages, and then evaluating source packages into "compiled" versions. For instance, I've been using a lot of Kubernetes go imports to get validation at the CUE level - which has been really valuable. But I wonder if I couldn't put all of that into a "schema" package, evaluate it once (or when-changed), and then import the pre-evaluated version. I think I'd need to do something to "hide" the imports I was making, or else they'd be brought into the "DLL" I was using.

[cueckoo]

Original reply by @verdverm in cuelang/cue#1074 (comment)

That is an interesting idea, please share any findings. I'd imagine a small example to prove out the workflow would be useful for everyone.

[cueckoo]

Original reply by @seh in cuelang/cue#1074 (comment)

I tried to use cue def to accomplish what @nyarly suggested, but it doesn't "consume and compact" the imported packages.

[myitcv]

Hello all - apologies for the migration-related delay in replying to this.

It goes without saying that ideally we should not be in a situation where people have to debug slow evaluations like this. That said, it's always a useful tool to have.

I don't have any particular ideas to share, but one thing does come to mind. Next time someone does have a slow public configuration they need to "debug", we could use this as an opportunity to pull together notes on different approaches? We could iterate on those notes in a wiki for example.