CUE at large

[cueckoo]

Originally opened by @stribb in cuelang/cue#817

I'm trying to figure out how best to fit CUE into my company's monorepo, initially for Kubernetes but also to support other outputs and tooling.

Right now, we have a deployment directory that currently contains Kustomize trees. I'd like to transform these into .cue files, but I'm struggling a bit to find out how to modularise things.

While I'm starting out with the basics of CUE: making opinionated specifications for Kubernetes objects, and instantiating them; I'd like a CUE directory to be able to specify more high level applications in a manner akin to the Kubernetes Tutorial.

For now, I need a couple of things:

• A subdirectory of configuration needs to be able to be applyed to k8s, both for real and in dry-run mode
• I need to be able to interpolate arbitrary secrets from Google Secret Manager

Ideally:

• Tools shouldn't be repeated across the repo
• We shouldn't have to maintain a list of config directories.

I'm sure these are possible but there's a documentation gap between the Kubernetes Tutorial and a real-world production example.

[cueckoo]

Original reply by @mpvl in cuelang/cue#817 (comment)

We have been adding some features like conditional includes (via the @if attribute) as part of the injection mechanism as a means of organizing along various orthogonal lines.

We are currently taking a broader look at modules and organization, though. So we would love to hear from users:

• what kind of limitations they are running into today,
• what they consider an ideal organization of their files, and/or
• what properties or invariants they would like such an organization to have.

[cueckoo]

Original reply by @PierreR in cuelang/cue#817 (comment)

FFIW it is not quite clear to me how you would include files in your config. For instance it is not really practical to copy the content of our crt files when building kubernetes secret from them.

[cueckoo]

Original reply by @mpvl in cuelang/cue#817 (comment)

Currently there are two main ways:

1. mark a CUE file as a named package (package name). All files with the same package name in a directory and all ancestor directories up tot he cue.mod root are automatically pulled in by the cue too in a single namespace.
   An @if(foo) attribute placed above the package clause allows files to be pulled in conditionally based on some expression.

2. create a separate package, somewhere with your CUE root (the directory containing cue.mod) and import it in any of the other files.

See https://cuelang.org/docs/concepts/packages/ for an overview.

We are also contemplating allowing embedding of individual files, including JSON or YAML directly. But it doesn't seem that this is what you want in this case.

Would be interested to learn if the above doc gives you some insights or if you are running into other issues.

[cueckoo]

Original reply by @PierreR in cuelang/cue#817 (comment)

@mpvl For the file to be pulled in, it needs to be a cue file, is that correct ? Hence, I need to convert the crt file to a cue file. That's not such a big deal but if this is the case it would be a nice addition if the cue import command would directly support bytes.

[cueckoo]

Original reply by @mpvl in cuelang/cue#817 (comment)

Right now you would have to convert, indeed. With the contemplated embedding option, it could convert on the fly.

With support for bytes, you mean that your crt file is in binary format (not base64)?

[cueckoo]

Original reply by @mpvl in cuelang/cue#817 (comment)

@PierreR Would this help: https://cue-review.googlesource.com/c/cue/+/9570?

[aakarim]

Using Cue as part of the CI/CD pipeline is working well on our end. We generate Deployments, and ConfigMaps to configure the actual services. However, the fundamental problem I'm having with Cue and Kubernetes is Secrets injection. We use native K8s secrets rather than, say, Hashicorp Vault, and we don't want to expose any secrets in the manifest YAML. Our secrets are encrypted at rest. Our ideal situation would be to generate an exported YAML file which will be used by the containers as configuration, including secrets.

We've explored four options:
a) Export from raw Cue files at container start
b) Use initcontainers to generate config files in configuration volumes
c) Use raw Cue files in the services
d) Store the whole exported config file as a secret

To each point:
a) in order to use 'injection', which would be the most straightforward way of passing in secrets at container start, we would need the Cue binary and tooling present in our container images to pass in the config/flags from a volume or environment variable as is the Kubernetes way. This adds image bloat, and also would need to be invoked at runtime as a startup wrapper script. We would need to generate the config file(s) before our app starts, but after the image has been built (we don't want to embed secrets in images). This extra step will probably annoy our developers enough that it would affect adoption in the org. It's just another thing to debug at runtime.

b) This is much the same as a), but instead of using a wrapper startup script, we use an initcontainer with access to volume-mounted secrets and config files to generate the final config YAML. This is a bit neater than a), but would require every single service to have a Cue initContainer that builds config - it's a hassle!

c) To interpret at runtime, we could install Cue as a library on our applications to interpret the raw Cue files, but Cue support isn't great in Python, for instance.

d) This reduces our granularity of configuration, which isn't ideal either. Also doesn't play nice with RBAC for individual secrets.

At the moment our solution is to use Cue for everything but secrets, so as to not add complexity to our workflow. Since 80% of our configuration are secrets for third-party services anyway we're in a bind.

Any suggestions on best practices here? How do we get all the config together at runtime effectively? Is a startup script the only way?

One option we have discussed, but not explored in detail is a move to an API service which can generate these config files on the fly and send them over the wire. We would give up on K8s ConfigMaps & Secrets entirely and manage our own config. This obviously has huge implications so we'd rather exhaust other options first.