secret block in jobspec

[tgross]

The template block is a powerful tool for writing arbitrary configuration files to disk or creating environment variables, and can read from Vault, Consul, or Nomad APIs. But we suspect that for many users, the template block is far more than they need and it's being used for cases like the following:

template {
  data = <<EOH
LOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"
API_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}"
EOH

  destination = "secrets/file.env"
  env         = true
}

In this example we're getting secrets from Vault and exposing them as environment variables to the task. Which works, but we could provide a more minimal version of this for many workloads. So we're proposing a secret block.

An example of what the secret block could look like is as follows:

task "task" { # or at group scope?

  secret {
    provider = "vault"
    path     = "secret/foo/bar"
    target   = "MY_DATABASE_SECRET"
    env      = true
    file     = true
  }

}

It identifies a provider (probably just Vault and Nomad Variables as options?), a path to get the value from, a target variable to expose, and whether to write that to file and/or disk (likely defaulting to env=true).

Because this is "native" to Nomad and not being driven by templates at the task level, we could potentially use these same secrets for interpolation within other client hooks. For example, we could expose these to artifact blocks:

task "task" {

  secret {
    provider = "nomad"
    path     = "job/example/web/httpd"
    target   = "SSH_PRIVATE_KEY"
  }

  artifact {
    source      = "git::[email protected]:example/nomad-examples"
    destination = "local/repo"
    options {
      sshkey = "${secret.SSH_PRIVATE_KEY}"
    }
  }

}

Example features this would enable and/or replace:

• CSI: support getting Secrets from Variables or Vault #7978
• [Feature] secrets in artifact stanza #3854
• Allow Vault secrets to be used inside the env stanza #15332
• Docker registry auth from Vault or Variables #9740
• HCL2: Fetch data from consul, vault, etc #9434
• Using vault values in health checks #7979
• ECR credentials for AWS using Vault #7832
• Provide vault tokens to nomad to use for docker-credential helper #4576

Most of the work we'd need to support Vault here would want to wait until we've completed #15617 which is planned for Nomad 1.7.0, otherwise there'd be a bunch more rework to do after the fact. But I wanted to open this issue and gather feedback from the community on this very rough design.

[apollo13]

But we suspect that for many users, the template block is far more than they need and it's being used for cases like the following [...]

Lol @tgross, you are not creative enough. This is what I do:

      template {
        data = <<EOH
{{- with nomadVar "nomad/jobs/netbox" }}
{{- range .Tuples }}
{{ .K }}={{ .V }}
{{- end }}
{{- end }}
EOH

        destination = "secrets/file.env"
        env         = true
      }

and guess what, the variable looks like this:

{
  "DATABASE_URL": "....",
  "AWS_API_TOKEN": "...",
   ...
}

An example of what the secret block could look like is as follows: [...]

I fear that this would become verbose quite fast for more than one variable. I would love to see some shortcuts like:

secrets { # note the plural
   provider = "nomad" # probably defaults to nomad
   job = true/false
   group = true/false
   task = true/false
   # or if the booleans above are not enough
  path = "some/other/path"
}

which would fetch all of the relevant variables (from job/group/task) and would put them into env or with file = secrets.bla put them into a file. The same could be done for vault by assuming a similar structure.

Note that I am not suggesting that there shouldn't be a way to fetch single values, but fetching them all would be great :)

Because this is "native" to Nomad and not being driven by templates at the task level, we could potentially use these same secrets for interpolation within other client hooks.

Yes please!

Most of the work we'd need to support Vault here would want to wait ...

I guess that would be okay, we could experiment with nomad variables first and then add vault support (I guess testing for the first is easier as well :))

[mwild1]

Just worth noting that, although we've all done it, environment variables are a bad place to put secrets.

There are conventions for using the filesystem, with a secret per file. This is baked into some application configuration mechanisms, e.g. Python's environ-config supports this for Docker/K8s compatibility.

Yet this style of secret-passing is, as far as I can tell, something that is quite tricky to achieve with the existing template stuff and your original proposal here (i.e. a verbose block for every secret, even if only the name differs between a collection of secrets).

[tgross]

Yet this style of secret-passing is, as far as I can tell, something that is quite tricky to achieve with the existing template stuff and your original proposal here (i.e. a verbose block for every secret, even if only the name differs between a collection of secrets).

The file = true + env = false options would be the way to achieve this. That'll result in the secret getting written to the Nomad secrets dir, which is an in-memory filesystem (on supported platforms like Linux).

The verbosity is a bit of a challenge and something we'll want to discuss in design. The most simple possible design would probably be something like:

secrets {
  # map of source to target
  "secret/foo/bar"                     = "MY_DATABASE_SECRET"
  "nomad/jobs/example/web/httpd.item0" = "MY_TLS_CERT"
  "nomad/jobs/example/web/httpd.item1" = "MY_TLS_KEY"
}

But then there's no good way to configure all the different kinds of options user will want for how secrets get rendered, which provider to use, etc. So we could take the list of sources and targets and put that in an inner block with the config values for provider, env vs file, etc. as field on the top-level secrets block. And then if you want to have multiple secrets blocks with different config you could do that.

secrets {
  provider = "nomad" # default provider, can be omitted
  env      = true    # default value, can be omitted
  file     = true    # default value, can be omitted

  # not a great name =)
  paths {
    # map of source to target
    "secret/foo/bar"                     = "MY_DATABASE_SECRET"
    "nomad/jobs/example/web/httpd.item0" = "MY_TLS_CERT"
    "nomad/jobs/example/web/httpd.item1" = "MY_TLS_KEY"
  }
}

[the-maldridge]

I just looked through my jobspecs and over 80% of my template usage is the way that @tgross shows in the first post of this thread (though I will be converting most of it to use @apollo13's far more elegant approach shortly). I think if you could support either the flavor where you say "everything under this key is a key/value tuple" like the example showed, that would make this really powerful. It would also provide a clear secrets schema that allows other tools to integrate by understanding that nomad expects to pull out key/value blocks.

One thing I'd like to know is how this interacts with secret lifetimes when the backend is vault. Does nomad still spin up a worker to handle renewals in that case?

[ddaws]

This proposal looks like it's specific to accessing secrets from the Vault KV store. This is a common use-case but how would this proposal handle support for things like the DB or PKI secrets engines in Vault?

For example, Vault is very good at managing PKI and requires a write request to a pki/issue/role-name endpoint to issue new certs. This returned the cert, private key, and issuing CA in the response. How would you handle something like

template {
  destination = "/secrets/cert.pem"
  data        = <<-EOT
{{ with secret "pki/issue/role-name" common_name="foobar.service.consul" }}
{{ .Data.certificate }}
{{ .Data.private_key }}
{{ end }}
EOT
}

How are the resulting files named when env = false? How are name collisions handled?

How does this proposal interact with Consul KV? Or should it not? I think there is a similar argument to be made for retrieving configuration from Consul KV and if this proposal doesn't cover this use case would you propose additional syntax for supporting similar Consul KV access in the future?

Have you tried Nomad Pack, or some other templating tool to template your Nomad specs? It may make more sense to implement helpers there instead of directly in the Nomad spec syntax to avoid complicating specs.

[ddaws]

In the future Hashicorp may decide to add support to Nomad for other secrets management solution (for example, AWS Secrets Manager) to ease adoption of Nomad in organizations that are already using a different secrets management solution. How would this solution support retrieving values from other secrets backends?