External Secret Store Integration e.g. AWS Secrets Manager/ECS, Gitlab CI/CD

[joshuakarp]

Specification

External secret stores are stateful systems that often store secrets that are intended for to be used DevOps work. Integrating into external secret stores prevents Secret Sprawl that leads to Configuration Divergence. Unifying secret configuration means there's one place to look at, edit and understand the provenance of secrets.

For example to use AWS ECS and to put in secret keys in during deployment we use environment variables. However sometimes these keys should not be exposed to the infrastructure team. So AWS allows one to get secret variables as paths to the AWS SSM. So if a user were to be using AWS ECS/SSM then where does polykey fit into that picture?

Consider the usecase of using Gitlab CI/CD or GitHub Actions. These systems also require secrets to be utilised to perform operations. Here's one example of all the environment variable configuration necessary for running CI/CD pipeline for our opensource code:

The problem of integrating External Secret Stores is a matter of Push vs Pull configuration. Note that has a relationship with push vs pull messaging systems, but the factors that are of concern are different in configuration management.

• https://serverfault.com/questions/568187/configuration-management-push-versus-pull-based-topology
• http://agiletesting.blogspot.com/2010/03/automated-deployment-systems-push-vs.html

To summarise:

• Push systems requires a centralised system to be aware of the entire infrastructure
• Push systems are easier to setup, they are synchronous and are simple
• Pull systems are more scalable and dynamic compared to push based systems
• Pull systems are more difficult to setup, they are asynchronous and have some complex overhead

With respect to external secret stores, one can integrate them in push or pull style. Which style depends on the user's preference and situational constraints.

With respect to PK, a push integration would mean these possibilities:

• A higher level configuration system/script that orchestrates PK and pushes secrets to a target system - e.g. shell piepline composing pk with gh CLI for GitHub Actions
  • gh - https://cli.github.com/manual/gh_secret
  • glab - Support creating project variables profclems/glab#515
• A plugin in PK that supports last-mile deployment of secrets to a target system - e.g. vault commits trigger automatic updates to secrets stored in GitLab projects via HTTP requests
  • example API in gitlab https://docs.gitlab.com/ee/api/project_level_variables.html

A pull integration would mean these possibilities:

• Inside the target system, they would call into the PK API by means of a PK client either CLI or directly via HTTP or GRPC API Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166, and pull down relevant secrets to use
  • secret zero is still needed to authenticate the target system to pull from PK
  • secret references are required for the target system to know what to pull from PK
• A plugin in the target system that understands PK as first-class system and pulls relevant secrets

Both situations can have plugins, but that is most expensive option as each target system will require development of a plugin. Not all systems can support plugins, or it can be difficult to know how to develop one. AWS ECS/SSM may require AWS lambda... etc.

The most realisable situation is to use higher level orchestration for push-integration, or using PK client for pull-integration. This can be demonstrated in our wikis #2.

Additional context

• Integration with OS Keychains #201 - OS keychains is a desktop/mobile local form of external secret stores, they are likely to require push-based integration and even plugins...
• File Descriptor Interface #170 - file descriptors will be needed for some forms of push-integration
• Pseudo file systems interface #169
• We had some plans to implement plugin architecture for IdP integration Add in Twitter as an Identity Provider #157 and Add in LinkedIn as an Identity Provider #156, the same architecture can be used for this, to allow arbitrary code, we might make use of webassembly which may overlap with React Native and Mobile OS (iOS and Android) Compatibility #155

Tasks

1. - Demonstrate a push-integration with Github actions and the gh command line
2. - Demonstrate a pull-integration with GitLab CI/CD and the usage of pk CLI
3. - Identify any relevant integrations that should be plugins, or spec out the plugin architecture
4. - ...

[CMCDragonkai]

For AWS Secrets Manager, an example of non-plugin push integration would be scripting pk with the aws CLI. https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/index.html#cli-aws-secretsmanager

Also see the difference between parameter store and secrets manager here: https://www.1strategy.com/blog/2019/02/28/aws-parameter-store-vs-aws-secrets-manager/

Any work on AWS should be applied to a demo experiment involving:

• ECS
• EC2
• AKS
• IAM - iam is interesting since it's about controlling user access or API access

[CMCDragonkai]

To avoid the cost of writing push-based plugins, one can provide recipes for scripts that automate the usage of pk and third party CLI applications.

Writing plugins that use PK's API and AWS's (or any external) API is lot more difficult compared to just scripting usage of the CLI commands in a shell context. API integrations are likely to be more performant and more flexible, but they are likely to be a lot more expensive to develop, and more complex to develop. API integrations are code that has to be executed in a particular lower-level runtime introducing plugin integration complexity and potential security vulnerabilities, it could work when WASM becomes more available and robust though.

It's possible to make these integrations a little easier by providing ways for PK to execute scripts when we have "automated" policies for secrets management workflows. This would mean users could program a secret management workflow into PK to have it run automatically on event triggers. Then PK would just end up calling external programs as dictated by the user. That might be the most cost-effective solution atm.

[CMCDragonkai]

Note a pull integration that exists in Gitlab CI/CD using hashicorp vault: https://docs.gitlab.com/ee/ci/yaml/index.html#secretsvault. This is provided by Gitlab CI/CD we could extend gitlab with our own service later.

[CMCDragonkai]

Plugin Design (Framework vs Library Control Flow)

Regarding plugin design, there's few ways we can design it:

1. Using standard streams similar to https://github.com/FiloSottile/age/releases/tag/v1.1.0-rc.1 (which now expect all plugins to take STDIN and return STDOUT), this allows plugins to be written in any language and just need to take in some input and return some output, however it limits the plugin's expressivity, since it must be placed in hooks in PK where this makes sense
2. Expect plugins to compile to WASM, and load them as WASM plugins, the WASM code can be given an "object", that object is a controlled access to the PK system, and the plugin can then do whatever they want

In the first case, PK is the framework that calls into the plugins.

In the second case, PK is the library that the plugin calls into.

Note that main difference between a library and a framework is that your program calls the library, while the framework calls your program.

You can see that the first case is more limited (at least for the plugin), as the framework dictates the flow of control. This is actually called inversion of control. And it ultimately depends on well abstracted design with many extension points and hook points that plugins can slot into. The plugins "fill in the blanks" with custom functionality.

In the second case, the plugin is the framework, and it can use PK like a library. In this sense, one may not really call it a plugin, since one could use PK as an embedded library. There's also licensing issues here, because the plugin depends on PK, the plugin is now GPL. In the first case, that does not occur.

The usage of wasm gives some sort of middle ground. The plugin still needs to be hooked in somewhere (predefined by the framework extension points), but it's not limited to only STDIN data and STDOUT, it can perform side-effects against PK.

It makes sense that if we wanted to "market" plugins, they need to be easy to develop, which means they have to be simple architecture, and not be infected by the GPL license, but also have some flexibility so it's not just functional input to output.

1. Licensing issues - plugins should not be infected by GPL
2. IoC - PK framework should be calling the plugin (and dictating control flow), this keeps the architecture simple, if it was the other way around, you're just using PK as a library
3. Extension points - Need to design relevant extension points where plugins can be called in a IoC manner
4. Not just functional input to output, can do more, meaning must give it an object "pk" that it can call on
5. WASM - no language limitations, but also limited environment for security