Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parsing base #1

Merged
merged 37 commits into from
Nov 29, 2021
Merged

Parsing base #1

merged 37 commits into from
Nov 29, 2021

Conversation

gusfcarvalho
Copy link
Member

@gusfcarvalho gusfcarvalho commented Oct 14, 2021

PR with alpha features of the project.

Working on AWS(auth with Env vars or Env from Secret Keys) and GCP(auth with Service account)

Implements already methods for Vault, AzureKV and IBM Cloud, but they need to be more tested yet.

CLI Examples

kestoeso -i path/to/kes/definition.yaml --to-stdout --kes-namespace=kes --kes-deployment-name=kubernetes-external-secrets

kestoeso -i path/to/kes/ --to-stdout --kes-namespace=kes --kes-deployment-name=my-custom-name

kestoeso -i path/to/kes -o path/to/eso

Testing Locally

To test locally, the first thing that needs to be done is to deploy kes for the given provider. After that, deploy eso.

After that, you can test adderence convertion with this one-liner:

kestoeso -i path/to/kes/ --to-stdout --kes-namespace=kes | kubectl apply --dry-run=server -f  - 

Examples

input

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: aws-secretsmanager
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123412341234:role/let-other-account-access-secrets
  # optional: specify region of the secret
  region: eu-west-1
  data:
    - key: demo-service/credentials
      name: password
      property: password
    - key: demo-service/credentials
      name: username
      property: username
  • kes deployment
Name:                   kubernetes-external-secrets
Namespace:              kes
CreationTimestamp:      Thu, 14 Oct 2021 11:05:45 -0300
Labels:                 app.kubernetes.io/instance=kubernetes-external-secrets
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=kubernetes-external-secrets
                        helm.sh/chart=kubernetes-external-secrets-8.3.0
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: kubernetes-external-secrets
                        meta.helm.sh/release-namespace: kes
Selector:               app.kubernetes.io/instance=kubernetes-external-secrets,app.kubernetes.io/name=kubernetes-external-secrets
Replicas:               0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=kubernetes-external-secrets
                    app.kubernetes.io/name=kubernetes-external-secrets
  Service Account:  kubernetes-external-secrets
  Containers:
   kubernetes-external-secrets:
    Image:      ghcr.io/external-secrets/kubernetes-external-secrets:8.3.0
    Port:       3001/TCP
    Host Port:  0/TCP
    Environment:
      AKEYLESS_API_ENDPOINT:           https://api.akeyless.io
      AWS_REGION:                      us-west-2
      GOOGLE_APPLICATION_CREDENTIALS:  /app/gcp-creds/gcp-creds.json
      LOG_LEVEL:                       info
      LOG_MESSAGE_KEY:                 msg
      METRICS_PORT:                    3001
      POLLER_INTERVAL_MILLISECONDS:    10000
      WATCH_TIMEOUT:                   60000
      AWS_ACCESS_KEY_ID:               <set to the key 'access-key' in secret 'aws-secret'>  Optional: false
      AWS_SECRET_ACCESS_KEY:           <set to the key 'secret' in secret 'aws-secret'>      Optional: false
    Mounts:
      /app/gcp-creds from gcp-creds (ro)
  Volumes:
   gcp-creds:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  gcp-creds
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-external-secrets-94b467b7 (0/0 replicas created)
Events:          <none>

command line:
kestoeso -i examples/aws-secretsmanager.yaml --kes-namespace=kes --stdout

output

apiVersion: external-secrets.io/v1alpha1
kind: ClusterSecretStore
metadata:
  creationTimestamp: null
  name: secretsmanager-secretstore-autogen-xvlbzgba
spec:
  controller: ""
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access-key
            name: aws-secret
            namespace: kes
          secretAccessKeySecretRef:
            key: secret
            name: aws-secret
            namespace: kes
      region: eu-west-1
      role: arn:aws:iam::123412341234:role/let-other-account-access-secrets
      service: SecretsManager
status:
  conditions: null

---
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  creationTimestamp: null
  name: aws-secretsmanager
spec:
  data:
  - remoteRef:
      key: demo-service/credentials
      property: password
    secretKey: password
  - remoteRef:
      key: demo-service/credentials
      property: username
    secretKey: username
  secretStoreRef:
    kind: ClusterSecretStore
    name: secretsmanager-secretstore-autogen-xvlbzgba
  target:
    name: aws-secretsmanager
    template:
      metadata: {}
status:
  refreshTime: null

---

This PR is a draft mainly because It's too big, and because I think some design decisions can still be taken at this time (although it's not supposed to be a long-living tool).

Gustavo Carvalho added 18 commits October 1, 2021 14:15
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
… stores

TODO: Refactor this code

Signed-off-by: Gustavo Carvalho <[email protected]>
* Adds AzureKeyVault migration
* Adds Vault migration
* Adds IBM migration
* Implements TODO list on README file

Signed-off-by: Gustavo Carvalho <[email protected]>
Now provider installs come from KES deployment

Signed-off-by: Gustavo Carvalho <[email protected]>
Implemented validation and some warning messages if things are not 100% smooth
Signed-off-by: Gustavo Carvalho <[email protected]>
Fixed ClusterStore being generated in different namespaces

Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Gustavo Carvalho added 8 commits October 15, 2021 17:22
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
(needed for kubernetes auth with no changes on vault configuration)

Signed-off-by: Gustavo Carvalho <[email protected]>
Added provider specific parsing tests

Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
IBM manually tested.

Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Improving coverage of parseGenerals

Signed-off-by: Gustavo Carvalho <[email protected]>
@gusfcarvalho
Copy link
Member Author

Hey all!

Today I've finished testing with the following providers:

  • IBM
  • GCP
  • AWS
  • Azure
  • Vault

I'm changing this PR from draft to an actual PR. Waiting on feedback.

@gusfcarvalho gusfcarvalho marked this pull request as ready for review October 20, 2021 15:25
Gustavo Carvalho added 8 commits October 21, 2021 09:26
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
Changed log info to be less verbose

Signed-off-by: Gustavo Carvalho <[email protected]>
Added README.md

Signed-off-by: Gustavo Carvalho <[email protected]>
@moolen
Copy link
Member

moolen commented Nov 10, 2021

Hey @gusfcarvalho nice work! I think we should clean up the code a bit before merging it. Could you please:

  1. remove comments of unneeded code (imports, example code)
  2. finish documentation of CLI help

I think then we're good to merge IMO.

@bobbywatson3
Copy link

Looking forward to checking this out!

Gustavo Carvalho added 2 commits November 27, 2021 15:19
Improved CLI
Removed comments

Signed-off-by: Gustavo Carvalho <[email protected]>
Signed-off-by: Gustavo Carvalho <[email protected]>
@gusfcarvalho
Copy link
Member Author

Hey @moolen and @knelasevero ! Sorry for the long delay, got a little bit busy with work.

I also took the chance to improve a little bit the map of templates, in order to allow some templating (the easy ones) to be compatible with the tool.

@knelasevero
Copy link
Member

Gonna merge this one. We can iterate further with more PRs :)

@gusfcarvalho maybe goreleaser makes sense for this project? Then it is easier for us to release and get people to use it. We can open an issue if it makes sense

@knelasevero knelasevero merged commit d97848a into main Nov 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants