Add credentials config that supports custom keys

[earmenda]

Closes #606
Closes #607

Code Changes

• Create new cli options for okta, aws. CLI commands now support credentials as options, fides config reference, and environment variables.
• Add helper functions to share code that validates and fetches credentials from options vs config. Not sure if click has some better ways to deal with options like this. see cli.utils.py
• Implement our own merging of environment variables that are prefixed with FIDESCTL__CREDENTIALS__. Does not use pydantic features because it doesn't work well with custom fields. see credentials_settings.py
• Created models DatabaseCredentials/OktaCredentials/AwsCredentials which are only used for validation when configs are retrieved. see credentials_settings.py
• Update all functions in okta/aws code paths to allow passing in credentials directly from cli.

Steps to Confirm

• Manually tested [scan/generate] x [db/aws/okta] with options, environment variables, and config values
• Added cli integration tests which test the usage of new options and usage of credentials from config
• Added test cases for merging of environment variable config
• Added test cases for handling new options and having multiple ways to pass credentials now

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation Updated:
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

Description Of Changes

First attempt was a little too complicated and got some help from @ThomasLaPiana to brainstorm a new solution. Will be closing #673 in favor of this PR. This PR should be simpler but it still took alot of effort to flesh out.

With this PR we now support sourcing credentials from command options or fides config reference for db/aws/okta commands.

Sample Usage/Output

Success with connection string option

Database generate using connection string

fidesctl generate dataset db dataset.yml --connection-string "redshift+psycopg2://fidesctl:$REDSHIFT_FIDESCTL_PASSWORD@redshift-cluster-1.cohs2e5eq2e4.us-east-1.redshift.amazonaws.com:5439/fidesctl_test"

Generated dataset manifest written to dataset.yml

Database generate using credentials-id option from env

export FIDESCTL__CREDENTIALS__MY_REDSHIFT__CONNECTION_STRING="redshift+psycopg2://fidesctl:$REDSHIFT_FIDESCTL_PASSWORD@redshift-cluster-1.cohs2e5eq2e4.us-east-1.redshift.amazonaws.com:5439/fidesctl_test"
fidesctl generate dataset db dataset.yml --credentials-id "my_redshift"

Generated dataset manifest written to dataset.yml

Database generate using credentials-id option from config

~inside fides.toml~
[credentials]
redshift_toml_1 = {connection_string="redshift+psycopg2://fidesctl:.wm!pV2kMc4ViTYE@redshift-cluster-1.cohs2e5eq2e4.us-east-1.redshift.amazonaws.com:5439/fidesctl_test"}

fidesctl generate dataset db dataset.yml --credentials-id "redshift_toml_1"

Generated dataset manifest written to dataset.yml

AWS generate using new secret_access_key/access_key_id options

fidesctl generate system aws manifest.yml --access_key_id $AWS_ACCESS_KEY_ID --secret_access_key $AWS_SECRET_ACCESS_KEY --region $AWS_DEFAULT_REGION```

Generated system manifest written to manifest.yml

AWS generate using credentials-id option from env

export FIDESCTL__CREDENTIALS__AWS_1__AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export FIDESCTL__CREDENTIALS__AWS_1__AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
export FIDESCTL__CREDENTIALS__AWS_1__REGION_NAME="us-east-1"
fidesctl generate system aws manifest.yml --credentials-id aws_1

Generated system manifest written to manifest.yml

AWS generate using credentials-id option from config

~inside fides.toml~
[credentials]
aws_1 = {aws_access_key_id="redacted", aws_secret_access_key="redacted", region_name="us-east-1"}

fidesctl generate system aws manifest.yml --credentials-id aws_1

Generated system manifest written to manifest.yml

[SteveDMurphy]

Reading through the PR and changes it all looks really good so far @earmenda 🙌🏽 love how you are solving for being unable to use env vars within a .toml and thinking that could be a big benefit for users. Great test coverage added as well. Planning to dive in and use it some more but I don't see anything that should hold this up

[SteveDMurphy]

@earmenda this looks really good to me! Focused some testing around defining a credential ID for aws and populating it partially with env vars and partially in fidsctl.toml and triggered some helpful Client error messages from boto 👍🏽

I do think it might be worthwhile to have a follow on issue (or add in here) to provide either further information around how to use env vars with credential IDs in configuration.md or a user guide for some of it to help users avoid any potential pitfalls

[earmenda]

@earmenda this looks really good to me! Focused some testing around defining a credential ID for aws and populating it partially with env vars and partially in fidsctl.toml and triggered some helpful Client error messages from boto 👍🏽

I do think it might be worthwhile to have a follow on issue (or add in here) to provide either further information around how to use env vars with credential IDs in configuration.md or a user guide for some of it to help users avoid any potential pitfalls

@SteveDMurphy ahh I hadn't thought of making changes to configuration.md. It would be worthwhile to add a section in there regarding credentials section. Will add.

Thanks for the review :)

[conceptualshark]

Just some typos/wording cleanup - thanks for attention to the docs! Also agreed that if this changes recommendations to configuration.md, noting it there would be great - if you end up breaking out another issue I can pick that up.

[ThomasLaPiana]

two small nits

[ThomasLaPiana]

@earmenda thanks for this! digging into it, looks great so far. Reorganizing the tests a bit but overall the logic looks excellent

[earmenda]

@conceptualshark @ThomasLaPiana @SteveDMurphy Thanks for all the help in this PR. Just posted latest changes, this should be ready to be merged