Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix profile yml error handling #820

Merged
merged 7 commits into from
Jul 5, 2018
Merged

Conversation

beckjake
Copy link
Contributor

@beckjake beckjake commented Jul 3, 2018

Fix #816 and add tests. The issue is that when I migrated to the JSON schema validator I tried to preserve too much of the old without really understanding what I was doing. I think the right thing is to move some of the logic into the ValidationException, store the intermediate steps on that object, and do all this regex unpleasantness there to figure out what kind of validation issues we had so we can generate a nice message for the user.

The new validator also seems to sometimes repeat the same errors multiple times so I stripped duplicates.

The new error looks like this (two extra keys, two missing keys):

Encountered an error while reading the project:
Runtime Error
  Invalid arguments passed to "CredentialsValidator" instance: 'schema' is a required property, 'dbname' is a required property, Additional properties are not allowed ('morejunk', 'junk' were unexpected)
Defined profiles:
 - postgres

For more information on configuring profiles, please consult the dbt docs:

https://docs.getdbt.com/docs/configure-your-profile

Encountered an error:
Could not run dbt

Is this obvious enough? Should we somehow try to make it more clear at this point that it's the profile that's probably wrong and not the dbt_project.yml file? The only line that we really can control without this turning into something huge is the Extra project configuration value(s)... line. Is Extra profile configuration value(s)... better? Or even Extra connection configuration value(s) in profile ...?

@beckjake beckjake requested a review from cmcarthur July 3, 2018 19:12
Copy link
Member

@cmcarthur cmcarthur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks generally ok, but i'm a little confused by the changes to dbt/exceptions.py. it looks like your changes to api/object.py, coupled with more straightforward rendering of the exception in dbt/project.py, should address the entire issue, right? it looks like you are trying to add another layer of user-friendliness here in dbt/exceptions.py, but i'm not sure that it hits the mark.

let me know what you think. my suggestion is to get rid of that intermediate error message manipulation code in dbt/exceptions.py and then this will be approved

EXTRA_PROPERTIES = re.compile(
r'Additional properties are not allowed \(([^)]*) (were|was) unexpected\)'
)
MISSING_PROPERTIES = re.compile(r'(.*) is a required property')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these messages returned by the json schema validator? is there an api where we can get structured data from the validation instead of strings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sort of. You can get that there are extra properties or that there are missing properties, but the names of those properties are only exposed in the message, which is why I went through all these ugly machinations.

self.causes = _causes_from_errors(errors)
msg = ('Invalid arguments passed to "{}" instance: {}'
).format(validator_name, self.causes)
super(ValidationException, self).__init__(msg, node)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like the idea of accepting structured data here. have you checked everywhere in the code that uses ValidationException to make sure this is compatible?

this ValidationException is generic across all of dbt -- it applies to other things, like models and schema.yml, etc. so i don't think this is the right place to wire in _causes_from_errors that refers specifically to project configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I misunderstood ValidatorException (the validator library) vs ValidationException (dbt). I'll move this stuff out into the APIObject or something

@beckjake beckjake merged commit 8aba382 into development Jul 5, 2018
@beckjake beckjake deleted the fix-profile-yml-error-handling branch July 5, 2018 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants