-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix profile yml error handling #820
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks generally ok, but i'm a little confused by the changes to dbt/exceptions.py. it looks like your changes to api/object.py, coupled with more straightforward rendering of the exception in dbt/project.py, should address the entire issue, right? it looks like you are trying to add another layer of user-friendliness here in dbt/exceptions.py, but i'm not sure that it hits the mark.
let me know what you think. my suggestion is to get rid of that intermediate error message manipulation code in dbt/exceptions.py and then this will be approved
dbt/exceptions.py
Outdated
EXTRA_PROPERTIES = re.compile( | ||
r'Additional properties are not allowed \(([^)]*) (were|was) unexpected\)' | ||
) | ||
MISSING_PROPERTIES = re.compile(r'(.*) is a required property') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are these messages returned by the json schema validator? is there an api where we can get structured data from the validation instead of strings?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sort of. You can get that there are extra properties or that there are missing properties, but the names of those properties are only exposed in the message, which is why I went through all these ugly machinations.
dbt/exceptions.py
Outdated
self.causes = _causes_from_errors(errors) | ||
msg = ('Invalid arguments passed to "{}" instance: {}' | ||
).format(validator_name, self.causes) | ||
super(ValidationException, self).__init__(msg, node) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i like the idea of accepting structured data here. have you checked everywhere in the code that uses ValidationException to make sure this is compatible?
this ValidationException is generic across all of dbt -- it applies to other things, like models and schema.yml, etc. so i don't think this is the right place to wire in _causes_from_errors that refers specifically to project configuration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I misunderstood ValidatorException (the validator library) vs ValidationException (dbt). I'll move this stuff out into the APIObject or something
Fix #816 and add tests. The issue is that when I migrated to the JSON schema validator I tried to preserve too much of the old without really understanding what I was doing. I think the right thing is to move some of the logic into the ValidationException, store the intermediate steps on that object, and do all this regex unpleasantness there to figure out what kind of validation issues we had so we can generate a nice message for the user.
The new validator also seems to sometimes repeat the same errors multiple times so I stripped duplicates.
The new error looks like this (two extra keys, two missing keys):
Is this obvious enough? Should we somehow try to make it more clear at this point that it's the profile that's probably wrong and not the
dbt_project.yml
file? The only line that we really can control without this turning into something huge is theExtra project configuration value(s)...
line. IsExtra profile configuration value(s)...
better? Or evenExtra connection configuration value(s) in profile ...
?