CT-2264, CT-2259, CT-1783: Improved event serialization failure handling #7249
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[CT-2259](#7108) identifies an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined object is attempted to be logged. This generally happened in the form of `{{ log(undefined_variable, info=True) }}`. This commit adding this test exists for two reasons 1. Ensure we don't have a regression in this going forward 2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3
[CT-1783](#6568) describes a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all `{{ log("logging stuff", info=True) }}` invocations break. This commit adds a test for this for two reasons: 1. Ensure we don't regress to this behavior going forward 2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3
[CT-2264](#7113) states that failed serialization should result in an exception handling path which will fire another event instead of raising an exception. This is hard to test perfectly because the exception handling path for serialization depending on whether pytest is present. If pytest isn't present, a new event documentation the failed serialization is fired. If pytest is present, the failed serialization gets raised as an exception. Thus this added test ensures that the expected exception is raised and assumes that the correct event will be fired normally.
This commit updates the `msg_to_dict` exception handling path to fire a warning level event instead of raising an exception. Truthfully, we're not sure if this exception handling path is even possible to hit. That's because we recently switched from betterproto to google's protobuf. However, exception path is the subject of [CT-2264](#7113). Though we don't think it's actually possible to hit it anymore, we still want to handle the case if it is.
…ent` [CT-2264](#7113) wants logging messages about event serialization failure to be `WARNING` level events. This does that.
This was
linked to
issues
Mar 31, 2023
gshank
approved these changes
Apr 5, 2023
QMalcolm
deleted the
ct-2264-ct-2259-ct-1783--improved-event-serialization-failure-handling
branch
emmyoop
pushed a commit
that referenced
this pull request
Apr 10, 2023
…ing (#7249) * Add tests for logging jinja2.Undefined objects [CT-2259](#7108) identifies an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined object is attempted to be logged. This generally happened in the form of `{{ log(undefined_variable, info=True) }}`. This commit adding this test exists for two reasons 1. Ensure we don't have a regression in this going forward 2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3 * Add tests for checking `DBT_ENV_SECRET_`s don't break logging [CT-1783](#6568) describes a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all `{{ log("logging stuff", info=True) }}` invocations break. This commit adds a test for this for two reasons: 1. Ensure we don't regress to this behavior going forward 2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3 * Add tests ensuring failed event serialization is handled correctly [CT-2264](#7113) states that failed serialization should result in an exception handling path which will fire another event instead of raising an exception. This is hard to test perfectly because the exception handling path for serialization depending on whether pytest is present. If pytest isn't present, a new event documentation the failed serialization is fired. If pytest is present, the failed serialization gets raised as an exception. Thus this added test ensures that the expected exception is raised and assumes that the correct event will be fired normally. * Log warning when event serialization fails in `msg_to_dict` This commit updates the `msg_to_dict` exception handling path to fire a warning level event instead of raising an exception. Truthfully, we're not sure if this exception handling path is even possible to hit. That's because we recently switched from betterproto to google's protobuf. However, exception path is the subject of [CT-2264](#7113). Though we don't think it's actually possible to hit it anymore, we still want to handle the case if it is. * Update serialization failure note to be a warn level event in `BaseEvent` [CT-2264](#7113) wants logging messages about event serialization failure to be `WARNING` level events. This does that. * Add changie info for changes * Add test to check exception handling of `msg_to_dict`
emmyoop
pushed a commit
that referenced
this pull request
Apr 11, 2023
…ing (#7249) * Add tests for logging jinja2.Undefined objects [CT-2259](#7108) identifies an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined object is attempted to be logged. This generally happened in the form of `{{ log(undefined_variable, info=True) }}`. This commit adding this test exists for two reasons 1. Ensure we don't have a regression in this going forward 2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3 * Add tests for checking `DBT_ENV_SECRET_`s don't break logging [CT-1783](#6568) describes a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all `{{ log("logging stuff", info=True) }}` invocations break. This commit adds a test for this for two reasons: 1. Ensure we don't regress to this behavior going forward 2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3 * Add tests ensuring failed event serialization is handled correctly [CT-2264](#7113) states that failed serialization should result in an exception handling path which will fire another event instead of raising an exception. This is hard to test perfectly because the exception handling path for serialization depending on whether pytest is present. If pytest isn't present, a new event documentation the failed serialization is fired. If pytest is present, the failed serialization gets raised as an exception. Thus this added test ensures that the expected exception is raised and assumes that the correct event will be fired normally. * Log warning when event serialization fails in `msg_to_dict` This commit updates the `msg_to_dict` exception handling path to fire a warning level event instead of raising an exception. Truthfully, we're not sure if this exception handling path is even possible to hit. That's because we recently switched from betterproto to google's protobuf. However, exception path is the subject of [CT-2264](#7113). Though we don't think it's actually possible to hit it anymore, we still want to handle the case if it is. * Update serialization failure note to be a warn level event in `BaseEvent` [CT-2264](#7113) wants logging messages about event serialization failure to be `WARNING` level events. This does that. * Add changie info for changes * Add test to check exception handling of `msg_to_dict`
QMalcolm
added a commit
that referenced
this pull request
Apr 11, 2023
…ing (#7249) * Add tests for logging jinja2.Undefined objects [CT-2259](#7108) identifies an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined object is attempted to be logged. This generally happened in the form of `{{ log(undefined_variable, info=True) }}`. This commit adding this test exists for two reasons 1. Ensure we don't have a regression in this going forward 2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3 * Add tests for checking `DBT_ENV_SECRET_`s don't break logging [CT-1783](#6568) describes a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all `{{ log("logging stuff", info=True) }}` invocations break. This commit adds a test for this for two reasons: 1. Ensure we don't regress to this behavior going forward 2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3 * Log warning when event serialization fails in `msg_to_dict` This commit updates the `msg_to_dict` exception handling path to fire a warning level event instead of raising an exception. Truthfully, we're not sure if this exception handling path is even possible to hit. That's because we recently switched from betterproto to google's protobuf. However, exception path is the subject of [CT-2264](#7113). Though we don't think it's actually possible to hit it anymore, we still want to handle the case if it is. * Add changie info for changes * Add test to check exception handling of `msg_to_dict`
This was referenced Apr 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #7113, #7108, #6568
Description
#7113 discusses a way to ensure event serialization failure doesn't become an exception. @gshank's work in #7190 actually fixed most of what #7113 discusses. This PR finishes off the additional requirements in #7113 (making the event emitted on serialization failure a
WARNINGlevel event) and adds some regression tests in relation to #7108 and #6568. This PR primarily serves as a "commit" for which to backport the fixes to dbt-core 1.0-1.4. I have tested the regression tests on 1.3 to ensure that they will fail. Thus when we backport this, the backport will have to include changes specific to the dbt-core version to satisfy the tests.Checklist
changie newto create a changelog entry