Add entity column validations when getting historical features from bigquery #1614
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1614 +/- ##
==========================================
+ Coverage 83.76% 83.88% +0.11%
==========================================
Files 67 67
Lines 5846 5883 +37
==========================================
+ Hits 4897 4935 +38
+ Misses 949 948 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| @@ -0,0 +1 @@ | |||
| <?xml version="1.0" encoding="utf-8"?><testsuites><testsuite errors="0" failures="0" hostname="Achals-MBP" name="pytest" skipped="0" tests="6" time="154.414" timestamp="2021-06-01T17:24:58.989137"><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="32" name="test_telemetry_on_v09" time="18.383"></testcase><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="59" name="test_telemetry_off_v09" time="32.697"></testcase><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="88" name="test_telemetry_on" time="18.214"></testcase><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="120" name="test_telemetry_off" time="32.748"></testcase><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="153" name="test_exception_telemetry_on" time="18.851"></testcase><testcase classname="telemetry_tests.test_telemetry" file="telemetry_tests/test_telemetry.py" line="170" name="test_exception_telemetry_off" time="32.693"></testcase></testsuite></testsuites> | |||
There was a problem hiding this comment.
I'm assuming this should be removed.
| missing_keys = expected_columns - entity_df_columns | ||
|
|
||
| if len(missing_keys) != 0: | ||
| raise ValueError( |
There was a problem hiding this comment.
@achals would love to get your take on whether we should be using a custom error type here (or ValueError)? I've accepted #1615, but now that I think about it we probably need tests that cover this failure case. We can always leave those tests as a TODO in the code until we have completed our test suite refactor, but at that point we probably want different offline stores to raise identical exceptions, which may be easier to do with a custom error type.
There was a problem hiding this comment.
I think a specific validation error is better to throw since they can be caught and handled better potentially.
And yeah, re: tests I'll fill in the gaps in the stuff I just merged.
achals
force-pushed
the
achal/bigquery-validation
branch
from
35d198b
to
049b588
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: achals, woop The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
…igquery Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah <[email protected]>
achals
force-pushed
the
achal/bigquery-validation
branch
from
9ee3ae6
to
2c434f3
Compare
|
New changes are detected. LGTM label has been removed. |
…igquery (#1614) * Add entity column validations when getting historical features from bigquery Signed-off-by: Achal Shah <[email protected]> * make format Signed-off-by: Achal Shah <[email protected]> * Remove wrong file Signed-off-by: Achal Shah <[email protected]> * Add tests Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah [email protected]
What this PR does / why we need it:
Right now, we don't validate that the entity_df contains the expected columns. This may result in strange errors downstream. But since we already have all the necessary information to validate that the needed columns exist, the validations can catch errors before we try to retrieve data from bigquery.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: