refactor(redshift): Improve redshift error handling with new structured reporting system

[jjoyce0510]

Scenarios Covered

Invalid Connection Host Port	✅
Invalid Credentials (user / pass)	✅
Invalid Permission to Navigate	✅
Invalid Permission to Extract Profiles	✅
Invalid Permission to Extract Lineage	✅
Invalid Permission to Extract Usage	✅
Failed to Parse Query (Lineage, Usage)	✅

Also some small fixes

• If the table is NOT external and NOT a view, but has no stats found in svv_table_info during initial scan, then we simply report 0 for row count and size in bytes. This is CORRECT because svv_table_info docs suggest that empty tables will not be present.
• If the table is EXTERNAL and profile_table_level_only, we can NEVER correctly profile it because EXTERNAL tables are never present in svv_table_info, where we get the "cheap" profiling stats for redshift. We log a warning to the user in such cases to tell them about this behavior.

Screenshots

QA

QA'd locally by attempting to replicate every scenario. Only one I couldn't was lineage parsing.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Summary by CodeRabbit

• New Features

  • Improved exception handling and specific error reporting for permission issues and unauthorized access during data profiling.
  • Added detailed error reporting for Redshift lineage extraction failures.
  • Enhanced profiling logic to skip and log messages for external Redshift tables based on certain settings.
  • Added methods for handling Redshift exceptions and reporting failures.

• Bug Fixes

  • Addressed issues with profiling tables without necessary attributes by logging warnings and skipping profile creation.

• Refactor

  • Updated error reporting formats and enriched table handling in various methods to improve clarity and logging.

• Chores

  • Added comments and improved logging for better maintenance and debugging.

[coderabbitai]

Walkthrough

The recent updates to metadata-ingestion encompass refined exception handling, augmented error reporting mechanisms, and enhanced profiling logic. Key changes include specific warnings for exception types within the ge_data_profiler, a new exception handling module for Redshift operations, refined error reporting in Redshift lineage extraction, and adjustments in profiling logic to handle external tables more effectively.

Changes

File/Path	Change Summary
ge_data_profiler.py	Refined exception handling in _generate_single_profile, distinguishing between "permission denied" and general warnings.
redshift/exception.py	Introduced functions to handle Redshift exceptions, report failures, and yield results while catching errors.
redshift/lineage_v2.py	Refactored error reporting in _populate_lineage_agg and added error reporting in generate.
redshift/profile.py	Updated profiling logic for external Redshift tables, adding messages and warnings when certain conditions are met.
redshift/query.py	Added a comment regarding the unused status of list_columns.
redshift/redshift.py	Included new methods and enhanced error handling and logging in several functions.
redshift/redshift_schema.py	Added logging, renamed variables, and updated the logic for table enrichment and statistics retrieval.
sql_generic_profiler.py	Added conditional checks in generate_profile_workunits to log warnings and skip profile creation if attributes are missing.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant GEDataProfiler
    participant Logger
    User ->> GEDataProfiler: Call _generate_single_profile()
    GEDataProfiler ->> GEDataProfiler: Try profiling
    alt Permission Denied Exception
        GEDataProfiler ->> Logger: Log specific unauthorized access warning
    else Other Exceptions
        GEDataProfiler ->> Logger: Log general warning
    end

Loading

Poem

In the land of code, where bugs sow fright,
Exception handling shines, as errors take flight.
From Redshift’s depths to GE’s vast terrain,
Each tweak, each change, a triumph plain.
With profiles now accurate, and warnings so spry,
The data streams clearer, under the coder's eye.
🌟🐇

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8722878 and 31f5788.

Files selected for processing (8)

• metadata-ingestion/src/datahub/ingestion/source/ge_data_profiler.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/exception.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/lineage_v2.py (3 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/profile.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/query.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py (7 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift_schema.py (7 hunks)
• metadata-ingestion/src/datahub/ingestion/source/sql/sql_generic_profiler.py (1 hunks)

Files skipped from review due to trivial changes (1)

• metadata-ingestion/src/datahub/ingestion/source/redshift/query.py

Additional context used

Ruff

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift_schema.py

276-277: Use a single if statement instead of nested if statements

(SIM102)

Additional comments not posted (17)

metadata-ingestion/src/datahub/ingestion/source/redshift/exception.py (3)

10-18: LGTM!

The handle_redshift_exceptions function correctly handles Redshift-specific exceptions and reports failures. The use of type hinting is appropriate.

---

20-27: LGTM!

The handle_redshift_exceptions_yield function correctly handles exceptions for callables that return iterables. The use of yield from is appropriate, and error reporting is consistent.

---

29-57: LGTM!

The report_redshift_failure function provides detailed failure reports based on the error content. The categorization of errors and the corresponding messages are well-structured.

metadata-ingestion/src/datahub/ingestion/source/redshift/profile.py (1)

51-69: LGTM!

The logic for handling profiling of external tables is well-implemented. The conditional checks and corresponding logging and reporting provide clear feedback to the user.

metadata-ingestion/src/datahub/ingestion/source/sql/sql_generic_profiler.py (1)

95-113: LGTM!

The logic for skipping profile generation based on missing table attributes is well-implemented. The logging provides clear feedback to the user.

metadata-ingestion/src/datahub/ingestion/source/redshift/lineage_v2.py (2)

243-246: LGTM!

The error reporting in _populate_lineage_agg is well-implemented. The detailed context and exception information will aid in debugging and understanding the failure.

---

413-418: LGTM!

The error reporting in generate is well-implemented. The detailed context and exception information will aid in debugging and understanding the failure.

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift_schema.py (3)

172-173: LGTM!

The warning comment about table enrichment behavior is informative and does not affect functionality.

---

Line range hint 212-221:
LGTM!

Logging the number of fetched tables/views is a good practice. The enriched tables are correctly utilized.

---

315-349: LGTM!

Handling tables with no profiles and logging the message when returning 0 improves clarity and correctness.

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py (4)

415-431: LGTM!

The inclusion of _try_get_redshift_connection and exception handling makes the method more robust.

---

432-439: LGTM!

Wrapping _extract_metadata with handle_redshift_exceptions_yield ensures better error handling and reporting.

---

Line range hint 574-628:
LGTM!

Including column count for tables/views and reporting informational messages when no tables/views are found improves clarity and completeness.

---

1117-1155: LGTM!

Handling various connection errors and reporting failures with specific messages improves robustness and user feedback.

metadata-ingestion/src/datahub/ingestion/source/ge_data_profiler.py (3)

1219-1219: LGTM! Improved error handling.

The addition of specific error handling for "permission denied" errors enhances the robustness of the profiling process.

---

1220-1227: Specific handling for "permission denied" errors is good.

The code correctly identifies "permission denied" errors and logs a specific warning message, providing clear feedback to the user.

---

1228-1234: General exception handling is well-implemented.

The code handles other exceptions by logging a general warning message, ensuring that unexpected errors are reported.

[hsheth2]

not a huge fan of this pattern - ideally error handling code should be close to the code that causes the errors, especially for these sorts of cases where the error message is extremely custom

For the generic permission denied / failed to extract metadata, they're fine here. But the svv and svl table ones feel like they should be caught closer to the cause of the error

[jjoyce0510]

These happen in like 3 places across lineage, usage, and just normal table stuff.

[jjoyce0510]

Also the code is a mess in there so I think this fallback is okay.

[jjoyce0510]

In general I agree though that this is too much of a fallback, but in this case it would be a pretty significant undertaking to cover each place we are issuing queries via the client object

[jjoyce0510]

(And this covered the cases I tested with a new sample user who I progressively added privileges for)

[jjoyce0510]

You can also see I'm just catching the "redshift" connector errors here as of now, also. So it's fairly targeted to mean that the exception came from a redshift connector call.

[hsheth2]

not a hard blocker, but ideally we'd have some tests for this logic to avoid regressions

[jjoyce0510]

basically this is skipping emitting a null profile. will check if there are any existing tests i can easily modify

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 31f5788 and 116d315.

Files selected for processing (1)

• metadata-ingestion/src/datahub/ingestion/source/ge_data_profiler.py (1 hunks)

Files skipped from review as they are similar to previous changes (1)

• metadata-ingestion/src/datahub/ingestion/source/ge_data_profiler.py

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 116d315 and 3aae262.

Files selected for processing (1)

• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py (7 hunks)

Files skipped from review as they are similar to previous changes (1)

• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 3aae262 and c8819dd.

Files selected for processing (3)

• metadata-ingestion/src/datahub/ingestion/source/redshift/exception.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/profile.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py (7 hunks)

Files skipped from review as they are similar to previous changes (3)

• metadata-ingestion/src/datahub/ingestion/source/redshift/exception.py
• metadata-ingestion/src/datahub/ingestion/source/redshift/profile.py
• metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py