Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ingestion/looker): skip personal folder independent looks #11415

Merged
merged 7 commits into from
Sep 24, 2024
Merged

fix(ingestion/looker): skip personal folder independent looks #11415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
da1fab3
update in test-cases
sid-acryl Sep 18, 2024
4340561
Merge branch 'master' into cus-2738-personal-looks-asset
sid-acryl Sep 18, 2024
2e9067b
Merge branch 'master' into cus-2738-personal-looks-asset
sid-acryl Sep 18, 2024
cb0ac6a
update golden files
sid-acryl Sep 18, 2024
b031754
address review comments
sid-acryl Sep 19, 2024
1716f38
Merge branch 'master' into cus-2738-personal-looks-asset
sid-acryl Sep 19, 2024
8f000ac
Merge branch 'master' into cus-2738-personal-looks-asset
sid-acryl Sep 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,15 +380,16 @@ def _get_input_fields_from_query(
)
)

# A query uses fields for filtering and those fields are defined in views, find the views those fields use
# A query uses fields for filtering, and those fields are defined in views, find the views those fields use
filters: MutableMapping[str, Any] = (
query.filters if query.filters is not None else {}
)
for field in filters.keys():
if field is None:
continue

# we haven't loaded in metadata about the explore yet, so we need to wait until explores are populated later to fetch this
# we haven't loaded in metadata about the explore yet, so we need to wait until explores are populated
# later to fetch this
result.append(
InputFieldElement(
name=field, view_field=None, model=query.model, explore=query.view
Expand Down Expand Up @@ -1486,13 +1487,27 @@ def extract_independent_looks(self) -> Iterable[MetadataWorkUnit]:
)
for look in all_looks:
if look.id in self.reachable_look_registry:
# This look is reachable from Dashboard
# This look is reachable from the Dashboard
continue

if look.query_id is None:
logger.info(f"query_id is None for look {look.title}({look.id})")
continue

if self.source_config.skip_personal_folders:
if look.folder is not None and (
look.folder.is_personal or look.folder.is_personal_descendant
):
self.reporter.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the benefit of having this reporter info vs collecting these in the report below and add a debug log?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to maintain consistency with the logic used in the personal dashboards.

title="Dropped Look",
message="Dropped due to being a personal folder",
context=f"Look ID: {look.id}",
)
sid-acryl marked this conversation as resolved.
Show resolved Hide resolved

assert look.id, "Looker id is null"
self.reporter.report_charts_dropped(look.id)
continue

if look.id is not None:
query: Optional[Query] = self.looker_api.get_look(
look.id, fields=["query"]
Expand All @@ -1510,11 +1525,12 @@ def extract_independent_looks(self) -> Iterable[MetadataWorkUnit]:
LookerDashboardElement
] = self._get_looker_dashboard_element(
DashboardElement(
id=f"looks_{look.id}", # to avoid conflict with non-standalone looks (element.id prefixes), we add the "looks_" prefix to look.id.
id=f"looks_{look.id}", # to avoid conflict with non-standalone looks (element.id prefixes),
# we add the "looks_" prefix to look.id.
title=look.title,
subtitle_text=look.description,
look_id=look.id,
dashboard_id=None, # As this is independent look
dashboard_id=None, # As this is an independent look
look=LookWithQuery(
query=query, folder=look.folder, user_id=look.user_id
),
Expand Down
Loading
Loading