Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(sdk): fix ownership emission for groups #7751

Merged
merged 1 commit into from
Apr 5, 2023
Merged

feat(sdk): fix ownership emission for groups #7751

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 4 additions & 66 deletions docs/api/tutorials/creating-users-and-groups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,13 @@ Update succeeded for urn urn:li:corpuser:datahub.

### Upsert Group

Save this `group.yaml` as a local file. Note that the group includes a list of users who are admins (these will be marked as owners) and members.
Save this `group.yaml` as a local file. Note that the group includes a list of users who are owners and members.
Within these lists, you can refer to the users by their ids or their urns, and can additionally specify their metadata inline within the group description itself. See the example below to understand how this works and feel free to make modifications to this file locally to see the effects of your changes in your local DataHub instance.

```yaml
id: [email protected]
display_name: Foo Group
admins:
owners:
- datahub
members:
- [email protected] # refer to a user either by id or by urn
Expand Down Expand Up @@ -88,33 +88,7 @@ The following code creates a user named `The Bar` with urn `urn:li:corpuser:bar@
You can refer to the full code in [upsert_user.py](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/upsert_user.py).

```python
import logging

from datahub.api.entities.corpuser.corpuser import CorpUser, CorpUserGenerationConfig
from datahub.ingestion.graph.client import DataHubGraph, DataHubGraphConfig

log = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

user_email = "[email protected]"

user: CorpUser = CorpUser(
id=user_email,
display_name="The Bar",
email=user_email,
title="Software Engineer",
first_name="The",
last_name="Bar",
full_name="The Bar",
)

# Create graph client
datahub_graph = DataHubGraph(DataHubGraphConfig(server="http://localhost:8080"))
for event in user.generate_mcp(
generation_config=CorpUserGenerationConfig(override_editable=False)
):
datahub_graph.emit(event)
log.info(f"Upserted user {user.urn}")
{{ inline /metadata-ingestion/examples/library/upsert_user.py show_path_as_comment }}
```

### Upsert Group
Expand All @@ -123,43 +97,7 @@ The following code creates a group called `Foo Group` with group `urn:li:corpgro
You can refer to the full code in [upsert_group.py](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/upsert_group.py).

```python
import logging

from datahub.api.entities.corpgroup.corpgroup import (
CorpGroup,
CorpGroupGenerationConfig,
)
from datahub.ingestion.graph.client import DataHubGraph, DataHubGraphConfig
from datahub.utilities.urns.corpuser_urn import CorpuserUrn

log = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

group_email = "[email protected]"
group = CorpGroup(
id=group_email,
admins=[str(CorpuserUrn.create_from_id("datahub"))],
members=[
str(CorpuserUrn.create_from_id("[email protected]")),
str(CorpuserUrn.create_from_id("[email protected]")),
],
groups=[],
display_name="Foo Group",
email=group_email,
description="Software engineering team",
slack="@foogroup",
)

# Create graph client
datahub_graph = DataHubGraph(DataHubGraphConfig(server="http://localhost:8080"))

for event in group.generate_mcp(
generation_config=CorpGroupGenerationConfig(
override_editable=False, datahub_graph=datahub_graph
)
):
datahub_graph.emit(event)
log.info(f"Upserted group {group.urn}")
{{ inline /metadata-ingestion/examples/library/upsert_group.py show_path_as_comment }}
```

We're using the `MetdataChangeProposalWrapper` to change entities in this example.
Expand Down
2 changes: 1 addition & 1 deletion metadata-ingestion/examples/library/upsert_group.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
group_email = "[email protected]"
group = CorpGroup(
id=group_email,
admins=[str(CorpuserUrn.create_from_id("datahub"))],
owners=[str(CorpuserUrn.create_from_id("datahub"))],
members=[
str(CorpuserUrn.create_from_id("[email protected]")),
str(CorpuserUrn.create_from_id("[email protected]")),
Expand Down
59 changes: 31 additions & 28 deletions metadata-ingestion/src/datahub/api/entities/corpgroup/corpgroup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@

import datahub.emitter.mce_builder as builder
from datahub.api.entities.corpuser.corpuser import CorpUser, CorpUserGenerationConfig
from datahub.configuration.common import ConfigurationError
from datahub.configuration.validate_field_rename import pydantic_renamed_field
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph
Expand Down Expand Up @@ -50,7 +52,7 @@ class CorpGroup(BaseModel):
overrideEditable (bool): If True, group information that is editable in the UI will be overridden
picture_link (Optional[str]): A URL which points to a picture which user wants to set as the photo for the group
slack (Optional[str]): Slack channel for the group
admins (List[Union[str, CorpUser]]): A list of administrator ids (or urns) for the group. You can also provide the user record for the admin inline within this section
owners (List[Union[str, CorpUser]]): A list of owner/administrator ids (or urns) for the group. You can also provide the user record for the owner inline within this section
members (List[Union[str, CorpUser]]): A list of member ids (or urns) for the group.
"""

Expand All @@ -65,10 +67,12 @@ class CorpGroup(BaseModel):
overrideEditable: bool = False
picture_link: Optional[str] = None
slack: Optional[str] = None
admins: List[Union[str, CorpUser]] = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets not make this a breaking change- just remove it from docs and mark it as deprecated please.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already taken care of by this line

_rename_admins_to_owners = pydantic_renamed_field("admins", "owners")

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

owners: List[Union[str, CorpUser]] = []
members: List[Union[str, CorpUser]] = []

@pydantic.validator("admins", "members", each_item=True)
_rename_admins_to_owners = pydantic_renamed_field("admins", "owners")

@pydantic.validator("owners", "members", each_item=True)
def make_urn_if_needed(v):
if isinstance(v, str):
return builder.make_user_urn(v)
Expand All @@ -88,22 +92,22 @@ def generate_mcp(
members_to_create: List[CorpUser] = (
[u for u in self.members if isinstance(u, CorpUser)] if self.members else []
)
admins_to_create: List[CorpUser] = (
[u for u in self.admins if isinstance(u, CorpUser)] if self.admins else []
owners_to_create: List[CorpUser] = (
[u for u in self.owners if isinstance(u, CorpUser)] if self.owners else []
)

member_urns: List[str] = (
[u.urn if isinstance(u, CorpUser) else u for u in self.members]
if self.members
else []
)
admin_urns: List[str] = (
[u.urn if isinstance(u, CorpUser) else u for u in self.admins]
if self.admins
owner_urns: List[str] = (
[u.urn if isinstance(u, CorpUser) else u for u in self.owners]
if self.owners
else []
)

for m in members_to_create + admins_to_create:
for m in members_to_create + owners_to_create:
if m.urn not in urns_created:
yield from m.generate_mcp(
generation_config=CorpUserGenerationConfig(
Expand All @@ -113,7 +117,7 @@ def generate_mcp(
urns_created.add(m.urn)
else:
logger.warn(
f"Supressing emission of member {m.urn} before we already emitted metadata for it"
f"Suppressing emission of member {m.urn} before we already emitted metadata for it"
)

aspects: List[_Aspect] = [StatusClass(removed=False)]
Expand All @@ -129,7 +133,7 @@ def generate_mcp(
else:
aspects.append(
CorpGroupInfoClass(
admins=admin_urns, # deprecated but we fill it out for consistency
admins=owner_urns, # deprecated but we fill it out for consistency
members=member_urns, # deprecated but we fill it out for consistency
groups=[], # deprecated
displayName=self.display_name,
Expand All @@ -151,23 +155,22 @@ def generate_mcp(
for aspect in aspects:
yield MetadataChangeProposalWrapper(entityUrn=self.urn, aspect=aspect)

# Unfortunately, admins and members fields in CorpGroupInfo has been deprecated
# So we need to emit Ownership and GroupMembership oriented to the individual users
# Add owners to the group.
if owner_urns:
ownership = OwnershipClass(owners=[])
for urn in owner_urns:
ownership.owners.append(
OwnerClass(owner=urn, type=OwnershipTypeClass.TECHNICAL_OWNER)
)
yield MetadataChangeProposalWrapper(entityUrn=self.urn, aspect=ownership)

# Unfortunately, the members in CorpGroupInfo has been deprecated.
# So we need to emit GroupMembership oriented to the individual users.
# TODO: Move this to PATCH MCP-s once these aspects are supported via patch.
if generation_config.datahub_graph is not None:
datahub_graph = generation_config.datahub_graph
for urn in admin_urns:
ownership = datahub_graph.get_aspect(
urn, OwnershipClass
) or OwnershipClass(owners=[])
if self.urn not in [owner.owner for owner in ownership.owners]:
ownership.owners = ownership.owners + [
OwnerClass(owner=urn, type=OwnershipTypeClass.TECHNICAL_OWNER)
]
yield MetadataChangeProposalWrapper(
entityUrn=self.urn, aspect=ownership
)

# Add group membership to each user.
for urn in member_urns:
group_membership = datahub_graph.get_aspect(
urn, GroupMembershipClass
Expand All @@ -180,13 +183,13 @@ def generate_mcp(
entityUrn=urn, aspect=group_membership
)
else:
if admin_urns or member_urns:
raise Exception(
"Unable to emit group ownership because admins or members are non-empty, and a DataHubGraph instance was not provided."
if member_urns:
raise ConfigurationError(
"Unable to emit group membership because members is non-empty, and a DataHubGraph instance was not provided."
)

# emit status aspects for all user urns referenced (to ensure they get created)
for urn in set(admin_urns).union(set(member_urns)):
for urn in set(owner_urns).union(set(member_urns)):
yield MetadataChangeProposalWrapper(
entityUrn=urn, aspect=StatusClass(removed=False)
)
Expand Down