Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] feat(gms): Adds custom ownership types #7623

Closed

Conversation

pedro93
Copy link
Collaborator

@pedro93 pedro93 commented Mar 17, 2023

Summary

Adds a new entity to DataHub's metadata model graph: OwnershipType which will allow DataHub Admins to specify their own ownership types beyond and possibly in place of the existing ones: Technical Owner, Business Owner, Data Steward, None

Details:

  • Adds a new bootstrap step which will create the existing ownership types: Technical Owner, Business Owner, Data Steward, None as new entity instances of the new OwnershipType entity.
  • Adds a new flag INGEST_DEFAULT_OWNERSHIP_TYPES, set to true by default which will control if the default ownership types should be ingested. If set to true, this ingestion process will only happen once.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@pedro93 pedro93 changed the title feat(gms): Add custom ownership types [WIP] feat(gms): Add custom ownership types Mar 17, 2023
@pedro93 pedro93 changed the title [WIP] feat(gms): Add custom ownership types [WIP] feat(gms): Adds custom ownership types Mar 17, 2023
@github-actions github-actions bot added devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata labels Mar 17, 2023
@vercel
Copy link

vercel bot commented Mar 22, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated
docs-website ✅ Ready (Inspect) Visit Preview 💬 Add your feedback Mar 22, 2023 at 3:53PM (UTC)

Copy link
Contributor

@mmmeeedddsss mmmeeedddsss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I'm not from the core datahub team, the pr is shared in https://datahubspace.slack.com/archives/C02QMLWJG12/p1679611195773309)

I assume you are already planning but this change will require some additional tweaks at least on the frontend. Hope these will not be that much.

Great PR with a huge value, I'm very excited to see it on action !

"name": "ownershipType",
"entityTypes": [ "ownershipType" ]
}
customType: optional Urn
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about adding a default to this field, it can be "None" as default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, a name like ownershipTypeUrn would fit better in here. WDYT? Are you planning to migrate current owner definitions to the new one in this pr?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mmmeeedddsss this PR includes a DataHub Bootstrap step that will populate the database with new ownership type entity instances based on the existing non-deprecated ownership types.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pedro93 As far as I can see, these custom values are not used in anywhere. Do you plan to open a separate pr for the usage of these? (Such as graphql changes, frontend changes etc)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is only the modelling and bootstrapping part of the feature, it was opened prematurely. I will close it and continue development in a fork. Once it is feature complete I will open the PR again.

@Aspect = {
"name": "ownershipTypeInfo"
}
record OwnershipTypeInfo {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could see from other schemas, you'll need to add a URN in here too, since the types will become a dynamic asset from now on. Is that correct?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This aspect is merely an information view of the new entity that this PR introduces. The urn is in the key aspect.

"enableAutocomplete": true,
"boostScore": 10.0
}
name: string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

displayName might be a better name in here

@codecov-commenter
Copy link

codecov-commenter commented Mar 24, 2023

Codecov Report

Patch coverage has no change and project coverage change: -7.84 ⚠️

Comparison is base (1324231) 74.87% compared to head (b02536e) 67.04%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7623      +/-   ##
==========================================
- Coverage   74.87%   67.04%   -7.84%     
==========================================
  Files         353      353              
  Lines       35386    35386              
==========================================
- Hits        26496    23723    -2773     
- Misses       8890    11663    +2773     
Flag Coverage Δ
pytest-testIntegration ?
pytest-testIntegrationBatch1 36.46% <ø> (ø)
pytest-testQuick 63.55% <ø> (ø)
pytest-testSlowIntegration 32.94% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ingestion/src/datahub/cli/quickstart_versioning.py 35.52% <ø> (ø)

... and 78 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@pedro93
Copy link
Collaborator Author

pedro93 commented Apr 5, 2023

Closing PR as it is incomplete, feature development will continue in a fork and will be made available once finished.

@pedro93 pedro93 closed this Apr 5, 2023
@ragerin
Copy link

ragerin commented Apr 18, 2023

@pedro93 where can we follow the fork for this feature?

@pedro93
Copy link
Collaborator Author

pedro93 commented Apr 18, 2023

The fork is private. An OSS PR will be made once the code has been developed internally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants