Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible to intergrate with other tool to implement lineage features? #356

Closed
fBedecarrats opened this issue Mar 30, 2020 · 2 comments
Closed
Labels
keep fresh Disables stalebot from closing an issue type:feature A new feature request type:question An issue or PR that needs more information or a user question

Comments

@fBedecarrats
Copy link
Contributor

(Don't know if this fits here or in Slack).
For Amundsen instances running on Neo4J, I can think of 3 directions to implement lineage with the current version of Amundsen.

  1. Integrating with ArcadeAnalytics which provides a ready to use facility to visualize (with embedable widgets-iframes) and edit graph data. Graph widgets are triggered by cypher queries so one can filter for tables and their lineage relationships. This already works out of the box and you can launch AA docker image and connect it to your Neo4J Amundsen graph database to navigate and edit it. Being able to edit the Neo4J Graph in an ergonomic ways makes it doable to manually document lineage relathonships.
  2. Using the Neo4J ETL tool to extract source schema that reflects the lineage between tables inside a RDBMS.
  3. Including in the Gobblin-Like scripts the corresponding ones to parse and ingest the ETL jobs that move the data from a system to another.

I tried 1, which works quite fine in Neo4J, but one needs to publish the changes to Elasticsearch frequently and I can't think of an easy way to do it.
Sorry in advance if this seems too much out of the scope you imagined for Amundsen. If so, please feel free to delete this issue.

@feng-tao feng-tao added keep fresh Disables stalebot from closing an issue lineage labels Mar 31, 2020
@feng-tao
Copy link
Member

Thanks for bringing this up. There are some previous discussions in #163. I think it is a good topic to discuss in the slack.

@Golodhros Golodhros added type:feature A new feature request status:in_progress Issue that is being worked on right now and removed lineage labels May 29, 2020
@anheiser1997
Copy link

@fBedecarrats @feng-tao Hi, I had a query. If we were to use one of the three options above, and also considering using DBT as another option, would the URL for the page we built for lineage be given as a third party URL as mentioned here: #69

gjxdxh pushed a commit to gjxdxh/amundsen that referenced this issue Dec 8, 2020
dorianj pushed a commit to dorianj/amundsen that referenced this issue Apr 25, 2021
- Added badge support for tables
- Added BadgeConfig to customize display names/styles of defined badges
dorianj pushed a commit to dorianj/amundsen that referenced this issue Apr 25, 2021
feng-tao pushed a commit that referenced this issue May 7, 2021
- Added badge support for tables
- Added BadgeConfig to customize display names/styles of defined badges
feng-tao pushed a commit that referenced this issue May 7, 2021
…iews (#356)

Signed-off-by: Nathan Lawrence <[email protected]>

Co-authored-by: Nathan Lawrence <[email protected]>
hansadriaans pushed a commit to DataChefHQ/amundsen that referenced this issue Jun 30, 2022
hansadriaans pushed a commit to DataChefHQ/amundsen that referenced this issue Jun 30, 2022
- Added badge support for tables
- Added BadgeConfig to customize display names/styles of defined badges
@Golodhros Golodhros added type:question An issue or PR that needs more information or a user question and removed status:in_progress Issue that is being worked on right now labels Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keep fresh Disables stalebot from closing an issue type:feature A new feature request type:question An issue or PR that needs more information or a user question
Projects
None yet
Development

No branches or pull requests

4 participants