Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rel2.0): default removal option to REMOVE_ALL_EDGES_FROM_SOURCE #478

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jsdonn
Copy link
Contributor

@jsdonn jsdonn commented Nov 16, 2024

Summary

Big PR with several major changes to relationship ingestion.

Major

  1. Default relationship removal options to REMOVE_ALL_EDGES_FROM_SOURCE during ingestion.
    One caveat is the case where relationships are removed when an aspect is soft-deleted. For now, we will only allow soft-deleting an aspect which has relationships when the DAO is in RelationshipSource.ASPECT_METADATA mode. By default, the DAO is in RelationshipSource.RELATIONSHIP_BUILDERS mode which does NOT allow soft deleting aspects that have relationships.

  2. RelationshipV2 can be built and ingested using relationship builders. This is to bridge the gap for existing aspects with data currently ingested into the metadata graph but do not yet have any relationships. Since new relationship models can only be defined following the Relationship 2.0 spec, we need a way to support building relationshipV2s with relationship builders.

  3. Removed the neo4j module since no one is using that anymore.

Minor

  1. Remove destination field parameter from checkSameUrn. Now throws an exception if the removal option is not REMOVE_ALL_EDGES_FROM_SOURCE.

  2. Removed validateRelationshipSchema methods which don't specify v1 or v2. Now, relationship schema validation must include a boolean parameter indicating that the relationship is v1 or v2.

Testing Done

Added several unit tests covering multiple ingestion cases.

  1. relationshipv2 -> use relationship builders when relationship builders exist and return non-empty relationship updates
  2. relationshipv2 -> use aspect metadata when relationship builders don't return any relationship updates
  3. relationshipv2 -> default to relationship builders when both relationship builders and relationship fields exist

Update several other unit tests to expect IllegalArgumentExceptions when passing in removal options that are not REMOVE_ALL_EDGES_FROM_SOURCE.

Updated several unit tests with new checkSameUrn and validateRelationshipSchema method signatures.

Added a new aspect for testing (AspectFooBaz) so updated a few resource class unit tests.

Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant