-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(ingest/dynamodb): add support for classification #10138
Conversation
- also addresses followups from datahub-project#10031 (review)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good, had one comment
isinstance(wu.metadata, MetadataChangeEvent) | ||
and len(wu.metadata.proposedSnapshot.aspects) > 1 | ||
) or not maybe_schema_metadata: | ||
yield wu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't we remove the SchemaMetadata aspect from the snapshot, since we're emitting it after
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, this is leaving the MCEs as is and not processing it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In followup, I am planning to add support for classifying SchemaMetadata in MCE and refractoring sql_common to use this.
Checklist