Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(models): chart schema fields mapping, add dataHubAction entity, t… #11040

Merged
merged 2 commits into from
Jul 30, 2024
Merged

fix(models): chart schema fields mapping, add dataHubAction entity, t… #11040

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
203ff79
fix(models): chart schema fields mapping, add dataHubAction entity, t…
shirshanka Jul 30, 2024
00ff600
add smoke test
shirshanka Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions ...ore/src/main/java/com/linkedin/datahub/graphql/types/chart/mappers/InputFieldsMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,14 @@
import com.linkedin.datahub.graphql.QueryContext;
import com.linkedin.datahub.graphql.generated.InputField;
import com.linkedin.datahub.graphql.types.dataset.mappers.SchemaFieldMapper;
import java.net.URISyntaxException;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import lombok.extern.slf4j.Slf4j;


@Slf4j
public class InputFieldsMapper {

public static final InputFieldsMapper INSTANCE = new InputFieldsMapper();
Expand All @@ -31,13 +35,19 @@ public com.linkedin.datahub.graphql.generated.InputFields apply(
.map(
field -> {
InputField fieldResult = new InputField();
Urn parentUrn = entityUrn;

if (field.hasSchemaField()) {
fieldResult.setSchemaField(
SchemaFieldMapper.map(context, field.getSchemaField(), entityUrn));
}
if (field.hasSchemaFieldUrn()) {
fieldResult.setSchemaFieldUrn(field.getSchemaFieldUrn().toString());
try {
parentUrn = Urn.createFromString(field.getSchemaFieldUrn().getEntityKey().get(0));
} catch (URISyntaxException e) {
log.error("Field urn resolution: failed to extract parentUrn successfully from {}. Falling back to {}", field.getSchemaFieldUrn(), entityUrn, e);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the case where this was happening?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess nowhere, just defensive

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see - this is where the bad mapping was happening. Got it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly a protection from bad data - in case we have mal formatted schema field urns.

}
}
if (field.hasSchemaField()) {
fieldResult.setSchemaField(
SchemaFieldMapper.map(context, field.getSchemaField(), parentUrn));
}
return fieldResult;
})
Expand Down
14 changes: 14 additions & 0 deletions metadata-models/src/main/pegasus/com/linkedin/metadata/key/DataHubActionKey.pdl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
namespace com.linkedin.metadata.key

/**
* Key for a DataHub Action Pipeline
*/
@Aspect = {
"name": "dataHubActionKey"
}
record DataHubActionKey {
/**
* A unique id for the Action, either generated or provided
*/
id: string
}
26 changes: 26 additions & 0 deletions metadata-models/src/main/resources/entity-registry.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ entities:
- forms
- subTypes
- incidentsSummary
- testResults
- name: dataFlow
category: core
keyAspect: dataFlowKey
Expand All @@ -95,12 +96,14 @@ entities:
- incidentsSummary
- forms
- subTypes
- testResults
- name: dataProcess
keyAspect: dataProcessKey
aspects:
- dataProcessInfo
- ownership
- status
- testResults
- name: dataProcessInstance
doc: DataProcessInstance represents an instance of a datajob/jobflow run
keyAspect: dataProcessInstanceKey
Expand All @@ -111,6 +114,7 @@ entities:
- dataProcessInstanceRelationships
- dataProcessInstanceRunEvent
- status
- testResults
- name: chart
category: core
keyAspect: chartKey
Expand All @@ -136,6 +140,7 @@ entities:
- structuredProperties
- incidentsSummary
- forms
- testResults
- name: dashboard
keyAspect: dashboardKey
aspects:
Expand All @@ -159,6 +164,7 @@ entities:
- structuredProperties
- incidentsSummary
- forms
- testResults
- name: notebook
doc: Notebook represents a combination of query, text, chart and etc. This is in BETA version
keyAspect: notebookKey
Expand All @@ -176,6 +182,7 @@ entities:
- subTypes
- dataPlatformInstance
- browsePathsV2
- testResults
- name: corpuser
doc: CorpUser represents an identity of a person (or an account) in the enterprise.
keyAspect: corpUserKey
Expand All @@ -193,6 +200,7 @@ entities:
- roleMembership
- structuredProperties
- forms
- testResults
- name: corpGroup
doc: CorpGroup represents an identity of a group of users in the enterprise.
keyAspect: corpGroupKey
Expand All @@ -206,6 +214,7 @@ entities:
- roleMembership
- structuredProperties
- forms
- testResults
- name: domain
doc: A data domain within an organization.
category: core
Expand All @@ -216,6 +225,7 @@ entities:
- ownership
- structuredProperties
- forms
- testResults
- name: container
doc: A container of related data assets.
category: core
Expand All @@ -236,6 +246,7 @@ entities:
- browsePathsV2
- structuredProperties
- forms
- testResults
- name: tag
category: core
keyAspect: tagKey
Expand All @@ -244,6 +255,7 @@ entities:
- ownership
- deprecation
- status
- testResults
- name: glossaryTerm
category: core
keyAspect: glossaryTermKey
Expand All @@ -259,6 +271,7 @@ entities:
- browsePaths
- structuredProperties
- forms
- testResults
- name: glossaryNode
category: core
keyAspect: glossaryNodeKey
Expand All @@ -269,6 +282,7 @@ entities:
- status
- structuredProperties
- forms
- testResults
- name: dataHubIngestionSource
category: internal
keyAspect: dataHubIngestionSourceKey
Expand Down Expand Up @@ -340,6 +354,7 @@ entities:
- browsePathsV2
- structuredProperties
- forms
- testResults
- name: mlModelGroup
category: core
keyAspect: mlModelGroupKey
Expand All @@ -357,6 +372,7 @@ entities:
- browsePathsV2
- structuredProperties
- forms
- testResults
- name: mlModelDeployment
category: core
keyAspect: mlModelDeploymentKey
Expand All @@ -367,6 +383,7 @@ entities:
- deprecation
- globalTags
- dataPlatformInstance
- testResults
- name: mlFeatureTable
category: core
keyAspect: mlFeatureTableKey
Expand All @@ -385,6 +402,7 @@ entities:
- browsePathsV2
- structuredProperties
- forms
- testResults
- name: mlFeature
category: core
keyAspect: mlFeatureKey
Expand All @@ -403,6 +421,7 @@ entities:
- browsePathsV2
- structuredProperties
- forms
- testResults
- name: mlPrimaryKey
category: core
keyAspect: mlPrimaryKeyKey
Expand All @@ -419,6 +438,7 @@ entities:
- dataPlatformInstance
- structuredProperties
- forms
- testResults
- name: telemetry
category: internal
keyAspect: telemetryKey
Expand Down Expand Up @@ -455,6 +475,7 @@ entities:
- forms
- businessAttributes
- documentation
- testResults
- name: globalSettings
doc: Global settings for an the platform
category: internal
Expand Down Expand Up @@ -522,6 +543,7 @@ entities:
- status
- structuredProperties
- forms
- testResults
- name: ownershipType
doc: Ownership Type represents a user-created ownership category for a person or group who is responsible for an asset.
category: core
Expand Down Expand Up @@ -549,6 +571,10 @@ entities:
keyAspect: dataHubPersonaKey
aspects:
- dataHubPersonaInfo
- name: dataHubAction
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the other aspects, or I guess that will come in a followup?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need other aspects - this is required to allow source in metadataAttribution to contain a dataHubAction urn.

category: internal
keyAspect: dataHubActionKey
aspects: []
- name: entityType
doc: A type of entity in the DataHub Metadata Model.
category: core
Expand Down
Empty file added smoke-test/tests/schema_fields/__init__.py
View file
Open in desktop
Empty file.
20 changes: 20 additions & 0 deletions smoke-test/tests/schema_fields/queries/get_chart_field.gql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
query($urn:String!) {
chart(urn: $urn) {
inputFields {
fields {
schemaFieldUrn
schemaField {
schemaFieldEntity {
urn
fieldPath
documentation {
documentations {
documentation
}
}
}
}
}
}
}
}
Loading
Loading