Add sample deduplication connector #6847

popa-raluca · 2022-08-23T10:45:50Z

Signed-off-by: Raluca Popa [email protected]

Add sample deduplication connector. Provides basic implementation for handling duplicate entities. The matching of the duplicates is made based on the qualified names of the entities.

Description

Related Issue(s)

Testing

Tested locally with GAF and DE OMAS

Release Notes & Documentation

Additional notes

Signed-off-by: Raluca Popa <[email protected]>

planetf1 · 2022-08-24T08:40:09Z

Is this the first step?

I didn't see build/packaging related content?
Should we place sample connectors in base egeria or elsewhere? We've got a general strategy of only keeping the core code in base egeria, though if it's a non-scenario based, basic connector (like in memory repo) I think there's a case for having in base.

popa-raluca · 2022-08-24T09:49:06Z

Is this the first step?

I didn't see build/packaging related content? Should we place sample connectors in base egeria or elsewhere? We've got a general strategy of only keeping the core code in base egeria, though if it's a non-scenario based, basic connector (like in memory repo) I think there's a case for having in base.

It's more a basic connector implementation for deduplication that we used in our tests so far. I don't think we're ready to have it scenario based yet, just wanted to have it somewhere in github. It is placed in governce-action-connectors along with the other basic connector implementations for governance. Maybe we can discuss about moving them to a more proper place at a later time?

planetf1 · 2022-08-24T10:46:35Z

@popa-raluca Thanks for clarifying - makes sense, I agree with you. It does bring up a question of how we want to handle meantime, but that's a longer term question

lpalashevski

Approved with minor remark about the name, since this is remediation action perhaps name describing exactly the action (?) i.e. "QualifiedNamePeerDuplicateGovernanceActionConnector"

I see that you explain the purpose in the javadoc, I think this is more explicit.

Signed-off-by: Raluca Popa <[email protected]>

sonatype-lift · 2022-08-31T15:34:09Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+                SearchProperties searchProperties = getSearchProperties(qualifiedName);
+                List<OpenMetadataElement> elements = store.findMetadataElements(targetElement.getType().getTypeId(),
+                        null, searchProperties, null, null, null,
+                        null, false, true, new Date(), 0, 0);


JavaUtilDate: Date has a bad API that leads to bugs; prefer java.time.Instant or LocalDate.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-08-31T15:34:11Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+     * @throws ConnectorCheckedException there is a problem within the governance action service.
+     */
+    @Override
+    public void start() throws ConnectorCheckedException {


UnsynchronizedOverridesSynchronized: Unsynchronized method start overrides synchronized method in ConnectorBase

Suggested change

public void start() throws ConnectorCheckedException {

public synchronized void start() throws ConnectorCheckedException {

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-08-31T15:34:12Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+        PropertyCondition condition = new PropertyCondition();
+        PrimitiveTypePropertyValue primitivePropertyValue = new PrimitiveTypePropertyValue();
+
+        primitivePropertyValue.setPrimitiveTypeCategory(PrimitiveTypeCategory.OM_PRIMITIVE_TYPE_STRING);


💬 2 similar findings have been found in this PR

NULL_DEREFERENCE: object primitivePropertyValue.primitiveValue last assigned on line 98 could be null and is dereferenced by call to setPrimitiveTypeCategory(...) at line 100.

🔎 Expand here to view all instances of this finding

File Path Line Number

open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java 60

open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java 62

Visit the Lift Web Console to find more details in your report.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-08-31T15:35:12Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+                SearchProperties searchProperties = getSearchProperties(qualifiedName);
+                List<OpenMetadataElement> elements = store.findMetadataElements(targetElement.getType().getTypeId(),
+                        null, searchProperties, null, null, null,
+                        null, false, true, new Date(), 0, 0);


JavaUtilDate: Date has a bad API that leads to bugs; prefer java.time.Instant or LocalDate.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-08-31T15:35:14Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+     * @throws ConnectorCheckedException there is a problem within the governance action service.
+     */
+    @Override
+    public void start() throws ConnectorCheckedException {


UnsynchronizedOverridesSynchronized: Unsynchronized method start overrides synchronized method in ConnectorBase

Suggested change

public void start() throws ConnectorCheckedException {

public synchronized void start() throws ConnectorCheckedException {

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-08-31T15:35:15Z

...ctors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java

+
+                String qualifiedName = targetElement.getElementProperties().getPropertyValueMap().get(QUALIFIED_NAME_PROPERTY).valueAsString();
+                SearchProperties searchProperties = getSearchProperties(qualifiedName);
+                List<OpenMetadataElement> elements = store.findMetadataElements(targetElement.getType().getTypeId(),


💬 2 similar findings have been found in this PR

NULL_DEREFERENCE: object returned by targetElement.getType() could be null and is dereferenced at line 62.

🔎 Expand here to view all instances of this finding

File Path Line Number

open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java 60

open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java 100

Visit the Lift Web Console to find more details in your report.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

Add sample deduplication connector

ba2c664

Signed-off-by: Raluca Popa <[email protected]>

popa-raluca requested review from lpalashevski and alexandra-bucur August 23, 2022 10:45

popa-raluca requested a review from mandy-chessell as a code owner August 23, 2022 10:45

planetf1 approved these changes Aug 24, 2022

View reviewed changes

lpalashevski approved these changes Aug 26, 2022

View reviewed changes

popa-raluca and others added 2 commits August 31, 2022 15:40

Rename sample connector and provider

cca5476

Signed-off-by: Raluca Popa <[email protected]>

Merge branch 'master' into dedup-connector

9918d1b

popa-raluca enabled auto-merge August 31, 2022 12:41

Merge branch 'master' into dedup-connector

df38922

popa-raluca merged commit ef4cbf5 into odpi:master Aug 31, 2022

sonatype-lift bot reviewed Aug 31, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sample deduplication connector #6847

Add sample deduplication connector #6847

popa-raluca commented Aug 23, 2022

planetf1 commented Aug 24, 2022

popa-raluca commented Aug 24, 2022

planetf1 commented Aug 24, 2022 •

edited by lcpopa

Loading

lpalashevski left a comment

sonatype-lift bot Aug 31, 2022

sonatype-lift bot Aug 31, 2022

sonatype-lift bot Aug 31, 2022

sonatype-lift bot Aug 31, 2022

sonatype-lift bot Aug 31, 2022

sonatype-lift bot Aug 31, 2022

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

	public void start() throws ConnectorCheckedException {
	public synchronized void start() throws ConnectorCheckedException {

File Path	Line Number
open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java	60
open-metadata-implementation/adapters/open-connectors/governance-action-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/governanceactions/remediation/QualifiedNamePeerDuplicateGovernanceActionConnector.java	62

Add sample deduplication connector #6847

Add sample deduplication connector #6847

Conversation

popa-raluca commented Aug 23, 2022

Description

Related Issue(s)

Testing

Release Notes & Documentation

Additional notes

planetf1 commented Aug 24, 2022

popa-raluca commented Aug 24, 2022

planetf1 commented Aug 24, 2022 • edited by lcpopa Loading

lpalashevski left a comment

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

sonatype-lift bot Aug 31, 2022

Choose a reason for hiding this comment

planetf1 commented Aug 24, 2022 •

edited by lcpopa

Loading