Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(search-test): update search tests from #10408 #11056

Merged
merged 2 commits into from
Aug 1, 2024

Conversation

david-leifker
Copy link
Collaborator

@david-leifker david-leifker commented Jul 31, 2024

Fixing merge conflict issues, from #10408

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Summary by CodeRabbit

  • New Features

    • Introduced the BulkProcessorProxyListener class to enhance bulk processing monitoring.
    • Added syncAfterWrite method to ensure all bulk operations are completed before proceeding.
    • Implemented getGraphQueryConfiguration method for improved graph query configuration management.
  • Bug Fixes

    • Removed flaky tests to improve reliability in both Neo4jGraphServiceTest and SearchGraphServiceTestBase.
  • Chores

    • Enhanced synchronization processes in various classes to ensure consistent test environments.
    • Made modifications to improve test methodologies and error handling across multiple test classes.

@david-leifker david-leifker changed the title feat(search-test): update serch tests from PR10408 feat(search-test): update search tests from #10408 Jul 31, 2024
@github-actions github-actions bot added product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Jul 31, 2024
Copy link
Contributor

coderabbitai bot commented Jul 31, 2024

Walkthrough

The recent changes involve significant enhancements to test configurations and synchronization mechanisms across multiple classes in the metadata management system. Key updates include the introduction of centralized configuration management for graph queries, improved methods for handling related entities, and enhancements to synchronization processes in test setups. These adjustments aim to improve test reliability, readability, and maintainability, ultimately laying a stronger foundation for future development.

Changes

Files Summary
metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java Introduced _graphQueryConfiguration for centralized configuration management; refactored test methods for clarity and utilized convertEdgesToRelatedEntities. Updated assertions for deduplication.
metadata-io/src/test/java/com/linkedin/metadata/graph/neo4j/Neo4jGraphServiceTest.java Removed testConcurrentRemoveEdgesFromNode and testConcurrentRemoveNodes due to issues with duplicate entries in concurrent operations.
metadata-io/src/test/java/com/linkedin/metadata/graph/search/SearchGraphServiceTestBase.java Replaced direct static configuration use with _graphQueryConfiguration; removed unreliable tests; introduced deduplicateRelatedEntitiesByRelationshipTypeAndDestination for improved entity handling.
metadata-io/src/test/java/com/linkedin/metadata/search/LineageServiceTestBase.java Added syncAfterWrite(getBulkProcessor()) in the wipe() method for enhanced synchronization.
metadata-io/src/test/java/com/linkedin/metadata/search/SearchServiceTestBase.java Modified wipe method to call syncAfterWrite(getBulkProcessor()) for better synchronization before clearing the service.
metadata-io/src/test/java/com/linkedin/metadata/search/TestEntityTestBase.java Enhanced wipe method with additional syncAfterWrite(getBulkProcessor()) calls for improved synchronization.
metadata-io/src/test/java/com/linkedin/metadata/search/indexbuilder/IndexBuilderTestBase.java Changed _indexClient and testDefaultBuilder from static to instance-level; transitioned wipe and getTestIndex methods to instance methods for better encapsulation.
metadata-io/src/test/java/com/linkedin/metadata/systemmetadata/SystemMetadataServiceTestBase.java Enhanced wipe method with added synchronization calls to ensure reliable test setup.
metadata-io/src/test/java/com/linkedin/metadata/timeseries/search/TimeseriesAspectServiceTestBase.java Modified testCountByFilterAfterDelete to throw a more general Exception instead of InterruptedException.
metadata-io/src/test/java/io/datahubproject/test/fixtures/search/SearchLineageFixtureConfiguration.java Updated configuration retrieval in graphService to use getGraphQueryConfiguration() instead of static defaults.
metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorProxyListener.java Introduced a new class to monitor unsent items during bulk processing and manage listener behavior.
metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java Added utility methods syncAfterWrite and replaceBulkProcessorListener for managing bulk operations.
metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java Updated syncAfterWrite to throw IOException and added getGraphQueryConfiguration for improved configuration management.
metadata-io/src/test/java/io/datahubproject/test/search/config/SearchTestContainerConfiguration.java Enhanced getBulkProcessor to include listener configuration for better monitoring of bulk processing events.
metadata-service/configuration/src/main/java/com/linkedin/metadata/config/search/GraphQueryConfiguration.java Removed static initialization block for testDefaults, shifting to explicit instance management of configuration parameters.

Sequence Diagram(s)

sequenceDiagram
    participant Test as Test Cases
    participant Config as GraphQueryConfiguration
    participant Service as Graph/Lineage Service
    
    Test->>Config: Retrieve configuration
    Config-->>Test: Provide _graphQueryConfiguration
    Test->>Service: Setup with configuration
    Service-->>Test: Initialized state
    Test->>Service: Execute tests
    Service-->>Test: Return results
Loading

🐰 In a world of code and tests so bright,
Where changes bring joy and bugs take flight,
We hop and we skip, with functions anew,
Improving our paths, as we leap and pursue!
So here’s to the changes, a cheer for the team,
May our tests run smoothly, like a sweet dream! 🌟


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between dffdef2 and 7f41839.

Files selected for processing (15)
  • metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java (7 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/graph/neo4j/Neo4jGraphServiceTest.java (1 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/graph/search/SearchGraphServiceTestBase.java (7 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/search/LineageServiceTestBase.java (1 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/search/SearchServiceTestBase.java (1 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/search/TestEntityTestBase.java (1 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/search/indexbuilder/IndexBuilderTestBase.java (3 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/systemmetadata/SystemMetadataServiceTestBase.java (1 hunks)
  • metadata-io/src/test/java/com/linkedin/metadata/timeseries/search/TimeseriesAspectServiceTestBase.java (1 hunks)
  • metadata-io/src/test/java/io/datahubproject/test/fixtures/search/SearchLineageFixtureConfiguration.java (2 hunks)
  • metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorProxyListener.java (1 hunks)
  • metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java (1 hunks)
  • metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java (4 hunks)
  • metadata-io/src/test/java/io/datahubproject/test/search/config/SearchTestContainerConfiguration.java (2 hunks)
  • metadata-service/configuration/src/main/java/com/linkedin/metadata/config/search/GraphQueryConfiguration.java (1 hunks)
Files skipped from review due to trivial changes (1)
  • metadata-io/src/test/java/com/linkedin/metadata/graph/neo4j/Neo4jGraphServiceTest.java
Additional comments not posted (32)
metadata-service/configuration/src/main/java/com/linkedin/metadata/config/search/GraphQueryConfiguration.java (2)

Line range hint 1-14:
LGTM!

The code changes are approved.


Line range hint 1-14:
Verify the impact of the removal of testDefaults.

The removal of the testDefaults static field might affect how default settings are handled in the application. Ensure that all references to testDefaults are updated accordingly.

Verification successful

No issues found related to the removal of testDefaults.

The searches for testDefaults did not yield any results, indicating that there are no remaining references or declarations in the codebase. The impact of its removal appears to be minimal.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify all references to `testDefaults` are updated accordingly.

# Test: Search for the usage of `testDefaults`. Expect: No occurrences of `testDefaults`.
rg --type java -A 5 $'testDefaults'

Length of output: 34


Script:

#!/bin/bash
# Description: Verify all references to `testDefaults` are updated accordingly.

# Test: Search for the usage of `testDefaults` in Java files.
rg --type java 'testDefaults'

# Test: Search for the declaration of `testDefaults` in Java files.
ast-grep --lang java --pattern '$_ testDefaults = $_;'

Length of output: 86

metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorProxyListener.java (2)

1-44: LGTM!

The code changes are approved.


34-42: Verify the correctness of the waitForBulkProcessed method.

Ensure that the waitForBulkProcessed method effectively waits for all bulk requests to be processed.

Verification successful

To verify the correctness of the waitForBulkProcessed method, we need to gather more information about the implementation and usage context. Specifically, we need to check:

  1. The complete implementation of the waitForBulkProcessed method in BulkProcessorProxyListener.
  2. How unsentItemsCounter is managed and updated.
  3. The context in which waitForBulkProcessed is used in BulkProcessorTestUtils.

Let's start by retrieving the complete implementation of the waitForBulkProcessed method and examining how unsentItemsCounter is managed.


To ensure the correctness of the waitForBulkProcessed method, we need to verify its usage context in BulkProcessorTestUtils. Let's retrieve the relevant section of the BulkProcessorTestUtils class where waitForBulkProcessed is called.


The waitForBulkProcessed method is correctly implemented and used.

The method effectively waits for all bulk requests to be processed by checking the unsentItemsCounter, which is properly managed. The usage context in BulkProcessorTestUtils further confirms its correct application to ensure all bulk requests are processed before proceeding with other tasks.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the correctness of the `waitForBulkProcessed` method.

# Test: Search for the usage of `waitForBulkProcessed`. Expect: Correct usage of the method.
rg --type java -A 5 $'waitForBulkProcessed'

Length of output: 1583


Script:

#!/bin/bash
# Retrieve the complete implementation of the `waitForBulkProcessed` method.
ast-grep --lang java --pattern $'class BulkProcessorProxyListener {
  $$$
  void waitForBulkProcessed() throws InterruptedException {
    $$$
  }
  $$$  
}'

# Search for the declaration and updates of `unsentItemsCounter` in the codebase.
rg --type java 'unsentItemsCounter'

Length of output: 911


Script:

#!/bin/bash
# Retrieve the relevant section of the BulkProcessorTestUtils class where waitForBulkProcessed is called.
rg --type java -A 10 'waitForBulkProcessed' metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java

Length of output: 598

metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java (4)

1-67: LGTM!

The code changes are approved.


58-66: Verify the correctness of the replaceBulkProcessorListener method.

Ensure that the replaceBulkProcessorListener method effectively replaces the bulk processor listener.

Verification successful

The replaceBulkProcessorListener method is correctly implemented.

The method effectively replaces the bulk processor listener by correctly retrieving and setting the necessary fields using reflection.

  • The bulkProcessor field is correctly retrieved from ESBulkProcessor.
  • The bulkRequestHandler field is correctly retrieved from bulkProcessor.
  • The listener field is correctly retrieved from bulkRequestHandler.
  • The listener field is successfully replaced with a new BulkProcessorProxyListener.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the correctness of the `replaceBulkProcessorListener` method.

# Test: Search for the usage of `replaceBulkProcessorListener`. Expect: Correct usage of the method.
rg --type java -A 5 $'replaceBulkProcessorListener'

Length of output: 2695


Script:

#!/bin/bash
# Verify the definitions and usages of bulkProcessor, bulkRequestHandler, and listener fields.

# Search for the bulkProcessor field in ESBulkProcessor class
rg --type java 'bulkProcessor' -A 5

# Search for the bulkRequestHandler field in BulkProcessor class
rg --type java 'bulkRequestHandler' -A 5

# Search for the listener field in bulkRequestHandler class
rg --type java 'listener' -A 5

# Search for the BulkProcessorProxyListener class
rg --type java 'class BulkProcessorProxyListener' -A 5

Length of output: 94146


29-44: Verify the correctness of the waitForCompletion method.

Ensure that the waitForCompletion method correctly handles waiting for completion of bulk operations.

Verification successful

Verify the correctness of the waitForCompletion method.

The waitForCompletion method is used within the syncAfterWrite method to ensure that all tasks are completed after a flush operation. The method appears to correctly handle waiting for the completion of bulk operations by checking if there are any tasks still running and waiting until they are completed.

  • The method ensures that it waits for tasks with actions "indices:,/put,*/update" to complete.
  • It uses a ListTasksRequest with setWaitForCompletion(true) and a timeout of 30 seconds.
  • It handles potential delays by sleeping for 5 milliseconds if tasks are still running.

No issues were found with the waitForCompletion method.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the correctness of the `waitForCompletion` method.

# Test: Search for the usage of `waitForCompletion`. Expect: Correct usage of the method.
rg --type java -A 5 $'waitForCompletion'

Length of output: 1500


Script:

#!/bin/bash
# Description: List all methods in the BulkProcessorTestUtils class to understand the context of waitForCompletion usage.

# Search for all method definitions in the BulkProcessorTestUtils class.
ast-grep --lang java --pattern 'class BulkProcessorTestUtils { $$$ }'

Length of output: 71


Script:

#!/bin/bash
# Description: List all methods in the BulkProcessorTestUtils class to understand the context of waitForCompletion usage.

# Search for all method definitions in the BulkProcessorTestUtils class.
rg --type java --no-heading -A 5 'class BulkProcessorTestUtils' -g 'BulkProcessorTestUtils.java'

Length of output: 820


17-27: Verify the correctness of the syncAfterWrite method.

Ensure that the syncAfterWrite method correctly handles synchronization after write operations.

Verification successful

The syncAfterWrite method correctly handles synchronization after write operations.

  • The method flushes the bulk processor, waits for bulk operations to be processed, waits for Elasticsearch to complete processing, and manually refreshes the indices.
  • The extensive usage of this method in various test files without any available logs indicating issues further supports its correctness.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the correctness of the `syncAfterWrite` method.

# Test: Search for the usage of `syncAfterWrite`. Expect: Correct usage of the method.
rg --type java -A 5 $'syncAfterWrite'

Length of output: 52973


Script:

#!/bin/bash
# Description: Check for any errors or exceptions related to the `syncAfterWrite` method in the test logs.

# Search for errors or exceptions in the test logs
rg --type log -A 5 'syncAfterWrite' | grep -iE 'error|exception'

Length of output: 213


Script:

#!/bin/bash
# Description: Locate log files in the repository and search for any errors or exceptions related to the `syncAfterWrite` method.

# Step 1: Locate log files in the repository
log_files=$(fd --type f --extension log)

# Step 2: Search the log files for errors or exceptions related to the `syncAfterWrite` method
if [ -n "$log_files" ]; then
  rg 'syncAfterWrite' $log_files | grep -iE 'error|exception'
else
  echo "No log files found."
fi

Length of output: 108

metadata-io/src/test/java/io/datahubproject/test/search/config/SearchTestContainerConfiguration.java (2)

3-3: Import added for replaceBulkProcessorListener.

The import statement for replaceBulkProcessorListener is correctly added.


69-83: Enhance bulk processor with a listener.

The addition of replaceBulkProcessorListener(esBulkProcessor) enhances the bulk processor by allowing it to handle events or callbacks related to bulk processing. This is a positive change for monitoring and debugging.

metadata-io/src/test/java/com/linkedin/metadata/systemmetadata/SystemMetadataServiceTestBase.java (1)

47-49: Synchronize bulk processor after clearing the client.

The addition of syncAfterWrite(getBulkProcessor()) ensures that all pending write operations to the bulk processor are completed before and after clearing the client. This enhances the reliability of the test setup by preventing potential race conditions or inconsistencies.

metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java (3)

18-18: Import added for GraphQueryConfiguration.

The import statement for GraphQueryConfiguration is correctly added.


46-48: Enhance syncAfterWrite method with IOException handling.

The addition of IOException in the throws clause of the syncAfterWrite method enhances its error handling capabilities, suggesting that it may now interact with components that can throw IO exceptions.


258-269: Add getGraphQueryConfiguration method.

The new method getGraphQueryConfiguration creates and returns an instance of GraphQueryConfiguration with initialized parameters. This addition provides new configuration capabilities for graph queries, which can impact how search operations are executed.

metadata-io/src/test/java/io/datahubproject/test/fixtures/search/SearchLineageFixtureConfiguration.java (1)

175-175: LGTM! But verify the getGraphQueryConfiguration method.

The code change replaces a direct reference to GraphQueryConfiguration.testDefaults with a call to getGraphQueryConfiguration(). This change is approved as it suggests a more dynamic approach to retrieving the configuration.

However, ensure that the getGraphQueryConfiguration method is correctly implemented and returns the expected configuration.

Verification successful

LGTM! But verify the getGraphQueryConfiguration method.

The code change replaces a direct reference to GraphQueryConfiguration.testDefaults with a call to getGraphQueryConfiguration(). This change is approved as it suggests a more dynamic approach to retrieving the configuration.

  • The getGraphQueryConfiguration method is correctly implemented and returns the expected configuration.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the implementation of `getGraphQueryConfiguration`.

# Test: Search for the `getGraphQueryConfiguration` method. Expect: Correct implementation.
rg --type java 'getGraphQueryConfiguration'

Length of output: 929


Script:

#!/bin/bash
# Extract the implementation of the `getGraphQueryConfiguration` method from `SearchTestUtils.java`.
ast-grep --lang java --pattern 'public static GraphQueryConfiguration getGraphQueryConfiguration() { $$$ }'

Length of output: 1301

metadata-io/src/test/java/com/linkedin/metadata/search/TestEntityTestBase.java (1)

73-75: LGTM! But verify the syncAfterWrite method.

The code change includes an additional invocation of syncAfterWrite(getBulkProcessor()) before and after the elasticSearchService.clear(opContext) method. This change is approved as it enhances the synchronization mechanism.

However, ensure that the syncAfterWrite method is correctly implemented and performs the expected synchronization.

Verification successful

LGTM! The syncAfterWrite method is correctly implemented and performs the expected synchronization.

The code change includes an additional invocation of syncAfterWrite(getBulkProcessor()) before and after the elasticSearchService.clear(opContext) method. This change enhances the synchronization mechanism.

  • The syncAfterWrite method in BulkProcessorTestUtils ensures that the bulk processor is flushed, waits for bulk processing to complete, and refreshes the indices.
  • The syncAfterWrite method in SearchTestUtils delegates to BulkProcessorTestUtils.syncAfterWrite, confirming its correctness.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the implementation of `syncAfterWrite`.

# Test: Search for the `syncAfterWrite` method. Expect: Correct implementation.
rg --type java 'syncAfterWrite'

Length of output: 9615


Script:

#!/bin/bash
# Extract the implementation of `syncAfterWrite` from `BulkProcessorTestUtils` and `SearchTestUtils`.

# Search for the implementation in `BulkProcessorTestUtils`.
ast-grep --lang java --pattern 'public static void syncAfterWrite(ESBulkProcessor bulkProcessor) { $$$ }' metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java

# Search for the implementation in `SearchTestUtils`.
ast-grep --lang java --pattern 'public static void syncAfterWrite(ESBulkProcessor bulkProcessor) { $$$ }' metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java

Length of output: 376


Script:

#!/bin/bash
# Extract the implementation of `syncAfterWrite` from `BulkProcessorTestUtils` and `SearchTestUtils`.

# Search for the implementation in `BulkProcessorTestUtils`.
rg -A 10 'public static void syncAfterWrite' metadata-io/src/test/java/io/datahubproject/test/search/BulkProcessorTestUtils.java

# Search for the implementation in `SearchTestUtils`.
rg -A 10 'public static void syncAfterWrite' metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java

Length of output: 1292


Script:

#!/bin/bash
# Extract the implementation of `syncAfterWrite` from `SearchTestUtils`.

# Search for the implementation in `SearchTestUtils`.
rg -A 10 'public static void syncAfterWrite' metadata-io/src/test/java/io/datahubproject/test/search/SearchTestUtils.java

Length of output: 533

metadata-io/src/test/java/com/linkedin/metadata/search/indexbuilder/IndexBuilderTestBase.java (4)

42-42: LGTM!

The _indexClient variable has been changed from static to instance-level. This change enhances encapsulation and allows for more flexible testing scenarios.


44-44: LGTM!

The testDefaultBuilder variable has been changed from static to instance-level. This change enhances encapsulation and allows for more flexible testing scenarios.


66-66: LGTM!

The wipe method has been changed from static to instance-level. This change aligns with the changes made to _indexClient and testDefaultBuilder, allowing the method to operate on instance variables.


89-89: LGTM!

The getTestIndex method has been changed from static to instance-level. This change aligns with the changes made to _indexClient and testDefaultBuilder, allowing the method to operate on instance variables.

metadata-io/src/test/java/com/linkedin/metadata/graph/search/SearchGraphServiceTestBase.java (5)

73-74: LGTM!

The initialization of _client with the correct configuration enhances encapsulation and consistency.


79-81: LGTM!

The synchronization steps before and after clearing the _client enhance the reliability of the test environment.


100-105: LGTM!

The initialization of ElasticSearchGraphService with the correct configuration enhances encapsulation and consistency.


Line range hint 117-121: LGTM!

The update ensures that the GraphService reflects the current configuration state accurately.


305-311: LGTM!

The new method enhances the clarity and efficiency of related entity handling using Java Streams for deduplication.

metadata-io/src/test/java/com/linkedin/metadata/search/SearchServiceTestBase.java (1)

111-113: LGTM!

The additional synchronization step ensures that any changes made during the test setup are fully reflected before the tests are executed.

metadata-io/src/test/java/com/linkedin/metadata/search/LineageServiceTestBase.java (1)

165-168: LGTM!

The additional synchronization step ensures that any changes made during the test setup are fully reflected before the tests are executed.

metadata-io/src/test/java/com/linkedin/metadata/timeseries/search/TimeseriesAspectServiceTestBase.java (1)

1294-1294: Verify the intention behind changing the thrown exception type.

The method's signature has been altered to change the thrown exception type from InterruptedException to a more general Exception. While this broadens the scope of exceptions that can be handled, ensure that this change is intentional and does not mask specific exception handling.

metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java (4)

1909-1914: Encapsulate conversion logic with convertEdgesToRelatedEntities.

The new method encapsulates the conversion logic, improving code readability and maintainability.

However, ensure that the method is used correctly in the tests.

Verification successful

Let's verify that the usages of convertEdgesToRelatedEntities are within test methods to ensure proper test coverage.


Encapsulate conversion logic with convertEdgesToRelatedEntities.

The new method encapsulates the conversion logic, improving code readability and maintainability. The method is correctly used within the test methods, ensuring proper test coverage.

  • GraphServiceTestBase.java:
    • testAddEdge
    • testFindRelatedEntitiesSourceEntityFilter
    • testFindRelatedEntitiesDestinationEntityFilter
    • testFindRelatedEntitiesSourceType
    • testFindRelatedEntitiesDestinationType
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the usage of `convertEdgesToRelatedEntities`.

# Test: Search for the usage of `convertEdgesToRelatedEntities`. Expect: Correct usage in the tests.
rg --type java 'convertEdgesToRelatedEntities'

Length of output: 541


Script:

#!/bin/bash
# Description: Verify that the usages of `convertEdgesToRelatedEntities` are within test methods.

# Search for test methods in the file to check if `convertEdgesToRelatedEntities` is used within them.
rg --type java '@Test' -A 10 metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java

Length of output: 11228


276-276: Centralize configuration management with _graphQueryConfiguration.

The introduction of _graphQueryConfiguration centralizes the configuration management, which improves maintainability.

However, ensure that the getGraphQueryConfiguration() method is correctly defined and imported.

Verification successful

The method getGraphQueryConfiguration is correctly defined, imported, and used.

  • The method getGraphQueryConfiguration is defined in SearchTestUtils.
  • It is correctly imported and used in GraphServiceTestBase.java.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the definition and import of `getGraphQueryConfiguration`.

# Test: Search for the method definition. Expect: Method should be defined.
rg --type java 'GraphQueryConfiguration getGraphQueryConfiguration'

Length of output: 217


Script:

#!/bin/bash
# Description: Verify the import and usage of `getGraphQueryConfiguration` in `GraphServiceTestBase`.

# Test: Search for the import statement in `GraphServiceTestBase.java`. Expect: The method should be imported.
rg --type java 'import io.datahubproject.test.search.SearchTestUtils;' metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java

# Test: Search for the usage of `getGraphQueryConfiguration` in `GraphServiceTestBase.java`. Expect: The method should be used.
rg --type java 'getGraphQueryConfiguration' metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java

Length of output: 463


1916-1918: Enhance test validation with deduplicateRelatedEntitiesByRelationshipTypeAndDestination.

The new method deduplicates related entities, enhancing test validation.

However, ensure that the method is used correctly in the tests.

Verification successful

The method deduplicateRelatedEntitiesByRelationshipTypeAndDestination is used correctly in the tests.

The method is utilized in assertions to compare deduplicated related entities against expected results, confirming its correct usage.

  • GraphServiceTestBase.java: Multiple occurrences
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the usage of `deduplicateRelatedEntitiesByRelationshipTypeAndDestination`.

# Test: Search for the usage of `deduplicateRelatedEntitiesByRelationshipTypeAndDestination`. Expect: Correct usage in the tests.
rg --type java 'deduplicateRelatedEntitiesByRelationshipTypeAndDestination'

Length of output: 1104


Script:

#!/bin/bash
# Extract lines around the usage of `deduplicateRelatedEntitiesByRelationshipTypeAndDestination` to verify correct usage.

# Extract 5 lines before and after each occurrence in `GraphServiceTestBase.java`
rg --type java 'deduplicateRelatedEntitiesByRelationshipTypeAndDestination' -C 5 metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java

Length of output: 1984


384-384: Ensure consistent configuration application in getLineagePopulatedGraphService.

The method now retrieves its multi-path search setting from _graphQueryConfiguration, ensuring consistent configuration application.

However, verify that _graphQueryConfiguration is correctly used throughout the code.

Verification successful

The _graphQueryConfiguration is correctly used throughout the code.

The GraphQueryConfiguration class contains the enableMultiPathSearch field, and the isEnableMultiPathSearch method is automatically generated by Lombok's @Data annotation. The usage of _graphQueryConfiguration is consistent and correct.

  • metadata-io/src/test/java/com/linkedin/metadata/graph/GraphServiceTestBase.java
  • metadata-io/src/test/java/com/linkedin/metadata/graph/search/SearchGraphServiceTestBase.java
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the usage of `_graphQueryConfiguration`.

# Test: Search for the usage of `_graphQueryConfiguration`. Expect: Correct usage throughout the code.
rg --type java '_graphQueryConfiguration'

Length of output: 1279


Script:

#!/bin/bash
# Description: Verify the implementation and usage of GraphQueryConfiguration and its isEnableMultiPathSearch method.

# Search for the GraphQueryConfiguration class definition.
ast-grep --lang java --pattern 'class GraphQueryConfiguration { $$$ }'

# Search for the isEnableMultiPathSearch method definition within the GraphQueryConfiguration class.
ast-grep --lang java --pattern 'class GraphQueryConfiguration { $$$ boolean isEnableMultiPathSearch() { $$$ } $$$ }'

Length of output: 191


Script:

#!/bin/bash
# Description: Verify the implementation and usage of GraphQueryConfiguration and its isEnableMultiPathSearch method.

# Search for the GraphQueryConfiguration class definition.
rg --type java 'class GraphQueryConfiguration'

# Search for the isEnableMultiPathSearch method definition within the GraphQueryConfiguration class.
rg --type java 'boolean isEnableMultiPathSearch'

Length of output: 248


Script:

#!/bin/bash
# Description: Display the contents of the GraphQueryConfiguration class to verify its implementation.

# Display the contents of the GraphQueryConfiguration class.
cat metadata-service/configuration/src/main/java/com/linkedin/metadata/config/search/GraphQueryConfiguration.java

Length of output: 784

@david-leifker david-leifker added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Jul 31, 2024
@anshbansal anshbansal merged commit c83907f into master Aug 1, 2024
54 of 65 checks passed
@anshbansal anshbansal deleted the search-test-update-fix branch August 1, 2024 08:18
arosanda added a commit to infobip/datahub that referenced this pull request Sep 23, 2024
* feat(forms) Handle deleting forms references when hard deleting forms (datahub-project#10820)

* refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2)  (datahub-project#10764)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* fix(ingestion/airflow-plugin): pipeline tasks discoverable in search (datahub-project#10819)

* feat(ingest/transformer): tags to terms transformer (datahub-project#10758)

Co-authored-by: Aseem Bansal <[email protected]>

* fix(ingestion/unity-catalog): fixed issue with profiling with GE turned on (datahub-project#10752)

Co-authored-by: Aseem Bansal <[email protected]>

* feat(forms) Add java SDK for form entity PATCH + CRUD examples (datahub-project#10822)

* feat(SDK) Add java SDK for structuredProperty entity PATCH + CRUD examples (datahub-project#10823)

* feat(SDK) Add StructuredPropertyPatchBuilder in python sdk and provide sample CRUD files (datahub-project#10824)

* feat(forms) Add CRUD endpoints to GraphQL for Form entities (datahub-project#10825)

* add flag for includeSoftDeleted in scroll entities API (datahub-project#10831)

* feat(deprecation) Return actor entity with deprecation aspect (datahub-project#10832)

* feat(structuredProperties) Add CRUD graphql APIs for structured property entities (datahub-project#10826)

* add scroll parameters to openapi v3 spec (datahub-project#10833)

* fix(ingest): correct profile_day_of_week implementation (datahub-project#10818)

* feat(ingest/glue): allow ingestion of empty databases from Glue (datahub-project#10666)

Co-authored-by: Harshal Sheth <[email protected]>

* feat(cli): add more details to get cli (datahub-project#10815)

* fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (datahub-project#10836)

* fix(ingestion): fix datajob patcher (datahub-project#10827)

* fix(smoke-test): add suffix in temp file creation (datahub-project#10841)

* feat(ingest/glue): add helper method to permit user or group ownership (datahub-project#10784)

* feat(): Show data platform instances in policy modal if they are set on the policy (datahub-project#10645)

Co-authored-by: Hendrik Richert <[email protected]>

* docs(patch): add patch documentation for how implementation works (datahub-project#10010)

Co-authored-by: John Joyce <[email protected]>

* fix(jar): add missing custom-plugin-jar task (datahub-project#10847)

* fix(): also check exceptions/stack trace when filtering log messages (datahub-project#10391)

Co-authored-by: John Joyce <[email protected]>

* docs(): Update posts.md (datahub-project#9893)

Co-authored-by: Hyejin Yoon <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* chore(ingest): update acryl-datahub-classify version (datahub-project#10844)

* refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (datahub-project#10828)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(restli): log aspect-not-found as a warning rather than as an error (datahub-project#10834)

* fix(ingest/nifi): remove duplicate upstream jobs (datahub-project#10849)

* fix(smoke-test): test access to create/revoke personal access tokens (datahub-project#10848)

* fix(smoke-test): missing test for move domain (datahub-project#10837)

* ci: update usernames to not considered for community (datahub-project#10851)

* env: change defaults for data contract visibility (datahub-project#10854)

* fix(ingest/tableau): quote special characters in external URL (datahub-project#10842)

* fix(smoke-test): fix flakiness of auto complete test

* ci(ingest): pin dask dependency for feast (datahub-project#10865)

* fix(ingestion/lookml): liquid template resolution and view-to-view cll (datahub-project#10542)

* feat(ingest/audit): add client id and version in system metadata props (datahub-project#10829)

* chore(ingest): Mypy 1.10.1 pin (datahub-project#10867)

* docs: use acryl-datahub-actions as expected python package to install (datahub-project#10852)

* docs: add new js snippet (datahub-project#10846)

* refactor(ingestion): remove company domain for security reason (datahub-project#10839)

* fix(ingestion/spark): Platform instance and column level lineage fix (datahub-project#10843)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(ingestion/tableau): optionally ingest multiple sites and create site containers (datahub-project#10498)

Co-authored-by: Yanik Häni <[email protected]>

* fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (datahub-project#10874)

* fix(manage-tokens): fix manage access token policy (datahub-project#10853)

* Batch get entity endpoints (datahub-project#10880)

* feat(system): support conditional write semantics (datahub-project#10868)

* fix(build): upgrade vercel builds to Node 20.x (datahub-project#10890)

* feat(ingest/lookml): shallow clone repos (datahub-project#10888)

* fix(ingest/looker): add missing dependency (datahub-project#10876)

* fix(ingest): only populate audit stamps where accurate (datahub-project#10604)

* fix(ingest/dbt): always encode tag urns (datahub-project#10799)

* fix(ingest/redshift): handle multiline alter table commands (datahub-project#10727)

* fix(ingestion/looker): column name missing in explore (datahub-project#10892)

* fix(lineage) Fix lineage source/dest filtering with explored per hop limit (datahub-project#10879)

* feat(conditional-writes): misc updates and fixes (datahub-project#10901)

* feat(ci): update outdated action (datahub-project#10899)

* feat(rest-emitter): adding async flag to rest emitter (datahub-project#10902)

Co-authored-by: Gabe Lyons <[email protected]>

* feat(ingest): add snowflake-queries source (datahub-project#10835)

* fix(ingest): improve `auto_materialize_referenced_tags_terms` error handling (datahub-project#10906)

* docs: add new company to adoption list (datahub-project#10909)

* refactor(redshift): Improve redshift error handling with new structured reporting system (datahub-project#10870)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* feat(ui) Finalize support for all entity types on forms (datahub-project#10915)

* Index ExecutionRequestResults status field (datahub-project#10811)

* feat(ingest): grafana connector (datahub-project#10891)

Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(gms) Add Form entity type to EntityTypeMapper (datahub-project#10916)

* feat(dataset): add support for external url in Dataset (datahub-project#10877)

* docs(saas-overview) added missing features to observe section (datahub-project#10913)

Co-authored-by: John Joyce <[email protected]>

* fix(ingest/spark): Fixing Micrometer warning (datahub-project#10882)

* fix(structured properties): allow application of structured properties without schema file (datahub-project#10918)

* fix(data-contracts-web) handle other schedule types (datahub-project#10919)

* fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (datahub-project#10866)

Co-authored-by: Harshal Sheth <[email protected]>

* Add feature flag for view defintions (datahub-project#10914)

Co-authored-by: Ethan Cartwright <[email protected]>

* feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (datahub-project#10884)

* fix(airflow): add error handling around render_template() (datahub-project#10907)

* feat(ingestion/sqlglot): add optional `default_dialect` parameter to sqlglot lineage (datahub-project#10830)

* feat(mcp-mutator): new mcp mutator plugin (datahub-project#10904)

* fix(ingest/bigquery): changes helper function to decode unicode scape sequences (datahub-project#10845)

* feat(ingest/postgres): fetch table sizes for profile (datahub-project#10864)

* feat(ingest/abs): Adding azure blob storage ingestion source (datahub-project#10813)

* fix(ingest/redshift): reduce severity of SQL parsing issues (datahub-project#10924)

* fix(build): fix lint fix web react (datahub-project#10896)

* fix(ingest/bigquery): handle quota exceeded for project.list requests (datahub-project#10912)

* feat(ingest): report extractor failures more loudly (datahub-project#10908)

* feat(ingest/snowflake): integrate snowflake-queries into main source (datahub-project#10905)

* fix(ingest): fix docs build (datahub-project#10926)

* fix(ingest/snowflake): fix test connection (datahub-project#10927)

* fix(ingest/lookml): add view load failures to cache (datahub-project#10923)

* docs(slack) overhauled setup instructions and screenshots (datahub-project#10922)

Co-authored-by: John Joyce <[email protected]>

* fix(airflow): Add comma parsing of owners to DataJobs (datahub-project#10903)

* fix(entityservice): fix merging sideeffects (datahub-project#10937)

* feat(ingest): Support System Ingestion Sources, Show and hide system ingestion sources with Command-S (datahub-project#10938)

Co-authored-by: John Joyce <[email protected]>

* chore() Set a default lineage filtering end time on backend when a start time is present (datahub-project#10925)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* Added relationships APIs to V3. Added these generic APIs to V3 swagger doc. (datahub-project#10939)

* docs: add learning center to docs (datahub-project#10921)

* doc: Update hubspot form id (datahub-project#10943)

* chore(airflow): add python 3.11 w/ Airflow 2.9 to CI (datahub-project#10941)

* fix(ingest/Glue): column upstream lineage between S3 and Glue (datahub-project#10895)

* fix(ingest/abs): split abs utils into multiple files (datahub-project#10945)

* doc(ingest/looker): fix doc for sql parsing documentation (datahub-project#10883)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(ingest/bigquery): Adding missing BigQuery types (datahub-project#10950)

* fix(ingest/setup): feast and abs source setup (datahub-project#10951)

* fix(connections) Harden adding /gms to connections in backend (datahub-project#10942)

* feat(siblings) Add flag to prevent combining siblings in the UI (datahub-project#10952)

* fix(docs): make graphql doc gen more automated (datahub-project#10953)

* feat(ingest/athena): Add option for Athena partitioned profiling (datahub-project#10723)

* fix(spark-lineage): default timeout for future responses (datahub-project#10947)

* feat(datajob/flow): add environment filter using info aspects (datahub-project#10814)

* fix(ui/ingest): correct privilege used to show tab (datahub-project#10483)

Co-authored-by: Kunal-kankriya <[email protected]>

* feat(ingest/looker): include dashboard urns in browse v2 (datahub-project#10955)

* add a structured type to batchGet in OpenAPI V3 spec (datahub-project#10956)

* fix(ui): scroll on the domain sidebar to show all domains (datahub-project#10966)

* fix(ingest/sagemaker): resolve incorrect variable assignment for SageMaker API call (datahub-project#10965)

* fix(airflow/build): Pinning mypy (datahub-project#10972)

* Fixed a bug where the OpenAPI V3 spec was incorrect. The bug was introduced in datahub-project#10939. (datahub-project#10974)

* fix(ingest/test): Fix for mssql integration tests (datahub-project#10978)

* fix(entity-service) exist check correctly extracts status (datahub-project#10973)

* fix(structuredProps) casing bug in StructuredPropertiesValidator (datahub-project#10982)

* bugfix: use anyOf instead of allOf when creating references in openapi v3 spec (datahub-project#10986)

* fix(ui): Remove ant less imports (datahub-project#10988)

* feat(ingest/graph): Add get_results_by_filter to DataHubGraph (datahub-project#10987)

* feat(ingest/cli): init does not actually support environment variables (datahub-project#10989)

* fix(ingest/graph): Update get_results_by_filter graphql query (datahub-project#10991)

* feat(ingest/spark): Promote beta plugin (datahub-project#10881)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(ingest): support domains in meta -> "datahub" section (datahub-project#10967)

* feat(ingest): add `check server-config` command (datahub-project#10990)

* feat(cli): Make consistent use of DataHubGraphClientConfig (datahub-project#10466)

Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig.
This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig.

I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled.

Fixes datahub-project#9705

* fix(ingest/s3): Fixing container creation when there is no folder in path (datahub-project#10993)

* fix(ingest/looker): support platform instance for dashboards & charts (datahub-project#10771)

* feat(ingest/bigquery): improve handling of information schema in sql parser (datahub-project#10985)

* feat(ingest): improve `ingest deploy` command (datahub-project#10944)

* fix(backend): allow excluding soft-deleted entities in relationship-queries; exclude soft-deleted members of groups (datahub-project#10920)

- allow excluding soft-deleted entities in relationship-queries
- exclude soft-deleted members of groups

* fix(ingest/looker): downgrade missing chart type log level (datahub-project#10996)

* doc(acryl-cloud): release docs for 0.3.4.x (datahub-project#10984)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: RyanHolstien <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Pedro Silva <[email protected]>

* fix(protobuf/build): Fix protobuf check jar script (datahub-project#11006)

* fix(ui/ingest): Support invalid cron jobs (datahub-project#10998)

* fix(ingest): fix graph config loading (datahub-project#11002)

Co-authored-by: Pedro Silva <[email protected]>

* feat(docs): Document __DATAHUB_TO_FILE_ directive (datahub-project#10968)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(graphql/upsertIngestionSource): Validate cron schedule; parse error in CLI (datahub-project#11011)

* feat(ece): support custom ownership type urns in ECE generation (datahub-project#10999)

* feat(assertion-v2): changed Validation tab to Quality and created new Governance tab (datahub-project#10935)

* fix(ingestion/glue): Add support for missing config options for profiling in Glue (datahub-project#10858)

* feat(propagation): Add models for schema field docs, tags, terms (datahub-project#2959) (datahub-project#11016)

Co-authored-by: Chris Collins <[email protected]>

* docs: standardize terminology to DataHub Cloud (datahub-project#11003)

* fix(ingestion/transformer): replace the externalUrl container (datahub-project#11013)

* docs(slack) troubleshoot docs (datahub-project#11014)

* feat(propagation): Add graphql API (datahub-project#11030)

Co-authored-by: Chris Collins <[email protected]>

* feat(propagation):  Add models for Action feature settings (datahub-project#11029)

* docs(custom properties): Remove duplicate from sidebar (datahub-project#11033)

* feat(models): Introducing Dataset Partitions Aspect (datahub-project#10997)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* feat(propagation): Add Documentation Propagation Settings (datahub-project#11038)

* fix(models): chart schema fields mapping, add dataHubAction entity, t… (datahub-project#11040)

* fix(ci): smoke test lint failures (datahub-project#11044)

* docs: fix learning center color scheme & typo (datahub-project#11043)

* feat: add cloud main page (datahub-project#11017)

Co-authored-by: Jay <[email protected]>

* feat(restore-indices): add additional step to also clear system metadata service (datahub-project#10662)

Co-authored-by: John Joyce <[email protected]>

* docs: fix typo (datahub-project#11046)

* fix(lint): apply spotless (datahub-project#11050)

* docs(airflow): example query to get datajobs for a dataflow (datahub-project#11034)

* feat(cli): Add run-id option to put sub-command (datahub-project#11023)

Adds an option to assign run-id to a given put command execution. 
This is useful when transformers do not exist for a given ingestion payload, we can follow up with custom metadata and assign it to an ingestion pipeline.

* fix(ingest): improve sql error reporting calls (datahub-project#11025)

* fix(airflow): fix CI setup (datahub-project#11031)

* feat(ingest/dbt): add experimental `prefer_sql_parser_lineage` flag (datahub-project#11039)

* fix(ingestion/lookml): enable stack-trace in lookml logs (datahub-project#10971)

* (chore): Linting fix (datahub-project#11015)

* chore(ci): update deprecated github actions (datahub-project#10977)

* Fix ALB configuration example (datahub-project#10981)

* chore(ingestion-base): bump base image packages (datahub-project#11053)

* feat(cli): Trim report of dataHubExecutionRequestResult to max GMS size (datahub-project#11051)

* fix(ingestion/lookml): emit dummy sql condition for lookml custom condition tag (datahub-project#11008)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(ingestion/powerbi): fix issue with broken report lineage (datahub-project#10910)

* feat(ingest/tableau): add retry on timeout (datahub-project#10995)

* change generate kafka connect properties from env (datahub-project#10545)

Co-authored-by: david-leifker <[email protected]>

* fix(ingest): fix oracle cronjob ingestion (datahub-project#11001)

Co-authored-by: david-leifker <[email protected]>

* chore(ci): revert update deprecated github actions (datahub-project#10977) (datahub-project#11062)

* feat(ingest/dbt-cloud): update metadata_endpoint inference (datahub-project#11041)

* build: Reduce size of datahub-frontend-react image by 50-ish% (datahub-project#10878)

Co-authored-by: david-leifker <[email protected]>

* fix(ci): Fix lint issue in datahub_ingestion_run_summary_provider.py (datahub-project#11063)

* docs(ingest): update developing-a-transformer.md (datahub-project#11019)

* feat(search-test): update search tests from datahub-project#10408 (datahub-project#11056)

* feat(cli): add aspects parameter to DataHubGraph.get_entity_semityped (datahub-project#11009)

Co-authored-by: Harshal Sheth <[email protected]>

* docs(airflow): update min version for plugin v2 (datahub-project#11065)

* doc(ingestion/tableau): doc update for derived permission (datahub-project#11054)

Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(py): remove dep on types-pkg_resources (datahub-project#11076)

* feat(ingest/mode): add option to exclude restricted (datahub-project#11081)

* fix(ingest): set lastObserved in sdk when unset (datahub-project#11071)

* doc(ingest): Update capabilities (datahub-project#11072)

* chore(vulnerability): Log Injection (datahub-project#11090)

* chore(vulnerability): Information exposure through a stack trace (datahub-project#11091)

* chore(vulnerability): Comparison of narrow type with wide type in loop condition (datahub-project#11089)

* chore(vulnerability): Insertion of sensitive information into log files (datahub-project#11088)

* chore(vulnerability): Risky Cryptographic Algorithm (datahub-project#11059)

* chore(vulnerability): Overly permissive regex range (datahub-project#11061)

Co-authored-by: Harshal Sheth <[email protected]>

* fix: update customer data (datahub-project#11075)

* fix(models): fixing the datasetPartition models (datahub-project#11085)

Co-authored-by: John Joyce <[email protected]>

* fix(ui): Adding view, forms GraphQL query, remove showing a fallback error message on unhandled GraphQL error (datahub-project#11084)

Co-authored-by: John Joyce <[email protected]>

* feat(docs-site): hiding learn more from cloud page (datahub-project#11097)

* fix(docs): Add correct usage of orFilters in search API docs (datahub-project#11082)

Co-authored-by: Jay <[email protected]>

* fix(ingest/mode): Regexp in mode name matcher didn't allow underscore (datahub-project#11098)

* docs: Refactor customer stories section (datahub-project#10869)

Co-authored-by: Jeff Merrick <[email protected]>

* fix(release): fix full/slim suffix on tag (datahub-project#11087)

* feat(config): support alternate hashing algorithm for doc id (datahub-project#10423)

Co-authored-by: david-leifker <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* fix(emitter): fix typo in get method of java kafka emitter (datahub-project#11007)

* fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (datahub-project#10898)

Co-authored-by: Harshal Sheth <[email protected]>

* chore: Update contributors list in PR labeler (datahub-project#11105)

* feat(ingest): tweak stale entity removal messaging (datahub-project#11064)

* fix(ingestion): enforce lastObserved timestamps in SystemMetadata (datahub-project#11104)

* fix(ingest/powerbi): fix broken lineage between chart and dataset (datahub-project#11080)

* feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view (datahub-project#11069)

* docs: update graphql docs on forms & structured properties (datahub-project#11100)

* test(search): search openAPI v3 test (datahub-project#11049)

* fix(ingest/tableau): prevent empty site content urls (datahub-project#11057)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(entity-client): implement client batch interface (datahub-project#11106)

* fix(snowflake): avoid reporting warnings/info for sys tables (datahub-project#11114)

* fix(ingest): downgrade column type mapping warning to info (datahub-project#11115)

* feat(api): add AuditStamp to the V3 API entity/aspect response (datahub-project#11118)

* fix(ingest/redshift): replace r'\n' with '\n' to avoid token error redshift serverless… (datahub-project#11111)

* fix(entiy-client): handle null entityUrn case for restli (datahub-project#11122)

* fix(sql-parser): prevent bad urns from alter table lineage (datahub-project#11092)

* fix(ingest/bigquery): use small batch size if use_tables_list_query_v2 is set (datahub-project#11121)

* fix(graphql): add missing entities to EntityTypeMapper and EntityTypeUrnMapper (datahub-project#10366)

* feat(ui): Changes to allow editable dataset name (datahub-project#10608)

Co-authored-by: Jay Kadambi <[email protected]>

* fix: remove saxo (datahub-project#11127)

* feat(mcl-processor): Update mcl processor hooks (datahub-project#11134)

* fix(openapi): fix openapi v2 endpoints & v3 documentation update

* Revert "fix(openapi): fix openapi v2 endpoints & v3 documentation update"

This reverts commit 573c1cb.

* docs(policies): updates to policies documentation (datahub-project#11073)

* fix(openapi): fix openapi v2 and v3 docs update (datahub-project#11139)

* feat(auth): grant type and acr values custom oidc parameters support (datahub-project#11116)

* fix(mutator): mutator hook fixes (datahub-project#11140)

* feat(search): support sorting on multiple fields (datahub-project#10775)

* feat(ingest): various logging improvements (datahub-project#11126)

* fix(ingestion/lookml): fix for sql parsing error (datahub-project#11079)

Co-authored-by: Harshal Sheth <[email protected]>

* feat(docs-site) cloud page spacing and content polishes (datahub-project#11141)

* feat(ui) Enable editing structured props on fields (datahub-project#11042)

* feat(tests): add md5 and last computed to testResult model (datahub-project#11117)

* test(openapi): openapi regression smoke tests (datahub-project#11143)

* fix(airflow): fix tox tests + update docs (datahub-project#11125)

* docs: add chime to adoption stories (datahub-project#11142)

* fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (datahub-project#11158)

* fix(kafka-setup): add missing script to image (datahub-project#11190)

* fix(config): fix hash algo config (datahub-project#11191)

* test(smoke-test): updates to smoke-tests (datahub-project#11152)

* fix(elasticsearch): refactor idHashAlgo setting (datahub-project#11193)

* chore(kafka): kafka version bump (datahub-project#11211)

* readd UsageStatsWorkUnit

* fix merge problems

* change logo

---------

Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: dushayntAW <[email protected]>
Co-authored-by: sagar-salvi-apptware <[email protected]>
Co-authored-by: Aseem Bansal <[email protected]>
Co-authored-by: Kevin Chun <[email protected]>
Co-authored-by: jordanjeremy <[email protected]>
Co-authored-by: skrydal <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>
Co-authored-by: david-leifker <[email protected]>
Co-authored-by: sid-acryl <[email protected]>
Co-authored-by: Julien Jehannet <[email protected]>
Co-authored-by: Hendrik Richert <[email protected]>
Co-authored-by: Hendrik Richert <[email protected]>
Co-authored-by: RyanHolstien <[email protected]>
Co-authored-by: Felix Lüdin <[email protected]>
Co-authored-by: Pirry <[email protected]>
Co-authored-by: Hyejin Yoon <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: cburroughs <[email protected]>
Co-authored-by: ksrinath <[email protected]>
Co-authored-by: Mayuri Nehate <[email protected]>
Co-authored-by: Kunal-kankriya <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: ipolding-cais <[email protected]>
Co-authored-by: Tamas Nemeth <[email protected]>
Co-authored-by: Shubham Jagtap <[email protected]>
Co-authored-by: haeniya <[email protected]>
Co-authored-by: Yanik Häni <[email protected]>
Co-authored-by: Gabe Lyons <[email protected]>
Co-authored-by: Gabe Lyons <[email protected]>
Co-authored-by: 808OVADOZE <[email protected]>
Co-authored-by: noggi <[email protected]>
Co-authored-by: Nicholas Pena <[email protected]>
Co-authored-by: Jay <[email protected]>
Co-authored-by: ethan-cartwright <[email protected]>
Co-authored-by: Ethan Cartwright <[email protected]>
Co-authored-by: Nadav Gross <[email protected]>
Co-authored-by: Patrick Franco Braz <[email protected]>
Co-authored-by: pie1nthesky <[email protected]>
Co-authored-by: Joel Pinto Mata (KPN-DSH-DEX team) <[email protected]>
Co-authored-by: Ellie O'Neil <[email protected]>
Co-authored-by: Ajoy Majumdar <[email protected]>
Co-authored-by: deepgarg-visa <[email protected]>
Co-authored-by: Tristan Heisler <[email protected]>
Co-authored-by: Andrew Sikowitz <[email protected]>
Co-authored-by: Davi Arnaut <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: amit-apptware <[email protected]>
Co-authored-by: Sam Black <[email protected]>
Co-authored-by: Raj Tekal <[email protected]>
Co-authored-by: Steffen Grohsschmiedt <[email protected]>
Co-authored-by: jaegwon.seo <[email protected]>
Co-authored-by: Renan F. Lima <[email protected]>
Co-authored-by: Matt Exchange <[email protected]>
Co-authored-by: Jonny Dixon <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: Pinaki Bhattacharjee <[email protected]>
Co-authored-by: Jeff Merrick <[email protected]>
Co-authored-by: skrydal <[email protected]>
Co-authored-by: AndreasHegerNuritas <[email protected]>
Co-authored-by: jayasimhankv <[email protected]>
Co-authored-by: Jay Kadambi <[email protected]>
Co-authored-by: David Leifker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops PR or Issue related to DataHub backend & deployment merge-pending-ci A PR that has passed review and should be merged once CI is green. product PR or Issue related to the DataHub UI/UX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants