Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/transformer): tags to terms transformer #10758

Conversation

sagar-salvi-apptware
Copy link
Contributor

@sagar-salvi-apptware sagar-salvi-apptware commented Jun 20, 2024

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Summary by CodeRabbit

  • New Features

    • Introduced a TagsToTermMapper transformer that maps specific tags to glossary terms in DataHub.
    • Added support for removing tags associated with specific resources.
  • Documentation

    • Added a new section on Tags to Term Mapping in the dataset transformer documentation, detailing usage and configuration.
  • Tests

    • Added unit tests to verify the functionality of the TagsToTermMapper transformer.

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Jun 20, 2024
@sagar-salvi-apptware sagar-salvi-apptware force-pushed the fix/ING-647/tags-to-term-transformer branch 2 times, most recently from 66a187c to 3b0eeef Compare June 20, 2024 22:33
@sagar-salvi-apptware sagar-salvi-apptware force-pushed the fix/ING-647/tags-to-term-transformer branch from 3b0eeef to c417c01 Compare June 21, 2024 05:19
@sagar-salvi-apptware sagar-salvi-apptware force-pushed the fix/ING-647/tags-to-term-transformer branch 2 times, most recently from 4165b16 to 3d1dfc3 Compare June 24, 2024 10:32
Copy link
Contributor

coderabbitai bot commented Jul 1, 2024

Walkthrough

The recent changes introduce a new transformer, TagsToTermMapper, that maps tags to glossary terms in DataHub, updating various documentation and module files. This update enhances the flexibility of dataset transformation processes by associating tags with glossary terms, adding new methods, and expanding unit tests to ensure functionality and reliability.

Changes

Files Change Summaries
metadata-ingestion/docs/transformer/dataset_transformer.md Added a section for Tags to Term Mapping to document the new transformer, tags_to_term. Updated the glossaryTerms list.
metadata-ingestion/setup.py Added TagsToTermMapper to the list of transformers.
metadata-ingestion/src/datahub/ingestion/graph/client.py Introduced a remove_tag method for removing tags from specific resources.
metadata-ingestion/src/datahub/ingestion/transformer/dataset_transformer.py Added TagTransformer and TagsToTermTransformer classes to handle tag and term transformations respectively.
metadata-ingestion/src/datahub/ingestion/transformer/tags_to_terms.py Added a new file to define the new transformer TagsToTermMapper and its associated methods.
metadata-ingestion/tests/unit/test_transform_dataset.py Expanded imports and added new test cases for TagsToTermMapper in the unit tests of dataset transformation.

Poem

In data's vast terrain, tags now transform
Linking to terms, a brand-new norm
Glossaries flourish, all mapped and neat
Bringing structure to datasets' feet
Transform with ease, let order unfurl
In DataHub's bright, evolving world 🌐✨


Tip

Early access features: enabled

We are currently testing the following features in early access:

  • OpenAI gpt-4o model for code reviews and chat: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available.

Note:

  • You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.
  • Please join our Discord Community to provide feedback and report issues.
  • OSS projects are currently opted into early access features by default.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@anshbansal anshbansal changed the title fix: implement tags-to-terms transformation based on tags feat(ingest/transformer): tags to terms transformer Jul 1, 2024
@anshbansal anshbansal added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Jul 1, 2024
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Outside diff range and nitpick comments (47)
metadata-ingestion/docs/transformer/dataset_transformer.md (46)

Line range hint 24-24: Fix grammatical errors.

Correct the sentence to: "Whether it should be considered a user or not."

- | `is_user`      |          | bool    | `true`   | Whether should be consider a user or not. If `false` then considered a group. |
+ | `is_user`      |          | bool    | `true`   | Whether it should be considered a user or not. If `false`, then considered a group. |
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 46-46: Add a comma after 'Hence'.

A comma is needed after the conjunctive adverb 'Hence'.

- Hence users `[email protected]` and `[email protected]` will be added as owners.
+ Hence, users `[email protected]` and `[email protected]` will be added as owners.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 68-68: Fix subject-verb agreement.

Change 'characters needs' to 'characters need'.

- Add owners, however some tag characters needs to replace with some other characters before extracting owner.
+ Add owners, however some tag characters need to be replaced with some other characters before extracting owner.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 80-80: Fix grammatical error.

Change 'owner type also need to extracted' to 'owner type also needs to be extracted'.

- Add owners, however owner type also need to extracted from tag pattern.
+ Add owners, however owner type also needs to be extracted from tag pattern.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 96-96: Correct article usage.

Change 'a Onwer URN' to 'an Owner URN'.

- Matches against a Onwer URN and remove the matching part from the Owner URN.
+ Matches against an Owner URN and remove the matching part from the Owner URN.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 128-128: Fix subject-verb agreement.

Change 'ownership type of the owners' to 'ownership type of the owners'.

- ownership type of the owners (either as enum or ownership type urn)
+ ownership type of the owners (either as enum or ownership type urns)
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 132-132: Add missing preposition.

Add 'to' before 'section'.

- please refer section [Relationship Between replace_existing And semantics]
+ please refer to section [Relationship Between replace_existing And semantics]
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 153-153: Complete the sentence.

Add a subject to form a complete sentence.

- `simple_add_dataset_ownership` can be configured in below different way
+ The `simple_add_dataset_ownership` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 179-179: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add owners, however keep the owners available for the dataset on DataHub GMS
+ Add owners, however, keep the owners available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 197-197: Fix subject-verb agreement.

Change 'ownership type of the owners' to 'ownership type of the owners'.

- ownership type of the owners (either as enum or ownership type urn)
+ ownership type of the owners (either as enum or ownership type urns)
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 216-216: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_ownership` can be configured in below different way
+ The `pattern_add_dataset_ownership` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 242-242: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add owner, however keep the owners available for the dataset on DataHub GMS
+ Add owner, however, keep the owners available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 271-271: Add a comma after 'Currently'.

A comma is needed after the conjunctive adverb 'Currently'.

- Currently only `urn` is supported.
+ Currently, only `urn` is supported.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 316-316: Complete the sentence.

Add a subject to form a complete sentence.

- `simple_add_dataset_tags` can be configured in below different way
+ The `simple_add_dataset_tags` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 370-370: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_tags` can be configured in below different way
+ The `pattern_add_dataset_tags` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 372-372: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add tags, however replace existing tags sent by ingestion source
+ Add tags, however, replace existing tags sent by ingestion source
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 449-449: Complete the sentence.

Add a subject to form a complete sentence.

- `add_dataset_tags` can be configured in below different way
+ The `add_dataset_tags` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 451-451: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add tags, however replace existing tags sent by ingestion source
+ Add tags, however, replace existing tags sent by ingestion source
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 497-497: Add a comma after 'name'.

A comma is needed after the phrase 'the database instance name'.

- browse path like the database instance name you can use this.
+ browse path like the database instance name, you can use this.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 530-530: Complete the sentence.

Add a subject to form a complete sentence.

- `set_dataset_browse_path` can be configured in below different way
+ The `set_dataset_browse_path` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 582-582: Complete the sentence.

Add a subject to form a complete sentence.

- `simple_add_dataset_terms` can be configured in below different way
+ The `simple_add_dataset_terms` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 594-594: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however overwrite the terms available for the dataset on DataHub GMS
+ Add terms, however, overwrite the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 604-604: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however keep the terms available for the dataset on DataHub GMS
+ Add terms, however, keep the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 635-635: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_terms` can be configured in below different way
+ The `pattern_add_dataset_terms` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 649-649: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however overwrite the terms available for the dataset on DataHub GMS
+ Add terms, however, overwrite the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 660-660: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however keep the terms available for the dataset on DataHub GMS
+ Add terms, however, keep the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 743-743: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_schema_terms` can be configured in below different way
+ The `pattern_add_dataset_schema_terms` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 756-756: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however overwrite the terms available for the dataset on DataHub GMS
+ Add terms, however, overwrite the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 767-767: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add terms, however keep the terms available for the dataset on DataHub GMS
+ Add terms, however, keep the terms available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 803-803: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_schema_tags` can be configured in below different way
+ The `pattern_add_dataset_schema_tags` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 805-805: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add tags, however replace existing tag sent by ingestion source
+ Add tags, however, replace existing tag sent by ingestion source
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 846-846: Add missing article.

Add 'the' before 'dataset entity'.

- `simple_add_dataset_properties` transformer assigns the properties to dataset entity from the configuration.
+ `simple_add_dataset_properties` transformer assigns the properties to the dataset entity from the configuration.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 847-847: Add a comma after 'Note'.

A comma is needed after the phrase 'Note'.

- Note in case of any key collision, the value in the config will overwrite the previous value.
+ Note, in case of any key collision, the value in the config will overwrite the previous value.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 859-859: Complete the sentence.

Add a subject to form a complete sentence.

- `simple_add_dataset_properties` can be configured in below different way
+ The `simple_add_dataset_properties` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 926-926: Complete the sentence.

Add a subject to form a complete sentence.

- `add_dataset_properties` can be configured in below different way
+ The `add_dataset_properties` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1018-1018: Complete the sentence.

Add a subject to form a complete sentence.

- `simple_add_dataset_domain` can be configured in below different way
+ The `simple_add_dataset_domain` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1020-1020: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however replace existing domains sent by ingestion source
+ Add domains, however, replace existing domains sent by ingestion source
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1030-1030: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however overwrite the domains available for the dataset on DataHub GMS
+ Add domains, however, overwrite the domains available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1040-1040: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however keep the domains available for the dataset on DataHub GMS
+ Add domains, however, keep the domains available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1055-1055: Fix grammatical error.

Change 'need to be apply' to 'need to be applied'.

- simple domain name or domain urn need to be apply on matching dataset urn.
+ simple domain name or domain urn need to be applied on matching dataset urn.
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1077-1077: Complete the sentence.

Add a subject to form a complete sentence.

- `pattern_add_dataset_domain` can be configured in below different way
+ The `pattern_add_dataset_domain` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1079-1079: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however replace existing domains sent by ingestion source
+ Add domains, however, replace existing domains sent by ingestion source
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1090-1090: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however overwrite the domains available for the dataset on DataHub GMS
+ Add domains, however, overwrite the domains available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1101-1101: Add a comma after 'however'.

A comma is needed after the adverb 'however'.

- Add domains, however keep the domains available for the dataset on DataHub GMS
+ Add domains, however, keep the domains available for the dataset on DataHub GMS
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1133-1133: Fix repeated word.

Remove the repeated word 'tag'.

- For example, instead of using the tag URN urn:li:tag:NeedsDocumentation,
+ For example, instead of using the tag URN urn:li:NeedsDocumentation,
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


Line range hint 1144-1144: Complete the sentence.

Add a subject to form a complete sentence.

- `domain_mapping_based_on_tags` can be configured in below different way
+ The `domain_mapping_based_on_tags` transformer can be configured in the following ways:
Tools
Markdownlint

11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)

metadata-ingestion/tests/unit/test_transform_dataset.py (1)

Line range hint 126-126: Avoid function calls in default arguments.

Do not perform function call models.StatusClass in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable.

-    aspects: Optional[List[Any]] = [models.StatusClass(removed=False)],
+    aspects: Optional[List[Any]] = None,
) -> models.MetadataChangeEventClass:
    if aspects is None:
        aspects = [models.StatusClass(removed=False)]
Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8edc94d and b5e3668.

Files selected for processing (6)
  • metadata-ingestion/docs/transformer/dataset_transformer.md (2 hunks)
  • metadata-ingestion/setup.py (1 hunks)
  • metadata-ingestion/src/datahub/ingestion/graph/client.py (1 hunks)
  • metadata-ingestion/src/datahub/ingestion/transformer/dataset_transformer.py (2 hunks)
  • metadata-ingestion/src/datahub/ingestion/transformer/tags_to_terms.py (1 hunks)
  • metadata-ingestion/tests/unit/test_transform_dataset.py (4 hunks)
Additional context used
Ruff
metadata-ingestion/src/datahub/ingestion/graph/client.py

317-317: Yoda conditions are discouraged, use aspect == aspect_type.ASPECT_NAME instead

Replace Yoda condition with aspect == aspect_type.ASPECT_NAME

(SIM300)


724-724: Use enumerate() for index variable entities_yielded in for loop

(SIM113)


1180-1180: Use of functools.lru_cache or functools.cache on methods can lead to memory leaks

(B019)


1282-1292: Use f-string instead of format call

Convert to f-string

(UP032)


1333-1333: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1352-1352: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1356-1365: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1385-1385: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1389-1405: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1424-1424: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1425-1425: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1429-1445: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1465-1465: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)

metadata-ingestion/tests/unit/test_transform_dataset.py

126-126: Do not perform function call models.StatusClass in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)


1901-1901: Do not perform function call PipelineContext in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

LanguageTool
metadata-ingestion/docs/transformer/dataset_transformer.md

[uncategorized] ~24-~24: Between ‘whether’ and ‘should’ a pronoun, proper noun or plural noun seems to be missing.
Context: ... | | bool | true | Whether should be consider a user or not. If false t...

(LACK_OF_PRP_OR_NOUN_FROM_WHETHER_SHOULD)


[grammar] ~24-~24: There may an error in the verb form ‘be consider’.
Context: ... | bool | true | Whether should be consider a user or not. If false then consider...

(MD_BE_NON_VBP)


[uncategorized] ~46-~46: A comma may be missing after the conjunctive/linking adverb ‘Hence’.
Context: ...attern will be converted into an owner. Hence users [email protected] and `[email protected]...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[uncategorized] ~68-~68: Possible missing comma found.
Context: ...ype:data_product" ``` - Add owners, however some tag characters needs to replace wi...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~68-~68: The verb form ‘needs’ does not seem to match the subject ‘characters’.
Context: ...Add owners, however some tag characters needs to replace with some other characters b...

(SUBJECT_VERB_AGREEMENT_PLURAL)


[uncategorized] ~80-~80: Possible missing comma found.
Context: ... "__": "_" ``` - Add owners, however owner type also need to extracted from ...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~80-~80: It appears that the verb form is incorrect or a word is missing.
Context: ...owners, however owner type also need to extracted from tag pattern. For example: from tag...

(HAS_TO_APPROVED_BY)


[misspelling] ~96-~96: Use “an” instead of ‘a’ if the following word starts with a vowel sound, e.g. ‘an article’, ‘an hour’.
Context: ...om the Owner URN(s) | Matches against a Onwer URN and remove the matching part ...

(EN_A_VS_AN)


[grammar] ~128-~128: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... string | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[uncategorized] ~132-~132: Possible missing preposition found.
Context: ...existingandsemantics`, please refer section [Relationship Between replace_existing ...

(AI_HYDRA_LEO_MISSING_TO)


[style] ~153-~153: To form a complete sentence, be sure to include a subject.
Context: ...R" ``` simple_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~179-~179: Possible missing comma found.
Context: ...ystem__producer" ``` - Add owners, however keep the owners available for the datas...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~197-~197: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[style] ~216-~216: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~242-~242: Possible missing comma found.
Context: ...system__producer" ``` - Add owner, however keep the owners available for the datas...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~271-~271: A comma may be missing after the conjunctive/linking adverb ‘Currently’.
Context: ... | Which field to extract tag from. Currently only urn is supported. | | `extract_...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[style] ~316-~316: To form a complete sentence, be sure to include a subject.
Context: ...egacy" ``` simple_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~370-~370: To form a complete sentence, be sure to include a subject.
Context: ...ion"] ``` pattern_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~372-~372: Possible missing comma found.
Context: ...ed in below different way - Add tags, however replace existing tags sent by ingestion...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~449-~449: To form a complete sentence, be sure to include a subject.
Context: ...lling-the-package). add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~451-~451: Possible missing comma found.
Context: ...ed in below different way - Add tags, however replace existing tags sent by ingestion...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~497-~497: Possible missing comma found.
Context: ... browse path like the database instance name you can use this. ```yaml transformers:...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~530-~530: To form a complete sentence, be sure to include a subject.
Context: ...e transform. set_dataset_browse_path can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~582-~582: To form a complete sentence, be sure to include a subject.
Context: ...ress" ``` simple_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~594-~594: Possible missing comma found.
Context: ...ssaryTerm:Address" ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~604-~604: Possible missing comma found.
Context: ...ssaryTerm:Address" ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~635-~635: To form a complete sentence, be sure to include a subject.
Context: ...de"] ``` pattern_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~649-~649: Possible missing comma found.
Context: ...Term:PostalCode"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~660-~660: Possible missing comma found.
Context: ...yTerm:PostalCode"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~697-~697: To form a complete sentence, be sure to include a subject.
Context: ... - "tag_name" ``` tags_to_term can be configured in below different way -...

(MISSING_IT_THERE)


[style] ~743-~743: To form a complete sentence, be sure to include a subject.
Context: ...``` pattern_add_dataset_schema_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~756-~756: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~767-~767: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~803-~803: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_schema_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~805-~805: Possible missing comma found.
Context: ...ed in below different way - Add tags, however replace existing tag sent by ingestion ...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~846-~846: Possible missing article found.
Context: ...transformer assigns the properties to dataset entity from the configuration.propert...

(AI_HYDRA_LEO_MISSING_THE)


[uncategorized] ~847-~847: Possible missing comma found.
Context: ...field is a dictionary of string values. Note in case of any key collision, the value...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~859-~859: To form a complete sentence, be sure to include a subject.
Context: ... ``` simple_add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~926-~926: To form a complete sentence, be sure to include a subject.
Context: ...perties ``` add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~1018-~1018: To form a complete sentence, be sure to include a subject.
Context: ...ering ``` simple_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1020-~1020: Possible missing comma found.
Context: ...in below different way - Add domains, however replace existing domains sent by ingest...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1030-~1030: Possible missing comma found.
Context: ...urn:li:domain:hr" ``` - Add domains, however overwrite the domains available for the...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1040-~1040: Possible missing comma found.
Context: ...rn:li:domain:hr" ``` - Add domains, however keep the domains available for the data...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~1055-~1055: The past participle is required after “to be”.
Context: ...le domain name or domain urn need to be apply on matching dataset urn. | | `replace_e...

(BE_VBP_IN)


[style] ~1077-~1077: To form a complete sentence, be sure to include a subject.
Context: ...e"] ``` pattern_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1079-~1079: Possible missing comma found.
Context: ...in below different way - Add domains, however replace existing domains sent by ingest...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1090-~1090: Possible missing comma found.
Context: ...omain:finance"] ``` - Add domains, however overwrite the domains available for the...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1101-~1101: Possible missing comma found.
Context: ...omain:finance"] ``` - Add domains, however keep the domains available for the data...

(AI_HYDRA_LEO_MISSING_COMMA)


[duplication] ~1133-~1133: Possible typo: you repeated a word
Context: ... For example, instead of using the tag URN urn:li:tag:NeedsDocumentation, you should s...

(ENGLISH_WORD_REPEAT_RULE)


[style] ~1144-~1144: To form a complete sentence, be sure to include a subject.
Context: ...n" ``` domain_mapping_based_on_tags can be configured in below different way -...

(MISSING_IT_THERE)


[typographical] ~1237-~1237: The word “however” is an adverb that can’t be used like a conjunction, and therefore needs to be separated from the sentence.
Context: ...ontext of simple_add_dataset_ownership, however it is applicable for all dataset transf...

(HOWEVER_SENTENCE)


[uncategorized] ~1364-~1364: Possible missing comma found.
Context: ..._dict) return cls(config, ctx) ``` Next we need to tell the helper classes whic...

(AI_HYDRA_LEO_MISSING_COMMA)


[typographical] ~1374-~1374: Consider adding a comma after ‘Finally’ for more clarity.
Context: ...lf) -> str: return "ownership" ``` Finally we need to implement the `transform_asp...

(RB_LY_COMMA)

Markdownlint
metadata-ingestion/docs/transformer/dataset_transformer.md

4-4: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


7-7: Expected: 0 or 2; Actual: 95
Trailing spaces

(MD009, no-trailing-spaces)


113-113: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


123-123: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


153-153: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


192-192: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


216-216: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


288-288: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


316-316: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


370-370: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


449-449: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


518-518: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


530-530: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


568-568: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


582-582: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


635-635: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


743-743: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


803-803: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


859-859: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


926-926: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


991-991: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1018-1018: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1051-1051: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1059-1059: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1063-1063: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1077-1077: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1192-1192: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1248-1248: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1255-1255: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1261-1261: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1268-1268: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1302-1302: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1402-1402: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1404-1404: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1426-1426: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1427-1427: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1417-1417: Column: 1
Hard tabs

(MD010, no-hard-tabs)


1517-1517: Column: 1
Hard tabs

(MD010, no-hard-tabs)


95-95: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


152-152: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


294-294: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


786-786: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


962-962: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


979-979: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


990-990: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1017-1017: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1113-1113: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1114-1114: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1143-1143: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1270-1270: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1271-1271: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


4-4: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


19-19: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


89-89: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


107-107: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


192-192: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


255-255: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


295-295: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


560-560: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


615-615: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


673-673: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


721-721: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


722-722: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


722-722: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


778-778: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


778-778: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


838-838: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


838-838: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


892-892: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


893-893: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


893-893: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


955-955: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


956-956: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


956-956: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


973-973: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


974-974: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


974-974: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


991-991: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


992-992: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


992-992: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1051-1051: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1052-1052: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1052-1052: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1115-1115: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1116-1116: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1167-1167: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1168-1168: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1168-1168: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1186-1186: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1187-1187: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1187-1187: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1206-1206: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1207-1207: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1207-1207: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1236-1236: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


4-4: null
Multiple top-level headings in the same document

(MD025, single-title, single-h1)


51-51: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


58-58: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


60-60: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


67-67: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


69-69: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


79-79: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


81-81: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


122-122: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


156-156: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


166-166: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


168-168: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


178-178: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


180-180: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


219-219: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


229-229: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


231-231: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


241-241: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


243-243: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


319-319: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


327-327: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


329-329: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


337-337: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


339-339: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


347-347: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


373-373: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


382-382: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


384-384: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


393-393: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


395-395: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


404-404: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


446-446: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


452-452: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


458-458: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


460-460: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


466-466: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


468-468: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


474-474: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


489-489: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


498-498: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


504-504: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


508-508: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


515-515: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


520-520: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


527-527: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


533-533: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


540-540: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


542-542: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


549-549: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


551-551: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


585-585: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


593-593: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


595-595: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


603-603: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


605-605: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


638-638: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


648-648: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


650-650: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


659-659: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


661-661: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


700-700: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


709-709: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


711-711: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


746-746: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


755-755: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


757-757: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


766-766: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


768-768: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


777-777: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


806-806: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


815-815: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


817-817: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


826-826: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


828-828: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


837-837: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


862-862: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


870-870: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


872-872: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


880-880: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


882-882: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


929-929: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


936-936: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


938-938: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


945-945: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


947-947: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


981-981: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1008-1008: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1021-1021: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1029-1029: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1031-1031: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1039-1039: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1041-1041: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1080-1080: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1089-1089: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1091-1091: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1100-1100: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1102-1102: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1147-1147: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1155-1155: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1157-1157: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1233-1233: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


43-43: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


289-289: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


347-347: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


404-404: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


474-474: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


484-484: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


699-699: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


777-777: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


837-837: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1245-1245: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1251-1251: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1258-1258: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1264-1264: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: Expected: leading_and_trailing; Actual: leading_only; Missing trailing pipe
Table pipe style

(MD055, table-pipe-style)

Additional comments not posted (16)
metadata-ingestion/src/datahub/ingestion/transformer/dataset_transformer.py (4)

30-31: Class Documentation

The class-level docstring is clear and concise.


33-35: Constructor Initialization

The constructor correctly calls the superclass initializer.


36-37: Entity Types Method

The entity_types method correctly returns the list of entity types.


143-145: Aspect Name Method

The aspect_name method correctly returns the aspect name.

metadata-ingestion/src/datahub/ingestion/transformer/tags_to_terms.py (7)

21-23: Class Documentation

The class-level docstring is missing. Consider adding a docstring to describe the purpose and usage of this class.

class TagsToTermMapperConfig(TransformerSemanticsConfigModel):
    """
    Configuration model for TagsToTermMapper.
    """
    tags: List[str]

28-32: Constructor Initialization

The constructor correctly initializes the ctx and config attributes.


33-37: Create Method

The create method correctly parses the configuration dictionary and returns an instance of TagsToTermMapper.


39-59: Merge with Server Glossary Terms

The _merge_with_server_glossary_terms method correctly handles the merging of terms.


62-67: Get Tags from Global Tags

The get_tags_from_global_tags method correctly extracts tags from GlobalTagsClass.


70-82: Get Tags from Schema Metadata

The get_tags_from_schema_metadata method correctly extracts tags from SchemaMetadataClass.


84-145: Transform Aspect

The transform_aspect method correctly handles the transformation of aspects by mapping tags to glossary terms and updating the glossary terms.

metadata-ingestion/setup.py (1)

718-718: Entry Point Addition

The new transformer tags_to_term has been correctly added to the entry points.

metadata-ingestion/tests/unit/test_transform_dataset.py (4)

3761-3796: LGTM!

The function test_tags_to_terms_with_no_matching_terms is well-structured and verifies the expected behavior correctly.


3798-3826: LGTM!

The function test_tags_to_terms_with_missing_tags is well-structured and verifies the expected behavior correctly.


3829-3867: LGTM!

The function test_tags_to_terms_with_partial_match is well-structured and verifies the expected behavior correctly.


Line range hint 135-140: LGTM!

The function make_generic_dataset_mcp is well-structured and creates the MCP correctly.

metadata-ingestion/src/datahub/ingestion/graph/client.py Outdated Show resolved Hide resolved
metadata-ingestion/docs/transformer/dataset_transformer.md Outdated Show resolved Hide resolved
metadata-ingestion/tests/unit/test_transform_dataset.py Outdated Show resolved Hide resolved
metadata-ingestion/tests/unit/test_transform_dataset.py Outdated Show resolved Hide resolved
metadata-ingestion/tests/unit/test_transform_dataset.py Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Outside diff range and nitpick comments (1)
metadata-ingestion/tests/unit/test_transform_dataset.py (1)

Line range hint 126-126: Avoid using function calls in default argument values.

Do not perform function call models.StatusClass in argument defaults; instead, perform the call within the function.

def make_generic_dataset(
    entity_urn: str = "urn:li:dataset:(urn:li:dataPlatform:bigquery,example1,PROD)",
    aspects: Optional[List[Any]] = None,
) -> models.MetadataChangeEventClass:
    if aspects is None:
        # Default to a status aspect if none is provided.
-        aspects = [models.StatusClass(removed=False)]
+        aspects = [default_status_aspect]
    return models.MetadataChangeEventClass(
        proposedSnapshot=models.DatasetSnapshotClass(
            urn=entity_urn,
            aspects=aspects,
        ),
    )

default_status_aspect = models.StatusClass(removed=False)
Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between b5e3668 and 12fd649.

Files selected for processing (3)
  • metadata-ingestion/docs/transformer/dataset_transformer.md (2 hunks)
  • metadata-ingestion/src/datahub/ingestion/graph/client.py (1 hunks)
  • metadata-ingestion/tests/unit/test_transform_dataset.py (4 hunks)
Additional context used
Ruff
metadata-ingestion/src/datahub/ingestion/graph/client.py

317-317: Yoda conditions are discouraged, use aspect == aspect_type.ASPECT_NAME instead

Replace Yoda condition with aspect == aspect_type.ASPECT_NAME

(SIM300)


724-724: Use enumerate() for index variable entities_yielded in for loop

(SIM113)


1180-1180: Use of functools.lru_cache or functools.cache on methods can lead to memory leaks

(B019)


1331-1331: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1350-1350: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1354-1363: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1383-1383: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1387-1403: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1422-1422: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1423-1423: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1427-1443: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1463-1463: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)

metadata-ingestion/tests/unit/test_transform_dataset.py

126-126: Do not perform function call models.StatusClass in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

LanguageTool
metadata-ingestion/docs/transformer/dataset_transformer.md

[uncategorized] ~24-~24: Between ‘whether’ and ‘should’ a pronoun, proper noun or plural noun seems to be missing.
Context: ... | | bool | true | Whether should be consider a user or not. If false t...

(LACK_OF_PRP_OR_NOUN_FROM_WHETHER_SHOULD)


[grammar] ~24-~24: There may an error in the verb form ‘be consider’.
Context: ... | bool | true | Whether should be consider a user or not. If false then consider...

(MD_BE_NON_VBP)


[uncategorized] ~46-~46: A comma may be missing after the conjunctive/linking adverb ‘Hence’.
Context: ...attern will be converted into an owner. Hence users [email protected] and `[email protected]...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[uncategorized] ~59-~59: Possible missing comma found.
Context: ...main: "email.com" ``` - Add owners, however owner type and owner type urn wanted to...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~68-~68: The verb form ‘needs’ does not seem to match the subject ‘characters’.
Context: ...Add owners, however some tag characters needs to replace with some other characters b...

(SUBJECT_VERB_AGREEMENT_PLURAL)


[grammar] ~80-~80: It appears that the verb form is incorrect or a word is missing.
Context: ...owners, however owner type also need to extracted from tag pattern. For example: from tag...

(HAS_TO_APPROVED_BY)


[misspelling] ~96-~96: Use “an” instead of ‘a’ if the following word starts with a vowel sound, e.g. ‘an article’, ‘an hour’.
Context: ...om the Owner URN(s) | Matches against a Onwer URN and remove the matching part ...

(EN_A_VS_AN)


[grammar] ~128-~128: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... string | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[style] ~153-~153: To form a complete sentence, be sure to include a subject.
Context: ...R" ``` simple_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~179-~179: Possible missing comma found.
Context: ...ystem__producer" ``` - Add owners, however keep the owners available for the datas...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~197-~197: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[style] ~216-~216: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~242-~242: Possible missing comma found.
Context: ...system__producer" ``` - Add owner, however keep the owners available for the datas...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~256-~256: Possible missing comma found.
Context: ...clear existing owners sent by ingestion source we can use the `simple_remove_dataset_o...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~271-~271: A comma may be missing after the conjunctive/linking adverb ‘Currently’.
Context: ... | Which field to extract tag from. Currently only urn is supported. | | `extract_...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[style] ~316-~316: To form a complete sentence, be sure to include a subject.
Context: ...egacy" ``` simple_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~370-~370: To form a complete sentence, be sure to include a subject.
Context: ...ion"] ``` pattern_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~383-~383: Possible missing comma found.
Context: ...eedsDocumentation"] ``` - Add tags, however overwrite the tags available for the da...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~394-~394: Possible missing comma found.
Context: ...eedsDocumentation"] ``` - Add tags, however keep the tags available for the dataset...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~449-~449: To form a complete sentence, be sure to include a subject.
Context: ...lling-the-package). add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~467-~467: Possible missing comma found.
Context: ...e>.<your_function>" ``` - Add tags, however keep the tags available for the dataset...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~530-~530: To form a complete sentence, be sure to include a subject.
Context: ...e transform. set_dataset_browse_path can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~541-~541: Possible missing comma found.
Context: ...ATASET_PARTS ``` - Add browsePath, however overwrite the browsePath available for ...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~582-~582: To form a complete sentence, be sure to include a subject.
Context: ...ress" ``` simple_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~594-~594: Possible missing comma found.
Context: ...ssaryTerm:Address" ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~604-~604: Possible missing comma found.
Context: ...ssaryTerm:Address" ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~635-~635: To form a complete sentence, be sure to include a subject.
Context: ...de"] ``` pattern_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~649-~649: Possible missing comma found.
Context: ...Term:PostalCode"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~660-~660: Possible missing comma found.
Context: ...yTerm:PostalCode"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~743-~743: To form a complete sentence, be sure to include a subject.
Context: ...``` pattern_add_dataset_schema_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~756-~756: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~767-~767: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~803-~803: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_schema_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~816-~816: Possible missing comma found.
Context: ...eedsDocumentation"] ``` - Add tags, however overwrite the tags available for the da...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~827-~827: Possible missing comma found.
Context: ...eedsDocumentation"] ``` - Add tags, however keep the tags available for the dataset...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~847-~847: Possible missing comma found.
Context: ...field is a dictionary of string values. Note in case of any key collision, the value...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~859-~859: To form a complete sentence, be sure to include a subject.
Context: ... ``` simple_add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~926-~926: To form a complete sentence, be sure to include a subject.
Context: ...perties ``` add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~999-~999: Possible missing preposition found.
Context: ...existingandsemantics`, please refer section [Relationship Between replace_existing ...

(AI_HYDRA_LEO_MISSING_TO)


[uncategorized] ~1003-~1003: Possible missing article found.
Context: ... we’d like to add a series of domain to dataset, in this case you can use `simple_add_d...

(AI_HYDRA_LEO_MISSING_A)


[style] ~1018-~1018: To form a complete sentence, be sure to include a subject.
Context: ...ering ``` simple_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1020-~1020: Possible missing comma found.
Context: ...in below different way - Add domains, however replace existing domains sent by ingest...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1030-~1030: Possible missing comma found.
Context: ...urn:li:domain:hr" ``` - Add domains, however overwrite the domains available for the...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~1055-~1055: The past participle is required after “to be”.
Context: ...le domain name or domain urn need to be apply on matching dataset urn. | | `replace_e...

(BE_VBP_IN)


[style] ~1077-~1077: To form a complete sentence, be sure to include a subject.
Context: ...e"] ``` pattern_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1090-~1090: Possible missing comma found.
Context: ...omain:finance"] ``` - Add domains, however overwrite the domains available for the...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1101-~1101: Possible missing comma found.
Context: ...omain:finance"] ``` - Add domains, however keep the domains available for the data...

(AI_HYDRA_LEO_MISSING_COMMA)


[duplication] ~1133-~1133: Possible typo: you repeated a word
Context: ... For example, instead of using the tag URN urn:li:tag:NeedsDocumentation, you should s...

(ENGLISH_WORD_REPEAT_RULE)


[style] ~1144-~1144: To form a complete sentence, be sure to include a subject.
Context: ...n" ``` domain_mapping_based_on_tags can be configured in below different way -...

(MISSING_IT_THERE)


[typographical] ~1237-~1237: The word “however” is an adverb that can’t be used like a conjunction, and therefore needs to be separated from the sentence.
Context: ...ontext of simple_add_dataset_ownership, however it is applicable for all dataset transf...

(HOWEVER_SENTENCE)


[uncategorized] ~1364-~1364: Possible missing comma found.
Context: ..._dict) return cls(config, ctx) ``` Next we need to tell the helper classes whic...

(AI_HYDRA_LEO_MISSING_COMMA)


[typographical] ~1374-~1374: Consider adding a comma after ‘Finally’ for more clarity.
Context: ...lf) -> str: return "ownership" ``` Finally we need to implement the `transform_asp...

(RB_LY_COMMA)

Markdownlint
metadata-ingestion/docs/transformer/dataset_transformer.md

4-4: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


7-7: Expected: 0 or 2; Actual: 95
Trailing spaces

(MD009, no-trailing-spaces)


113-113: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


123-123: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


153-153: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


192-192: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


216-216: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


288-288: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


316-316: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


370-370: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


449-449: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


518-518: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


530-530: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


568-568: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


582-582: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


635-635: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


743-743: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


803-803: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


859-859: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


926-926: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


991-991: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1018-1018: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1051-1051: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1059-1059: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1063-1063: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1077-1077: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1192-1192: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1248-1248: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1255-1255: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1261-1261: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1268-1268: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1302-1302: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1402-1402: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1404-1404: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1426-1426: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1427-1427: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1417-1417: Column: 1
Hard tabs

(MD010, no-hard-tabs)


1517-1517: Column: 1
Hard tabs

(MD010, no-hard-tabs)


95-95: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


152-152: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


294-294: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


786-786: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


962-962: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


979-979: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


990-990: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1017-1017: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1113-1113: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1114-1114: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1143-1143: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1270-1270: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1271-1271: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


4-4: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


19-19: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


89-89: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


107-107: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


192-192: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


255-255: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


295-295: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


560-560: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


615-615: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


673-673: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


721-721: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


722-722: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


722-722: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


778-778: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


778-778: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


838-838: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


838-838: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


892-892: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


893-893: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


893-893: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


955-955: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


956-956: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


956-956: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


973-973: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


974-974: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


974-974: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


991-991: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


992-992: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


992-992: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1051-1051: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1052-1052: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1052-1052: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1115-1115: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1116-1116: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1167-1167: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1168-1168: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1168-1168: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1186-1186: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1187-1187: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1187-1187: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1206-1206: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1207-1207: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1207-1207: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1236-1236: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


4-4: null
Multiple top-level headings in the same document

(MD025, single-title, single-h1)


51-51: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


58-58: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


60-60: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


67-67: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


69-69: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


79-79: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


81-81: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


122-122: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


156-156: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


166-166: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


168-168: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


178-178: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


180-180: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


219-219: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


229-229: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


231-231: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


241-241: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


243-243: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


319-319: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


327-327: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


329-329: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


337-337: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


339-339: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


347-347: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


373-373: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


382-382: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


384-384: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


393-393: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


395-395: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


404-404: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


446-446: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


452-452: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


458-458: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


460-460: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


466-466: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


468-468: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


474-474: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


489-489: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


498-498: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


504-504: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


508-508: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


515-515: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


520-520: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


527-527: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


533-533: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


540-540: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


542-542: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


549-549: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


551-551: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


585-585: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


593-593: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


595-595: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


603-603: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


605-605: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


638-638: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


648-648: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


650-650: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


659-659: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


661-661: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


700-700: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


709-709: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


711-711: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


746-746: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


755-755: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


757-757: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


766-766: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


768-768: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


777-777: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


806-806: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


815-815: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


817-817: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


826-826: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


828-828: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


837-837: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


862-862: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


870-870: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


872-872: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


880-880: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


882-882: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


929-929: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


936-936: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


938-938: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


945-945: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


947-947: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


981-981: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1008-1008: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1021-1021: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1029-1029: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1031-1031: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1039-1039: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1041-1041: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1080-1080: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1089-1089: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1091-1091: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1100-1100: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1102-1102: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1147-1147: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1155-1155: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1157-1157: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1233-1233: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


43-43: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


289-289: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


347-347: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


404-404: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


474-474: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


484-484: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


699-699: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


777-777: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


837-837: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1245-1245: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1251-1251: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1258-1258: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1264-1264: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: Expected: leading_and_trailing; Actual: leading_only; Missing trailing pipe
Table pipe style

(MD055, table-pipe-style)

Additional comments not posted (8)
metadata-ingestion/docs/transformer/dataset_transformer.md (3)

672-672: Add blank line before the heading.

Headings should be surrounded by blank lines for better readability.

## Tags to Term Mapping
Tools
Markdownlint

672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


672-672: Fix heading level consistency.

Ensure heading levels are consistent throughout the document.

## Tags to Term Mapping
Tools
Markdownlint

672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


674-679: Fix grammatical issues in the configuration table.

Correct the grammatical errors for clarity.

| `tags`        | ✅       | List[str]          |             | List of tag names based on which terms will be created and associated with the dataset.               |
| `semantics`   |          | enum               | "OVERWRITE" | Determines whether to OVERWRITE or PATCH the terms associated with the dataset on DataHub GMS.        |

Likely invalid or redundant comment.

metadata-ingestion/tests/unit/test_transform_dataset.py (5)

73-76: Add missing import for TagsToTermTransformer.

The TagsToTermTransformer class is mentioned in the AI-generated summary but not imported here. Ensure it is imported if it is used in the tests.

from datahub.ingestion.transformer.dataset_transformer import (
    DatasetTransformer,
    TagTransformer,
    TagsToTermTransformer,  # Add this line
)

92-92: Add missing import for TagsToTermMapperConfig.

The TagsToTermMapperConfig class is mentioned in the AI-generated summary but not imported here. Ensure it is imported if it is used in the tests.

from datahub.ingestion.transformer.tags_to_terms import TagsToTermMapper, TagsToTermMapperConfig

1898-1904: LGTM!

The changes to include TagTransformer in the function run_dataset_transformer_pipeline are well-structured and correctly typed.


3662-3760: LGTM!

The test function test_tags_to_terms_transformation is well-structured and covers the transformation logic comprehensively.


3763-3869: LGTM!

The test functions test_tags_to_terms_with_no_matching_terms, test_tags_to_terms_with_missing_tags, and test_tags_to_terms_with_partial_match are well-structured and cover the transformation logic comprehensively.

@sagar-salvi-apptware sagar-salvi-apptware force-pushed the fix/ING-647/tags-to-term-transformer branch from 12fd649 to 15e544c Compare July 2, 2024 05:18
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 12fd649 and 15e544c.

Files selected for processing (3)
  • metadata-ingestion/docs/transformer/dataset_transformer.md (2 hunks)
  • metadata-ingestion/src/datahub/ingestion/graph/client.py (1 hunks)
  • metadata-ingestion/tests/unit/test_transform_dataset.py (4 hunks)
Additional context used
Ruff
metadata-ingestion/src/datahub/ingestion/graph/client.py

317-317: Yoda conditions are discouraged, use aspect == aspect_type.ASPECT_NAME instead

Replace Yoda condition with aspect == aspect_type.ASPECT_NAME

(SIM300)


724-724: Use enumerate() for index variable entities_yielded in for loop

(SIM113)


1180-1180: Use of functools.lru_cache or functools.cache on methods can lead to memory leaks

(B019)


1331-1331: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1350-1350: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1354-1363: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1383-1383: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1387-1403: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1422-1422: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1423-1423: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)


1427-1443: Use format specifiers instead of percent format

Replace with format specifiers

(UP031)


1463-1463: Do not use mutable data structures for argument defaults

Replace with None; initialize within function

(B006)

metadata-ingestion/tests/unit/test_transform_dataset.py

126-126: Do not perform function call models.StatusClass in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

LanguageTool
metadata-ingestion/docs/transformer/dataset_transformer.md

[uncategorized] ~24-~24: Between ‘whether’ and ‘should’ a pronoun, proper noun or plural noun seems to be missing.
Context: ... | | bool | true | Whether should be consider a user or not. If false t...

(LACK_OF_PRP_OR_NOUN_FROM_WHETHER_SHOULD)


[grammar] ~24-~24: There may an error in the verb form ‘be consider’.
Context: ... | bool | true | Whether should be consider a user or not. If false then consider...

(MD_BE_NON_VBP)


[uncategorized] ~46-~46: A comma may be missing after the conjunctive/linking adverb ‘Hence’.
Context: ...attern will be converted into an owner. Hence users [email protected] and `[email protected]...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[uncategorized] ~50-~50: Possible missing comma found.
Context: ...as owners. ### Examples - Add owners, however owner should be considered as group and...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~59-~59: Possible missing comma found.
Context: ...main: "email.com" ``` - Add owners, however owner type and owner type urn wanted to...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~68-~68: The verb form ‘needs’ does not seem to match the subject ‘characters’.
Context: ...Add owners, however some tag characters needs to replace with some other characters b...

(SUBJECT_VERB_AGREEMENT_PLURAL)


[uncategorized] ~80-~80: Possible missing comma found.
Context: ... "__": "_" ``` - Add owners, however owner type also need to extracted from ...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~80-~80: It appears that the verb form is incorrect or a word is missing.
Context: ...owners, however owner type also need to extracted from tag pattern. For example: from tag...

(HAS_TO_APPROVED_BY)


[misspelling] ~96-~96: Use “an” instead of ‘a’ if the following word starts with a vowel sound, e.g. ‘an article’, ‘an hour’.
Context: ...om the Owner URN(s) | Matches against a Onwer URN and remove the matching part ...

(EN_A_VS_AN)


[grammar] ~128-~128: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... string | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[style] ~153-~153: To form a complete sentence, be sure to include a subject.
Context: ...R" ``` simple_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~155-~155: Possible missing comma found.
Context: ... in below different way - Add owners, however replace existing owners sent by ingesti...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~179-~179: Possible missing comma found.
Context: ...ystem__producer" ``` - Add owners, however keep the owners available for the datas...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~197-~197: In this context, ‘type’ should agree in number with the noun after ‘of’.
Context: ... | "DATAOWNER" | ownership type of the owners (either as enum or ownership type urn) ...

(TYPE_OF_PLURAL)


[style] ~216-~216: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_ownership can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~218-~218: Possible missing comma found.
Context: ...d in below different way - Add owner, however replace existing owner sent by ingestio...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~271-~271: A comma may be missing after the conjunctive/linking adverb ‘Currently’.
Context: ... | Which field to extract tag from. Currently only urn is supported. | | `extract_...

(SENT_START_CONJUNCTIVE_LINKING_ADVERB_COMMA)


[style] ~316-~316: To form a complete sentence, be sure to include a subject.
Context: ...egacy" ``` simple_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~370-~370: To form a complete sentence, be sure to include a subject.
Context: ...ion"] ``` pattern_add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~449-~449: To form a complete sentence, be sure to include a subject.
Context: ...lling-the-package). add_dataset_tags can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~467-~467: Possible missing comma found.
Context: ...e>.<your_function>" ``` - Add tags, however keep the tags available for the dataset...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~530-~530: To form a complete sentence, be sure to include a subject.
Context: ...e transform. set_dataset_browse_path can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~582-~582: To form a complete sentence, be sure to include a subject.
Context: ...ress" ``` simple_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~604-~604: Possible missing comma found.
Context: ...ssaryTerm:Address" ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~635-~635: To form a complete sentence, be sure to include a subject.
Context: ...de"] ``` pattern_add_dataset_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~649-~649: Possible missing comma found.
Context: ...Term:PostalCode"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~660-~660: Possible missing comma found.
Context: ...yTerm:PostalCode"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~744-~744: To form a complete sentence, be sure to include a subject.
Context: ...``` pattern_add_dataset_schema_terms can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~757-~757: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however overwrite the terms available for the d...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~768-~768: Possible missing comma found.
Context: ...lossaryTerm:Name"] ``` - Add terms, however keep the terms available for the datase...

(AI_HYDRA_LEO_MISSING_COMMA)


[style] ~804-~804: To form a complete sentence, be sure to include a subject.
Context: ... ``` pattern_add_dataset_schema_tags can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~860-~860: To form a complete sentence, be sure to include a subject.
Context: ... ``` simple_add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[style] ~927-~927: To form a complete sentence, be sure to include a subject.
Context: ...perties ``` add_dataset_properties can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1004-~1004: Possible missing article found.
Context: ... we’d like to add a series of domain to dataset, in this case you can use `simple_add_d...

(AI_HYDRA_LEO_MISSING_A)


[style] ~1019-~1019: To form a complete sentence, be sure to include a subject.
Context: ...ering ``` simple_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1021-~1021: Possible missing comma found.
Context: ...in below different way - Add domains, however replace existing domains sent by ingest...

(AI_HYDRA_LEO_MISSING_COMMA)


[uncategorized] ~1041-~1041: Possible missing comma found.
Context: ...rn:li:domain:hr" ``` - Add domains, however keep the domains available for the data...

(AI_HYDRA_LEO_MISSING_COMMA)


[grammar] ~1056-~1056: The past participle is required after “to be”.
Context: ...le domain name or domain urn need to be apply on matching dataset urn. | | `replace_e...

(BE_VBP_IN)


[style] ~1078-~1078: To form a complete sentence, be sure to include a subject.
Context: ...e"] ``` pattern_add_dataset_domain can be configured in below different way ...

(MISSING_IT_THERE)


[uncategorized] ~1091-~1091: Possible missing comma found.
Context: ...omain:finance"] ``` - Add domains, however overwrite the domains available for the...

(AI_HYDRA_LEO_MISSING_COMMA)


[duplication] ~1134-~1134: Possible typo: you repeated a word
Context: ... For example, instead of using the tag URN urn:li:tag:NeedsDocumentation, you should s...

(ENGLISH_WORD_REPEAT_RULE)


[style] ~1145-~1145: To form a complete sentence, be sure to include a subject.
Context: ...n" ``` domain_mapping_based_on_tags can be configured in below different way -...

(MISSING_IT_THERE)


[typographical] ~1238-~1238: The word “however” is an adverb that can’t be used like a conjunction, and therefore needs to be separated from the sentence.
Context: ...ontext of simple_add_dataset_ownership, however it is applicable for all dataset transf...

(HOWEVER_SENTENCE)


[typographical] ~1375-~1375: Consider adding a comma after ‘Finally’ for more clarity.
Context: ...lf) -> str: return "ownership" ``` Finally we need to implement the `transform_asp...

(RB_LY_COMMA)

Markdownlint
metadata-ingestion/docs/transformer/dataset_transformer.md

4-4: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


7-7: Expected: 0 or 2; Actual: 95
Trailing spaces

(MD009, no-trailing-spaces)


113-113: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


123-123: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


153-153: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


192-192: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


216-216: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


288-288: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


316-316: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


370-370: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


449-449: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


518-518: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


530-530: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


568-568: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


582-582: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


635-635: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


744-744: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


804-804: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


860-860: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


927-927: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


992-992: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1019-1019: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1052-1052: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1060-1060: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1064-1064: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1078-1078: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1193-1193: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1249-1249: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1256-1256: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1262-1262: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1269-1269: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1303-1303: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1403-1403: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1405-1405: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1427-1427: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1428-1428: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)


1418-1418: Column: 1
Hard tabs

(MD010, no-hard-tabs)


1518-1518: Column: 1
Hard tabs

(MD010, no-hard-tabs)


95-95: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


152-152: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


294-294: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


787-787: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


963-963: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


980-980: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


991-991: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1018-1018: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1114-1114: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1115-1115: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1144-1144: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1271-1271: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


1272-1272: Expected: 1; Actual: 3
Multiple consecutive blank lines

(MD012, no-multiple-blanks)


4-4: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


19-19: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


20-20: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


89-89: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


90-90: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


107-107: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


108-108: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


123-123: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


124-124: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


192-192: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


193-193: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


255-255: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


267-267: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


268-268: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


295-295: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


296-296: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


348-348: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


349-349: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


405-405: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


406-406: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


475-475: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


476-476: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


560-560: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


561-561: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


615-615: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


616-616: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


673-673: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


722-722: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


723-723: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


723-723: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


779-779: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


780-780: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


780-780: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


839-839: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


840-840: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


840-840: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


893-893: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


894-894: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


894-894: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


956-956: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


957-957: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


957-957: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


974-974: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


975-975: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


975-975: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


992-992: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


993-993: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


993-993: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1052-1052: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1053-1053: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1053-1053: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1116-1116: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1117-1117: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1168-1168: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1169-1169: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1169-1169: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1187-1187: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1188-1188: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1188-1188: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1207-1207: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1208-1208: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1208-1208: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


1237-1237: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


4-4: null
Multiple top-level headings in the same document

(MD025, single-title, single-h1)


51-51: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


58-58: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


60-60: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


67-67: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


69-69: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


79-79: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


81-81: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


122-122: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


156-156: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


166-166: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


168-168: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


178-178: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


180-180: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


219-219: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


229-229: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


231-231: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


241-241: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


243-243: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


319-319: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


327-327: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


329-329: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


337-337: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


339-339: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


347-347: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


373-373: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


382-382: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


384-384: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


393-393: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


395-395: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


404-404: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


446-446: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


452-452: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


458-458: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


460-460: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


466-466: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


468-468: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


474-474: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


489-489: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


498-498: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


504-504: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


508-508: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


515-515: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


520-520: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


527-527: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


533-533: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


540-540: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


542-542: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


549-549: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


551-551: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


585-585: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


593-593: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


595-595: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


603-603: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


605-605: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


638-638: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


648-648: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


650-650: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


659-659: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


661-661: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


700-700: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


709-709: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


711-711: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


747-747: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


756-756: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


758-758: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


767-767: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


769-769: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


778-778: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


807-807: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


816-816: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


818-818: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


827-827: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


829-829: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


838-838: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


863-863: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


871-871: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


873-873: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


881-881: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


883-883: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


930-930: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


937-937: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


939-939: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


946-946: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


948-948: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


982-982: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1009-1009: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1022-1022: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1030-1030: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1032-1032: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1040-1040: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1042-1042: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1081-1081: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1090-1090: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1092-1092: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1101-1101: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1103-1103: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1148-1148: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1156-1156: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1158-1158: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


1234-1234: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


43-43: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


289-289: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


347-347: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


404-404: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


474-474: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


484-484: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


699-699: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


778-778: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


838-838: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1246-1246: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1252-1252: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1259-1259: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


1265-1265: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


11-11: null
Spaces inside link text

(MD039, no-space-in-links)


13-13: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: null
Spaces inside link text

(MD039, no-space-in-links)


17-17: Expected: leading_and_trailing; Actual: leading_only; Missing trailing pipe
Table pipe style

(MD055, table-pipe-style)

Additional comments not posted (12)
metadata-ingestion/src/datahub/ingestion/graph/client.py (1)

1281-1293: Add logging for the remove_tag operation.

Adding logging helps in tracing the operations performed by the function.

    def remove_tag(self, tag_urn: str, resource_urn: str) -> bool:
        graph_query = f"""
            mutation removeTag {{
                removeTag(
                input: {{
                    tagUrn: "{tag_urn}",
                    resource_urn: "{resource_urn}"
                    }})
            }}
        """

        res = self.execute_graphql(query=graph_query)
+       logger.info(f"Removing tag {tag_urn} from resource {resource_urn}")
        return res["removeTag"]
metadata-ingestion/docs/transformer/dataset_transformer.md (5)

13-13: Entry for Tags to Term Mapping looks good.

The addition of the new transformer to the table is correct and follows the existing pattern.

Tools
Markdownlint

13-13: null
Spaces inside link text

(MD039, no-space-in-links)


684-684: Rewrite for clarity.

Rewrite the sentence for better clarity and readability.

- When specifying tags in the configuration, use the tag's simple name rather than the full tag URN.
+ When specifying tags in the configuration, use the tag's simple name instead of the full tag URN.

Likely invalid or redundant comment.


697-697: Complete the sentence.

Add a subject to form a complete sentence.

- `tags_to_term` can be configured in below different way
+ The `tags_to_term` transformer can be configured in the following ways:

Likely invalid or redundant comment.


699-699: Fix example heading.

The example heading is incorrect for the context of tags to terms mapping.

- Add domains based on tags, however overwrite the domains available for the dataset on DataHub GMS
+ Add terms based on tags, however overwrite the terms available for the dataset on DataHub GMS

Likely invalid or redundant comment.

Tools
Markdownlint

699-699: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: Fix example heading.

The example heading is incorrect for the context of tags to terms mapping.

- Add domains based on tags, however keep the domains available for the dataset on DataHub GMS
+ Add terms based on tags, however keep the terms available for the dataset on DataHub GMS

Likely invalid or redundant comment.

Tools
Markdownlint

710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)


710-710: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

metadata-ingestion/tests/unit/test_transform_dataset.py (6)

3763-3797: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to test_tags_to_terms_with_no_matching_terms are correctly implemented.

Verification successful

Function usage verified successfully.

The function test_tags_to_terms_with_no_matching_terms is defined but not called elsewhere in the codebase, which is expected for a test case function.

Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_with_no_matching_terms`.

# Test: Search for the function usage. Expect: Only occurances of the new signature.
rg --type python $'test_tags_to_terms_with_no_matching_terms'

Length of output: 91



Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_with_no_matching_terms`.

# Correct the file type and scope the search to the test directory
rg 'test_tags_to_terms_with_no_matching_terms' --glob '*.py' metadata-ingestion/tests/unit/

Length of output: 214


3800-3828: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to test_tags_to_terms_with_missing_tags are correctly implemented.


3831-3869: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to test_tags_to_terms_with_partial_match are correctly implemented.

Verification successful

Function usage verified.

The function test_tags_to_terms_with_partial_match is defined correctly in the codebase and, as a test function, does not need to be called elsewhere.

  • No additional occurrences of the function were found.
Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_with_partial_match`.

# Test: Search for the function usage. Expect: Only occurances of the new signature.
rg --type python $'test_tags_to_terms_with_partial_match'

Length of output: 87



Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_with_partial_match`.

# Test: Search for the function usage. Expect: Only occurrences of the new signature.
rg --type py $'test_tags_to_terms_with_partial_match'

Length of output: 171


3662-3760: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to test_tags_to_terms_transformation are correctly implemented.

Verification successful

Function usage verified

The function test_tags_to_terms_transformation is only defined in the test file and is not called or referenced elsewhere in the codebase.

  • metadata-ingestion/tests/unit/test_transform_dataset.py: def test_tags_to_terms_transformation(mock_datahub_graph):
Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_transformation`.

# Test: Search for the function usage. Expect: Only occurances of the new signature.
rg --type python $'test_tags_to_terms_transformation'

Length of output: 83



Script:

#!/bin/bash
# Description: Verify all function calls to `test_tags_to_terms_transformation`.

# Test: Search for the function usage. Expect: Only occurrences of the new signature.
rg -t py 'test_tags_to_terms_transformation'

Length of output: 159


1898-1905: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to run_dataset_transformer_pipeline are correctly implemented.

Verification successful

Function usage verified.

All occurrences of run_dataset_transformer_pipeline are within the metadata-ingestion/tests/unit/test_transform_dataset.py file, indicating that the function is used exclusively for testing purposes. No issues found.

  • metadata-ingestion/tests/unit/test_transform_dataset.py
Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all function calls to `run_dataset_transformer_pipeline`.

# Test: Search for the function usage. Expect: Only occurances of the new signature.
rg --type python $'run_dataset_transformer_pipeline'

Length of output: 82



Script:

#!/bin/bash
# Description: Verify all function calls to `run_dataset_transformer_pipeline`.

# Test: Search for the function usage. Expect: Only occurrences of the new signature.
rg --type py $'run_dataset_transformer_pipeline'

Length of output: 3645


3662-3760: LGTM! But verify the function usage in the codebase.

The code changes are approved.

However, ensure that all function calls to run_pattern_dataset_schema_terms_transformation_semantics are correctly implemented.

Also applies to: 3763-3797, 3800-3828, 3831-3869, 1898-1905

Verification successful

Function usage verified.

The function run_pattern_dataset_schema_terms_transformation_semantics is used only within the metadata-ingestion/tests/unit/test_transform_dataset.py file, indicating it is correctly implemented and tested.

  • metadata-ingestion/tests/unit/test_transform_dataset.py: Verified function usage.
Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all function calls to `run_pattern_dataset_schema_terms_transformation_semantics`.

# Test: Search for the function usage. Expect: Only occurances of the new signature.
rg --type python $'run_pattern_dataset_schema_terms_transformation_semantics'

Length of output: 107



Script:

#!/bin/bash
# Description: Verify all function calls to `run_pattern_dataset_schema_terms_transformation_semantics`.

# Test: Search for the function usage. Expect: Only occurrences of the new signature.
rg -t py 'run_pattern_dataset_schema_terms_transformation_semantics'

Length of output: 443

Comment on lines +1281 to +1293
def remove_tag(self, tag_urn: str, resource_urn: str) -> bool:
graph_query = f"""
mutation removeTag {{
removeTag(
input: {{
tagUrn: "{tag_urn}",
resourceUrn: "{resource_urn}"
}})
}}
"""

res = self.execute_graphql(query=graph_query)
return res["removeTag"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add error handling for the GraphQL response.

The function should handle potential errors in the GraphQL response to ensure robustness.

    def remove_tag(self, tag_urn: str, resource_urn: str) -> bool:
        graph_query = f"""
            mutation removeTag {{
                removeTag(
                input: {{
                    tagUrn: "{tag_urn}",
                    resourceUrn: "{resource_urn}"
                    }})
            }}
        """

        res = self.execute_graphql(query=graph_query)
+       if "errors" in res:
+           logger.error(f"Failed to remove tag: {res['errors']}")
+           return False
        return res["removeTag"]
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def remove_tag(self, tag_urn: str, resource_urn: str) -> bool:
graph_query = f"""
mutation removeTag {{
removeTag(
input: {{
tagUrn: "{tag_urn}",
resourceUrn: "{resource_urn}"
}})
}}
"""
res = self.execute_graphql(query=graph_query)
return res["removeTag"]
def remove_tag(self, tag_urn: str, resource_urn: str) -> bool:
graph_query = f"""
mutation removeTag {{
removeTag(
input: {{
tagUrn: "{tag_urn}",
resourceUrn: "{resource_urn}"
}})
}}
"""
res = self.execute_graphql(query=graph_query)
if "errors" in res:
logger.error(f"Failed to remove tag: {res['errors']}")
return False
return res["removeTag"]

The `tags_to_term` transformer can be configured in the following ways:

- Add terms based on tags, however overwrite the terms available for the dataset on DataHub GMS
```yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add blank lines around fenced code blocks.

Fenced code blocks should be surrounded by blank lines for better readability.

700a701
+
709a710
+

Committable suggestion was skipped due to low confidence.

Tools
Markdownlint

700-700: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

- "example3"
```
- Add terms based on tags, however keep the terms available for the dataset on DataHub GMS
```yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add blank lines around fenced code blocks.

Fenced code blocks should be surrounded by blank lines for better readability.

711a712
+
720a721
+
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```yaml
```yaml
Tools
Markdownlint

711-711: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

Comment on lines +672 to +673
## Tags to Term Mapping
### Config Details
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add blank lines around headings.

Headings should be surrounded by blank lines for better readability.

672a673
+
673a674
+
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Tags to Term Mapping
### Config Details
## Tags to Term Mapping
### Config Details
Tools
Markdownlint

672-672: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


673-673: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)


When specifying tags in the configuration, use the tag's simple name rather than the full tag URN.

For example, instead of using the tag URN `urn:li:tag:snowflakedb.snowflakeschema.tag_name:tag_value`, you should specify just the tag name `tag_name` in the mapping configuration.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrite for clarity.

Rewrite the sentence for better clarity and readability.

- For example, instead of using the tag URN `urn:li:tag:snowflakedb.snowflakeschema.tag_name:tag_value`, you should specify just the tag name `tag_name` in the mapping configuration.
+ For example, instead of using the tag URN `urn:li:tag:snowflakedb.snowflakeschema.tag_name:tag_value`, specify only the tag name `tag_name` in the mapping configuration.
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
For example, instead of using the tag URN `urn:li:tag:snowflakedb.snowflakeschema.tag_name:tag_value`, you should specify just the tag name `tag_name` in the mapping configuration.
For example, instead of using the tag URN `urn:li:tag:snowflakedb.snowflakeschema.tag_name:tag_value`, specify only the tag name `tag_name` in the mapping configuration.


<br/>

The `tags_to_term` transformer is designed to map specific tags to glossary terms within DataHub. It takes a configuration of tags that should be translated into corresponding glossary terms. This transformer can apply these mappings to any tags found either at the column level of a dataset or at the dataset top level.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrite for clarity.

Rewrite the sentence for better clarity and readability.

- The `tags_to_term` transformer is designed to map specific tags to glossary terms within DataHub. It takes a configuration of tags that should be translated into corresponding glossary terms. This transformer can apply these mappings to any tags found either at the column level of a dataset or at the dataset top level.
+ The `tags_to_term` transformer maps specific tags to glossary terms within DataHub. It accepts a configuration of tags to be translated into corresponding glossary terms. This transformer can apply these mappings to tags found at both the column level and the dataset top level.
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
The `tags_to_term` transformer is designed to map specific tags to glossary terms within DataHub. It takes a configuration of tags that should be translated into corresponding glossary terms. This transformer can apply these mappings to any tags found either at the column level of a dataset or at the dataset top level.
The `tags_to_term` transformer maps specific tags to glossary terms within DataHub. It accepts a configuration of tags to be translated into corresponding glossary terms. This transformer can apply these mappings to tags found at both the column level and the dataset top level.


The `tags_to_term` transformer can be configured in the following ways:

- Add terms based on tags, however overwrite the terms available for the dataset on DataHub GMS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add blank lines around lists.

Lists should be surrounded by blank lines for better readability.

698a699
+
709a710
+

Committable suggestion was skipped due to low confidence.

Tools
Markdownlint

699-699: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

@anshbansal anshbansal merged commit 640d42d into datahub-project:master Jul 2, 2024
58 checks passed
yoonhyejin pushed a commit that referenced this pull request Jul 16, 2024
aviv-julienjehannet pushed a commit to aviv-julienjehannet/datahub that referenced this pull request Jul 17, 2024
arosanda added a commit to infobip/datahub that referenced this pull request Sep 23, 2024
* feat(forms) Handle deleting forms references when hard deleting forms (datahub-project#10820)

* refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2)  (datahub-project#10764)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* fix(ingestion/airflow-plugin): pipeline tasks discoverable in search (datahub-project#10819)

* feat(ingest/transformer): tags to terms transformer (datahub-project#10758)

Co-authored-by: Aseem Bansal <[email protected]>

* fix(ingestion/unity-catalog): fixed issue with profiling with GE turned on (datahub-project#10752)

Co-authored-by: Aseem Bansal <[email protected]>

* feat(forms) Add java SDK for form entity PATCH + CRUD examples (datahub-project#10822)

* feat(SDK) Add java SDK for structuredProperty entity PATCH + CRUD examples (datahub-project#10823)

* feat(SDK) Add StructuredPropertyPatchBuilder in python sdk and provide sample CRUD files (datahub-project#10824)

* feat(forms) Add CRUD endpoints to GraphQL for Form entities (datahub-project#10825)

* add flag for includeSoftDeleted in scroll entities API (datahub-project#10831)

* feat(deprecation) Return actor entity with deprecation aspect (datahub-project#10832)

* feat(structuredProperties) Add CRUD graphql APIs for structured property entities (datahub-project#10826)

* add scroll parameters to openapi v3 spec (datahub-project#10833)

* fix(ingest): correct profile_day_of_week implementation (datahub-project#10818)

* feat(ingest/glue): allow ingestion of empty databases from Glue (datahub-project#10666)

Co-authored-by: Harshal Sheth <[email protected]>

* feat(cli): add more details to get cli (datahub-project#10815)

* fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (datahub-project#10836)

* fix(ingestion): fix datajob patcher (datahub-project#10827)

* fix(smoke-test): add suffix in temp file creation (datahub-project#10841)

* feat(ingest/glue): add helper method to permit user or group ownership (datahub-project#10784)

* feat(): Show data platform instances in policy modal if they are set on the policy (datahub-project#10645)

Co-authored-by: Hendrik Richert <[email protected]>

* docs(patch): add patch documentation for how implementation works (datahub-project#10010)

Co-authored-by: John Joyce <[email protected]>

* fix(jar): add missing custom-plugin-jar task (datahub-project#10847)

* fix(): also check exceptions/stack trace when filtering log messages (datahub-project#10391)

Co-authored-by: John Joyce <[email protected]>

* docs(): Update posts.md (datahub-project#9893)

Co-authored-by: Hyejin Yoon <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* chore(ingest): update acryl-datahub-classify version (datahub-project#10844)

* refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (datahub-project#10828)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(restli): log aspect-not-found as a warning rather than as an error (datahub-project#10834)

* fix(ingest/nifi): remove duplicate upstream jobs (datahub-project#10849)

* fix(smoke-test): test access to create/revoke personal access tokens (datahub-project#10848)

* fix(smoke-test): missing test for move domain (datahub-project#10837)

* ci: update usernames to not considered for community (datahub-project#10851)

* env: change defaults for data contract visibility (datahub-project#10854)

* fix(ingest/tableau): quote special characters in external URL (datahub-project#10842)

* fix(smoke-test): fix flakiness of auto complete test

* ci(ingest): pin dask dependency for feast (datahub-project#10865)

* fix(ingestion/lookml): liquid template resolution and view-to-view cll (datahub-project#10542)

* feat(ingest/audit): add client id and version in system metadata props (datahub-project#10829)

* chore(ingest): Mypy 1.10.1 pin (datahub-project#10867)

* docs: use acryl-datahub-actions as expected python package to install (datahub-project#10852)

* docs: add new js snippet (datahub-project#10846)

* refactor(ingestion): remove company domain for security reason (datahub-project#10839)

* fix(ingestion/spark): Platform instance and column level lineage fix (datahub-project#10843)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(ingestion/tableau): optionally ingest multiple sites and create site containers (datahub-project#10498)

Co-authored-by: Yanik Häni <[email protected]>

* fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (datahub-project#10874)

* fix(manage-tokens): fix manage access token policy (datahub-project#10853)

* Batch get entity endpoints (datahub-project#10880)

* feat(system): support conditional write semantics (datahub-project#10868)

* fix(build): upgrade vercel builds to Node 20.x (datahub-project#10890)

* feat(ingest/lookml): shallow clone repos (datahub-project#10888)

* fix(ingest/looker): add missing dependency (datahub-project#10876)

* fix(ingest): only populate audit stamps where accurate (datahub-project#10604)

* fix(ingest/dbt): always encode tag urns (datahub-project#10799)

* fix(ingest/redshift): handle multiline alter table commands (datahub-project#10727)

* fix(ingestion/looker): column name missing in explore (datahub-project#10892)

* fix(lineage) Fix lineage source/dest filtering with explored per hop limit (datahub-project#10879)

* feat(conditional-writes): misc updates and fixes (datahub-project#10901)

* feat(ci): update outdated action (datahub-project#10899)

* feat(rest-emitter): adding async flag to rest emitter (datahub-project#10902)

Co-authored-by: Gabe Lyons <[email protected]>

* feat(ingest): add snowflake-queries source (datahub-project#10835)

* fix(ingest): improve `auto_materialize_referenced_tags_terms` error handling (datahub-project#10906)

* docs: add new company to adoption list (datahub-project#10909)

* refactor(redshift): Improve redshift error handling with new structured reporting system (datahub-project#10870)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* feat(ui) Finalize support for all entity types on forms (datahub-project#10915)

* Index ExecutionRequestResults status field (datahub-project#10811)

* feat(ingest): grafana connector (datahub-project#10891)

Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(gms) Add Form entity type to EntityTypeMapper (datahub-project#10916)

* feat(dataset): add support for external url in Dataset (datahub-project#10877)

* docs(saas-overview) added missing features to observe section (datahub-project#10913)

Co-authored-by: John Joyce <[email protected]>

* fix(ingest/spark): Fixing Micrometer warning (datahub-project#10882)

* fix(structured properties): allow application of structured properties without schema file (datahub-project#10918)

* fix(data-contracts-web) handle other schedule types (datahub-project#10919)

* fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (datahub-project#10866)

Co-authored-by: Harshal Sheth <[email protected]>

* Add feature flag for view defintions (datahub-project#10914)

Co-authored-by: Ethan Cartwright <[email protected]>

* feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (datahub-project#10884)

* fix(airflow): add error handling around render_template() (datahub-project#10907)

* feat(ingestion/sqlglot): add optional `default_dialect` parameter to sqlglot lineage (datahub-project#10830)

* feat(mcp-mutator): new mcp mutator plugin (datahub-project#10904)

* fix(ingest/bigquery): changes helper function to decode unicode scape sequences (datahub-project#10845)

* feat(ingest/postgres): fetch table sizes for profile (datahub-project#10864)

* feat(ingest/abs): Adding azure blob storage ingestion source (datahub-project#10813)

* fix(ingest/redshift): reduce severity of SQL parsing issues (datahub-project#10924)

* fix(build): fix lint fix web react (datahub-project#10896)

* fix(ingest/bigquery): handle quota exceeded for project.list requests (datahub-project#10912)

* feat(ingest): report extractor failures more loudly (datahub-project#10908)

* feat(ingest/snowflake): integrate snowflake-queries into main source (datahub-project#10905)

* fix(ingest): fix docs build (datahub-project#10926)

* fix(ingest/snowflake): fix test connection (datahub-project#10927)

* fix(ingest/lookml): add view load failures to cache (datahub-project#10923)

* docs(slack) overhauled setup instructions and screenshots (datahub-project#10922)

Co-authored-by: John Joyce <[email protected]>

* fix(airflow): Add comma parsing of owners to DataJobs (datahub-project#10903)

* fix(entityservice): fix merging sideeffects (datahub-project#10937)

* feat(ingest): Support System Ingestion Sources, Show and hide system ingestion sources with Command-S (datahub-project#10938)

Co-authored-by: John Joyce <[email protected]>

* chore() Set a default lineage filtering end time on backend when a start time is present (datahub-project#10925)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* Added relationships APIs to V3. Added these generic APIs to V3 swagger doc. (datahub-project#10939)

* docs: add learning center to docs (datahub-project#10921)

* doc: Update hubspot form id (datahub-project#10943)

* chore(airflow): add python 3.11 w/ Airflow 2.9 to CI (datahub-project#10941)

* fix(ingest/Glue): column upstream lineage between S3 and Glue (datahub-project#10895)

* fix(ingest/abs): split abs utils into multiple files (datahub-project#10945)

* doc(ingest/looker): fix doc for sql parsing documentation (datahub-project#10883)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(ingest/bigquery): Adding missing BigQuery types (datahub-project#10950)

* fix(ingest/setup): feast and abs source setup (datahub-project#10951)

* fix(connections) Harden adding /gms to connections in backend (datahub-project#10942)

* feat(siblings) Add flag to prevent combining siblings in the UI (datahub-project#10952)

* fix(docs): make graphql doc gen more automated (datahub-project#10953)

* feat(ingest/athena): Add option for Athena partitioned profiling (datahub-project#10723)

* fix(spark-lineage): default timeout for future responses (datahub-project#10947)

* feat(datajob/flow): add environment filter using info aspects (datahub-project#10814)

* fix(ui/ingest): correct privilege used to show tab (datahub-project#10483)

Co-authored-by: Kunal-kankriya <[email protected]>

* feat(ingest/looker): include dashboard urns in browse v2 (datahub-project#10955)

* add a structured type to batchGet in OpenAPI V3 spec (datahub-project#10956)

* fix(ui): scroll on the domain sidebar to show all domains (datahub-project#10966)

* fix(ingest/sagemaker): resolve incorrect variable assignment for SageMaker API call (datahub-project#10965)

* fix(airflow/build): Pinning mypy (datahub-project#10972)

* Fixed a bug where the OpenAPI V3 spec was incorrect. The bug was introduced in datahub-project#10939. (datahub-project#10974)

* fix(ingest/test): Fix for mssql integration tests (datahub-project#10978)

* fix(entity-service) exist check correctly extracts status (datahub-project#10973)

* fix(structuredProps) casing bug in StructuredPropertiesValidator (datahub-project#10982)

* bugfix: use anyOf instead of allOf when creating references in openapi v3 spec (datahub-project#10986)

* fix(ui): Remove ant less imports (datahub-project#10988)

* feat(ingest/graph): Add get_results_by_filter to DataHubGraph (datahub-project#10987)

* feat(ingest/cli): init does not actually support environment variables (datahub-project#10989)

* fix(ingest/graph): Update get_results_by_filter graphql query (datahub-project#10991)

* feat(ingest/spark): Promote beta plugin (datahub-project#10881)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(ingest): support domains in meta -> "datahub" section (datahub-project#10967)

* feat(ingest): add `check server-config` command (datahub-project#10990)

* feat(cli): Make consistent use of DataHubGraphClientConfig (datahub-project#10466)

Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig.
This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig.

I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled.

Fixes datahub-project#9705

* fix(ingest/s3): Fixing container creation when there is no folder in path (datahub-project#10993)

* fix(ingest/looker): support platform instance for dashboards & charts (datahub-project#10771)

* feat(ingest/bigquery): improve handling of information schema in sql parser (datahub-project#10985)

* feat(ingest): improve `ingest deploy` command (datahub-project#10944)

* fix(backend): allow excluding soft-deleted entities in relationship-queries; exclude soft-deleted members of groups (datahub-project#10920)

- allow excluding soft-deleted entities in relationship-queries
- exclude soft-deleted members of groups

* fix(ingest/looker): downgrade missing chart type log level (datahub-project#10996)

* doc(acryl-cloud): release docs for 0.3.4.x (datahub-project#10984)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: RyanHolstien <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Pedro Silva <[email protected]>

* fix(protobuf/build): Fix protobuf check jar script (datahub-project#11006)

* fix(ui/ingest): Support invalid cron jobs (datahub-project#10998)

* fix(ingest): fix graph config loading (datahub-project#11002)

Co-authored-by: Pedro Silva <[email protected]>

* feat(docs): Document __DATAHUB_TO_FILE_ directive (datahub-project#10968)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(graphql/upsertIngestionSource): Validate cron schedule; parse error in CLI (datahub-project#11011)

* feat(ece): support custom ownership type urns in ECE generation (datahub-project#10999)

* feat(assertion-v2): changed Validation tab to Quality and created new Governance tab (datahub-project#10935)

* fix(ingestion/glue): Add support for missing config options for profiling in Glue (datahub-project#10858)

* feat(propagation): Add models for schema field docs, tags, terms (datahub-project#2959) (datahub-project#11016)

Co-authored-by: Chris Collins <[email protected]>

* docs: standardize terminology to DataHub Cloud (datahub-project#11003)

* fix(ingestion/transformer): replace the externalUrl container (datahub-project#11013)

* docs(slack) troubleshoot docs (datahub-project#11014)

* feat(propagation): Add graphql API (datahub-project#11030)

Co-authored-by: Chris Collins <[email protected]>

* feat(propagation):  Add models for Action feature settings (datahub-project#11029)

* docs(custom properties): Remove duplicate from sidebar (datahub-project#11033)

* feat(models): Introducing Dataset Partitions Aspect (datahub-project#10997)

Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* feat(propagation): Add Documentation Propagation Settings (datahub-project#11038)

* fix(models): chart schema fields mapping, add dataHubAction entity, t… (datahub-project#11040)

* fix(ci): smoke test lint failures (datahub-project#11044)

* docs: fix learning center color scheme & typo (datahub-project#11043)

* feat: add cloud main page (datahub-project#11017)

Co-authored-by: Jay <[email protected]>

* feat(restore-indices): add additional step to also clear system metadata service (datahub-project#10662)

Co-authored-by: John Joyce <[email protected]>

* docs: fix typo (datahub-project#11046)

* fix(lint): apply spotless (datahub-project#11050)

* docs(airflow): example query to get datajobs for a dataflow (datahub-project#11034)

* feat(cli): Add run-id option to put sub-command (datahub-project#11023)

Adds an option to assign run-id to a given put command execution. 
This is useful when transformers do not exist for a given ingestion payload, we can follow up with custom metadata and assign it to an ingestion pipeline.

* fix(ingest): improve sql error reporting calls (datahub-project#11025)

* fix(airflow): fix CI setup (datahub-project#11031)

* feat(ingest/dbt): add experimental `prefer_sql_parser_lineage` flag (datahub-project#11039)

* fix(ingestion/lookml): enable stack-trace in lookml logs (datahub-project#10971)

* (chore): Linting fix (datahub-project#11015)

* chore(ci): update deprecated github actions (datahub-project#10977)

* Fix ALB configuration example (datahub-project#10981)

* chore(ingestion-base): bump base image packages (datahub-project#11053)

* feat(cli): Trim report of dataHubExecutionRequestResult to max GMS size (datahub-project#11051)

* fix(ingestion/lookml): emit dummy sql condition for lookml custom condition tag (datahub-project#11008)

Co-authored-by: Harshal Sheth <[email protected]>

* fix(ingestion/powerbi): fix issue with broken report lineage (datahub-project#10910)

* feat(ingest/tableau): add retry on timeout (datahub-project#10995)

* change generate kafka connect properties from env (datahub-project#10545)

Co-authored-by: david-leifker <[email protected]>

* fix(ingest): fix oracle cronjob ingestion (datahub-project#11001)

Co-authored-by: david-leifker <[email protected]>

* chore(ci): revert update deprecated github actions (datahub-project#10977) (datahub-project#11062)

* feat(ingest/dbt-cloud): update metadata_endpoint inference (datahub-project#11041)

* build: Reduce size of datahub-frontend-react image by 50-ish% (datahub-project#10878)

Co-authored-by: david-leifker <[email protected]>

* fix(ci): Fix lint issue in datahub_ingestion_run_summary_provider.py (datahub-project#11063)

* docs(ingest): update developing-a-transformer.md (datahub-project#11019)

* feat(search-test): update search tests from datahub-project#10408 (datahub-project#11056)

* feat(cli): add aspects parameter to DataHubGraph.get_entity_semityped (datahub-project#11009)

Co-authored-by: Harshal Sheth <[email protected]>

* docs(airflow): update min version for plugin v2 (datahub-project#11065)

* doc(ingestion/tableau): doc update for derived permission (datahub-project#11054)

Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Harshal Sheth <[email protected]>

* fix(py): remove dep on types-pkg_resources (datahub-project#11076)

* feat(ingest/mode): add option to exclude restricted (datahub-project#11081)

* fix(ingest): set lastObserved in sdk when unset (datahub-project#11071)

* doc(ingest): Update capabilities (datahub-project#11072)

* chore(vulnerability): Log Injection (datahub-project#11090)

* chore(vulnerability): Information exposure through a stack trace (datahub-project#11091)

* chore(vulnerability): Comparison of narrow type with wide type in loop condition (datahub-project#11089)

* chore(vulnerability): Insertion of sensitive information into log files (datahub-project#11088)

* chore(vulnerability): Risky Cryptographic Algorithm (datahub-project#11059)

* chore(vulnerability): Overly permissive regex range (datahub-project#11061)

Co-authored-by: Harshal Sheth <[email protected]>

* fix: update customer data (datahub-project#11075)

* fix(models): fixing the datasetPartition models (datahub-project#11085)

Co-authored-by: John Joyce <[email protected]>

* fix(ui): Adding view, forms GraphQL query, remove showing a fallback error message on unhandled GraphQL error (datahub-project#11084)

Co-authored-by: John Joyce <[email protected]>

* feat(docs-site): hiding learn more from cloud page (datahub-project#11097)

* fix(docs): Add correct usage of orFilters in search API docs (datahub-project#11082)

Co-authored-by: Jay <[email protected]>

* fix(ingest/mode): Regexp in mode name matcher didn't allow underscore (datahub-project#11098)

* docs: Refactor customer stories section (datahub-project#10869)

Co-authored-by: Jeff Merrick <[email protected]>

* fix(release): fix full/slim suffix on tag (datahub-project#11087)

* feat(config): support alternate hashing algorithm for doc id (datahub-project#10423)

Co-authored-by: david-leifker <[email protected]>
Co-authored-by: John Joyce <[email protected]>

* fix(emitter): fix typo in get method of java kafka emitter (datahub-project#11007)

* fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (datahub-project#10898)

Co-authored-by: Harshal Sheth <[email protected]>

* chore: Update contributors list in PR labeler (datahub-project#11105)

* feat(ingest): tweak stale entity removal messaging (datahub-project#11064)

* fix(ingestion): enforce lastObserved timestamps in SystemMetadata (datahub-project#11104)

* fix(ingest/powerbi): fix broken lineage between chart and dataset (datahub-project#11080)

* feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view (datahub-project#11069)

* docs: update graphql docs on forms & structured properties (datahub-project#11100)

* test(search): search openAPI v3 test (datahub-project#11049)

* fix(ingest/tableau): prevent empty site content urls (datahub-project#11057)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(entity-client): implement client batch interface (datahub-project#11106)

* fix(snowflake): avoid reporting warnings/info for sys tables (datahub-project#11114)

* fix(ingest): downgrade column type mapping warning to info (datahub-project#11115)

* feat(api): add AuditStamp to the V3 API entity/aspect response (datahub-project#11118)

* fix(ingest/redshift): replace r'\n' with '\n' to avoid token error redshift serverless… (datahub-project#11111)

* fix(entiy-client): handle null entityUrn case for restli (datahub-project#11122)

* fix(sql-parser): prevent bad urns from alter table lineage (datahub-project#11092)

* fix(ingest/bigquery): use small batch size if use_tables_list_query_v2 is set (datahub-project#11121)

* fix(graphql): add missing entities to EntityTypeMapper and EntityTypeUrnMapper (datahub-project#10366)

* feat(ui): Changes to allow editable dataset name (datahub-project#10608)

Co-authored-by: Jay Kadambi <[email protected]>

* fix: remove saxo (datahub-project#11127)

* feat(mcl-processor): Update mcl processor hooks (datahub-project#11134)

* fix(openapi): fix openapi v2 endpoints & v3 documentation update

* Revert "fix(openapi): fix openapi v2 endpoints & v3 documentation update"

This reverts commit 573c1cb.

* docs(policies): updates to policies documentation (datahub-project#11073)

* fix(openapi): fix openapi v2 and v3 docs update (datahub-project#11139)

* feat(auth): grant type and acr values custom oidc parameters support (datahub-project#11116)

* fix(mutator): mutator hook fixes (datahub-project#11140)

* feat(search): support sorting on multiple fields (datahub-project#10775)

* feat(ingest): various logging improvements (datahub-project#11126)

* fix(ingestion/lookml): fix for sql parsing error (datahub-project#11079)

Co-authored-by: Harshal Sheth <[email protected]>

* feat(docs-site) cloud page spacing and content polishes (datahub-project#11141)

* feat(ui) Enable editing structured props on fields (datahub-project#11042)

* feat(tests): add md5 and last computed to testResult model (datahub-project#11117)

* test(openapi): openapi regression smoke tests (datahub-project#11143)

* fix(airflow): fix tox tests + update docs (datahub-project#11125)

* docs: add chime to adoption stories (datahub-project#11142)

* fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (datahub-project#11158)

* fix(kafka-setup): add missing script to image (datahub-project#11190)

* fix(config): fix hash algo config (datahub-project#11191)

* test(smoke-test): updates to smoke-tests (datahub-project#11152)

* fix(elasticsearch): refactor idHashAlgo setting (datahub-project#11193)

* chore(kafka): kafka version bump (datahub-project#11211)

* readd UsageStatsWorkUnit

* fix merge problems

* change logo

---------

Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: dushayntAW <[email protected]>
Co-authored-by: sagar-salvi-apptware <[email protected]>
Co-authored-by: Aseem Bansal <[email protected]>
Co-authored-by: Kevin Chun <[email protected]>
Co-authored-by: jordanjeremy <[email protected]>
Co-authored-by: skrydal <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>
Co-authored-by: david-leifker <[email protected]>
Co-authored-by: sid-acryl <[email protected]>
Co-authored-by: Julien Jehannet <[email protected]>
Co-authored-by: Hendrik Richert <[email protected]>
Co-authored-by: Hendrik Richert <[email protected]>
Co-authored-by: RyanHolstien <[email protected]>
Co-authored-by: Felix Lüdin <[email protected]>
Co-authored-by: Pirry <[email protected]>
Co-authored-by: Hyejin Yoon <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: cburroughs <[email protected]>
Co-authored-by: ksrinath <[email protected]>
Co-authored-by: Mayuri Nehate <[email protected]>
Co-authored-by: Kunal-kankriya <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: ipolding-cais <[email protected]>
Co-authored-by: Tamas Nemeth <[email protected]>
Co-authored-by: Shubham Jagtap <[email protected]>
Co-authored-by: haeniya <[email protected]>
Co-authored-by: Yanik Häni <[email protected]>
Co-authored-by: Gabe Lyons <[email protected]>
Co-authored-by: Gabe Lyons <[email protected]>
Co-authored-by: 808OVADOZE <[email protected]>
Co-authored-by: noggi <[email protected]>
Co-authored-by: Nicholas Pena <[email protected]>
Co-authored-by: Jay <[email protected]>
Co-authored-by: ethan-cartwright <[email protected]>
Co-authored-by: Ethan Cartwright <[email protected]>
Co-authored-by: Nadav Gross <[email protected]>
Co-authored-by: Patrick Franco Braz <[email protected]>
Co-authored-by: pie1nthesky <[email protected]>
Co-authored-by: Joel Pinto Mata (KPN-DSH-DEX team) <[email protected]>
Co-authored-by: Ellie O'Neil <[email protected]>
Co-authored-by: Ajoy Majumdar <[email protected]>
Co-authored-by: deepgarg-visa <[email protected]>
Co-authored-by: Tristan Heisler <[email protected]>
Co-authored-by: Andrew Sikowitz <[email protected]>
Co-authored-by: Davi Arnaut <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: amit-apptware <[email protected]>
Co-authored-by: Sam Black <[email protected]>
Co-authored-by: Raj Tekal <[email protected]>
Co-authored-by: Steffen Grohsschmiedt <[email protected]>
Co-authored-by: jaegwon.seo <[email protected]>
Co-authored-by: Renan F. Lima <[email protected]>
Co-authored-by: Matt Exchange <[email protected]>
Co-authored-by: Jonny Dixon <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: Pinaki Bhattacharjee <[email protected]>
Co-authored-by: Jeff Merrick <[email protected]>
Co-authored-by: skrydal <[email protected]>
Co-authored-by: AndreasHegerNuritas <[email protected]>
Co-authored-by: jayasimhankv <[email protected]>
Co-authored-by: Jay Kadambi <[email protected]>
Co-authored-by: David Leifker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants