Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest/dagster): Add automatic snowflake_pandas_io_manager asset capture #11189

Merged
merged 94 commits into from
Aug 28, 2024
Merged

feat(ingest/dagster): Add automatic snowflake_pandas_io_manager asset capture #11189

merged 94 commits into from
Aug 28, 2024

Commits on Mar 12, 2024

  1. Dagster source metadata ingestion code added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    1d5b308 View commit details
    Browse the repository at this point in the history

  2. Job failed and canceled execution metadata emit code added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    92e3cea View commit details
    Browse the repository at this point in the history

  3. Code changes as per review comment and dagster test cases added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    e006275 View commit details
    Browse the repository at this point in the history

  4. Dagster source integration doc added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    d24e27f View commit details
    Browse the repository at this point in the history

  5. Dagster sensor renamed to datahub sensor

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    28fcad2 View commit details
    Browse the repository at this point in the history

  6. Code and doc changes as per review comment

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    78220aa View commit details
    Browse the repository at this point in the history

  7. File reformatted

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    9b8b50a View commit details
    Browse the repository at this point in the history

  8. Lint error fixed

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    f7c6b08 View commit details
    Browse the repository at this point in the history

  9. Dagster package added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    fbd73d8 View commit details
    Browse the repository at this point in the history

  10. Temp changes

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    de5ad77 View commit details
    Browse the repository at this point in the history

  11. Dagster version set to >1.3.3

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    12bc920 View commit details
    Browse the repository at this point in the history

  12. Datahub dagster plugin code added

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a9c1a83 View commit details
    Browse the repository at this point in the history

  13. Extra init file removed

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    29bb752 View commit details
    Browse the repository at this point in the history

  14. cl failed error resolved

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    c419a61 View commit details
    Browse the repository at this point in the history

  15. Revert doc changes

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b996394 View commit details
    Browse the repository at this point in the history

  16. Added missing command in lint task

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    50e5724 View commit details
    Browse the repository at this point in the history

  17. Temp changes

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    e5e51fd View commit details
    Browse the repository at this point in the history

  18. Typo error resolved

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    d134b74 View commit details
    Browse the repository at this point in the history

  19. dataset entity added in datahub/api/entities. Dagster example files a… 

    …dded
    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    c7a3d18 View commit details
    Browse the repository at this point in the history

  20. lint error fixed

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    00cb35e View commit details
    Browse the repository at this point in the history

  21. Code changes as per review comment

    @shubhamjagtap639 @treff7es
    shubhamjagtap639 authored and treff7es committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a7efd7c View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. Initial commit of the reworked Dagster plugin

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    9fad5c4 View commit details
    Browse the repository at this point in the history

  2. Adding entities

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    74dc9cf View commit details
    Browse the repository at this point in the history

  3. Using java17 build

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    5f373dc View commit details
    Browse the repository at this point in the history

  4. Black formatting

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    281e666 View commit details
    Browse the repository at this point in the history

  5. Silenting some mypy error temporary

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    5ca6867 View commit details
    Browse the repository at this point in the history

  6. Black formatting

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    7eb7a8e View commit details
    Browse the repository at this point in the history

  7. Fixing test

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    059ec35 View commit details
    Browse the repository at this point in the history

  8. Add missing import

    @treff7es
    treff7es committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b7bb9e3 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. Fixing url generation

    @treff7es
    treff7es committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    6efc54f View commit details
    Browse the repository at this point in the history

  2. Fixing linter issues

    @treff7es
    treff7es committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    b48b061 View commit details
    Browse the repository at this point in the history

  3. Fixes

    @treff7es
    treff7es committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    5d68929 View commit details
    Browse the repository at this point in the history

  4. Fixing build

    @treff7es
    treff7es committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    c1f7557 View commit details
    Browse the repository at this point in the history

  5. Not pinning datahub client version

    @treff7es
    treff7es committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    512824d View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Adding way to capture assets

    @treff7es
    treff7es committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    f788ec5 View commit details
    Browse the repository at this point in the history

  2. Add way to test multiple dagster version

    @treff7es
    treff7es committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    508fcbc View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Adding way to bring your own lineage extractor

    @treff7es
    treff7es committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    8ec8f4e View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. Pr review fixes

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    63df8c1 View commit details
    Browse the repository at this point in the history

  2. Fixing imports

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    6bfe22d View commit details
    Browse the repository at this point in the history

  3. Fixing golden files and pydantic v2 issues with s3 tests

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    479c2cf View commit details
    Browse the repository at this point in the history

  4. Fixing test

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    80d9a89 View commit details
    Browse the repository at this point in the history

  5. Fixing tests 

    reverting path_spec change
    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    1450b83 View commit details
    Browse the repository at this point in the history

  6. Disabling Dagster tests

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    975b19d View commit details
    Browse the repository at this point in the history

  7. Reverting accidentally downgraded package

    @treff7es
    treff7es committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    295b91a View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. - Moving examples out from the package 

    - Removing tox.ini
    @treff7es
    treff7es committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    39f23cc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dagster_source-rebase

    @treff7es
    treff7es authored Mar 25, 2024
    Configuration menu
    Copy the full SHA
    186069e View commit details
    Browse the repository at this point in the history

  3. Fixing doc links

    @treff7es
    treff7es committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    85cff86 View commit details
    Browse the repository at this point in the history

  4. Fixing doc build concurrency issue

    @treff7es
    treff7es committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    d0f5365 View commit details
    Browse the repository at this point in the history

  5. Fixing typo

    @treff7es
    treff7es committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    39e2a7f View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Opinionated way to capture Asset inlets/outlets automatically

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    53b0560 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dagster_asset_dep_capture

    @treff7es
    treff7es authored May 27, 2024
    Configuration menu
    Copy the full SHA
    bc4cbca View commit details
    Browse the repository at this point in the history

  3. Fixing lint issues

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e7b31c2 View commit details
    Browse the repository at this point in the history

  4. revertin change

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    86bafd5 View commit details
    Browse the repository at this point in the history

  5. delete file

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e2df73e View commit details
    Browse the repository at this point in the history

  6. linting

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0b81dbe View commit details
    Browse the repository at this point in the history

  7. Mypy fixes

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    6156651 View commit details
    Browse the repository at this point in the history

  8. isort

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    a79af54 View commit details
    Browse the repository at this point in the history

  9. mypying

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    b40a886 View commit details
    Browse the repository at this point in the history

  10. Isort

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    4ac71fc View commit details
    Browse the repository at this point in the history

  11. Additional check to make mypy happy

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    ef0851e View commit details
    Browse the repository at this point in the history

  12. fix

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    40c5537 View commit details
    Browse the repository at this point in the history

  13. Mypy fixes

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    f169dfe View commit details
    Browse the repository at this point in the history

  14. Adding sqlglot as dep

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0dc07b8 View commit details
    Browse the repository at this point in the history

  15. fix

    @treff7es
    treff7es committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e17830d View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Updating doc and various refactors

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    c0e0cd1 View commit details
    Browse the repository at this point in the history

  2. Fixing example formatting

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    4655a94 View commit details
    Browse the repository at this point in the history

  3. isorting

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    127c516 View commit details
    Browse the repository at this point in the history

  4. remove unneded deps

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    0b0dc95 View commit details
    Browse the repository at this point in the history

  5. Fixing mypy issues

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    da4af37 View commit details
    Browse the repository at this point in the history

  6. Fix

    @treff7es
    treff7es committed May 28, 2024
    Configuration menu
    Copy the full SHA
    553cf2f View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Update dagster.md 

    rewriting
    @jjoyce0510
    jjoyce0510 authored Aug 8, 2024
    Configuration menu
    Copy the full SHA
    64ac9f0 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Add snowflake_pandas_io_manager drop in replacement to automatically … 

    …capture snowflake datasets
    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    3802dba View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dagster_asset_dep_capture

    @treff7es
    treff7es authored Aug 15, 2024
    Configuration menu
    Copy the full SHA
    1084300 View commit details
    Browse the repository at this point in the history

  3. Adding missing init

    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    6ee5e6f View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/dagster_asset_dep_capture' int… 

    …o dagster_asset_dep_capture

# Conflicts:
#	docs/lineage/dagster.md
    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    c45a82b View commit details
    Browse the repository at this point in the history

  5. lint fix

    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    5eb067e View commit details
    Browse the repository at this point in the history

  6. remove accidentally added import

    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    aeb7b79 View commit details
    Browse the repository at this point in the history

  7. mypy fix

    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    a3b94c2 View commit details
    Browse the repository at this point in the history

  8. Isorting

    @treff7es
    treff7es committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    719d3c8 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. Fixing datajob input/output type 

    Lineage fix
    @treff7es
    treff7es committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    63a4862 View commit details
    Browse the repository at this point in the history

  2. Fix set type

    @treff7es
    treff7es committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    e077c87 View commit details
    Browse the repository at this point in the history

  3. Not parsing select queries

    @treff7es
    treff7es committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    7d85629 View commit details
    Browse the repository at this point in the history

  4. Update doc

    @treff7es
    treff7es committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    72e175c View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Merge branch 'master' into dagster_asset_dep_capture

    @treff7es
    treff7es authored Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b866f8b View commit details
    Browse the repository at this point in the history

  2. Generating status aspect for dataflow/datajobs

    @treff7es
    treff7es committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    21aa495 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dagster_asset_dep_capture' of github.com:treff7es/datah… 

    …ub into dagster_asset_dep_capture
    @treff7es
    treff7es committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c80d9e5 View commit details
    Browse the repository at this point in the history

  4. Adding more debug log

    @treff7es
    treff7es committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c0f3e3d View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Fix airflow unit tests

    @treff7es
    treff7es committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    ce9eae6 View commit details
    Browse the repository at this point in the history

  2. Fix golden files

    @treff7es
    treff7es committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    deef278 View commit details
    Browse the repository at this point in the history

  3. Add option to enable more verbose logging

    @treff7es
    treff7es committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    319b132 View commit details
    Browse the repository at this point in the history

  4. fix black formatting

    @treff7es
    treff7es committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    aab0f67 View commit details
    Browse the repository at this point in the history

  5. Add doc about the supported version

    @treff7es
    treff7es committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    a1bd12c View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Update dagster.md

    @jjoyce0510
    jjoyce0510 authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f1065ce View commit details
    Browse the repository at this point in the history

  2. Update dagster.md

    @jjoyce0510
    jjoyce0510 authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    c1aeed1 View commit details
    Browse the repository at this point in the history