Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data pond: expose readable datasets as dataframes and arrow tables #1507

Merged
merged 119 commits into from
Oct 8, 2024

Commits on Jun 19, 2024

  1. add simple ibis helper

    sh-rp committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    af6a40e View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. Configuration menu
    Copy the full SHA
    3a69ece View commit details
    Browse the repository at this point in the history
  2. a bit more work

    sh-rp committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    4324650 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. first simple implementation

    sh-rp committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    7c960df View commit details
    Browse the repository at this point in the history
  2. small change

    sh-rp committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    86b89ac View commit details
    Browse the repository at this point in the history
  3. more work on dataset

    sh-rp committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    5a8ea54 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. Configuration menu
    Copy the full SHA
    36e94af View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. add support for parquet files and compression on jsonl files in files…

    …ystem dataframe implementation
    sh-rp committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    20bf9ce View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/common/destination/reference.py
    #	dlt/destinations/sql_client.py
    #	dlt/pipeline/pipeline.py
    sh-rp committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    6dce626 View commit details
    Browse the repository at this point in the history
  2. fix test after devel merge

    sh-rp committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    a0ff55f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c297e96 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. small updates to demo

    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    d020403 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/destinations/impl/filesystem/filesystem.py
    sh-rp committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    5c3db47 View commit details
    Browse the repository at this point in the history
  2. enable tests for all bucket providers

    remove resource based dataset accessor
    sh-rp committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    79ef7dd View commit details
    Browse the repository at this point in the history
  3. fix tests

    sh-rp committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    ff40079 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ac415b9 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c92a527 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    13ec73b View commit details
    Browse the repository at this point in the history
  7. move code for accessing frames and tables to the cursor and use duckd…

    …b dbapi cursor in filesystem
    sh-rp committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    46e0226 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Configuration menu
    Copy the full SHA
    7cf69a7 View commit details
    Browse the repository at this point in the history
  2. some small changes

    sh-rp committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    6ffe302 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c200262 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    226454f View commit details
    Browse the repository at this point in the history
  5. change iter functions to always expect chunk size (None will default …

    …to full frame/table)
    sh-rp committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    3296e63 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6f6500f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    152b788 View commit details
    Browse the repository at this point in the history
  8. fix filesystem test

    make filesystem duckdb instance use glob pattern
    sh-rp committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    6d73bc5 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    bdb39ba View commit details
    Browse the repository at this point in the history
  10. fix empty dataframe retrieval

    sh-rp committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    3ead92b View commit details
    Browse the repository at this point in the history
  11. remove old df test

    sh-rp committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    9fcbd00 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. clean up interfaces a bit (more to come?)

    remove pipeline dependency from dataset
    sh-rp committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    28ee1c6 View commit details
    Browse the repository at this point in the history
  2. move dataset creation into destination client and clean up interfaces…

    … / reference a bit more
    sh-rp committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    28cb282 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    77926fa View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6ef04bc View commit details
    Browse the repository at this point in the history
  5. fix tests for snowflake

    sh-rp committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    ec13b49 View commit details
    Browse the repository at this point in the history
  6. make data set a function

    sh-rp committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    b222d1d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    9f0a6a5 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Configuration menu
    Copy the full SHA
    289b63c View commit details
    Browse the repository at this point in the history
  2. fix example pipeline

    sh-rp committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    779bca6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    584ab47 View commit details
    Browse the repository at this point in the history
  4. add comments

    sh-rp committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    6594053 View commit details
    Browse the repository at this point in the history
  5. rename sql to query

    remove unneeded code
    sh-rp committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    9e0a61d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    dd47326 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/common/destination/reference.py
    #	dlt/destinations/fs_client.py
    #	dlt/destinations/impl/filesystem/filesystem.py
    #	poetry.lock
    #	pyproject.toml
    #	tests/load/test_sql_client.py
    sh-rp committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    9f8f79b View commit details
    Browse the repository at this point in the history
  2. post merge cleanups

    sh-rp committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    fda1cb5 View commit details
    Browse the repository at this point in the history
  3. move imports around a bit

    sh-rp committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c7a0e05 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. exclude abfss buckets from test

    sh-rp committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    8497036 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3dc2c90 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d6bec38 View commit details
    Browse the repository at this point in the history
  4. fix bug

    sh-rp committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    62ea3ba View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3fd4d61 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	poetry.lock
    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    eeca4ac View commit details
    Browse the repository at this point in the history
  2. add sql glot to filesystem deps

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    52f8523 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    90c669a View commit details
    Browse the repository at this point in the history
  4. move cursor columns location

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    7657fb1 View commit details
    Browse the repository at this point in the history
  5. fix snowflake and mssql

    disable tests with sftp
    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    352b238 View commit details
    Browse the repository at this point in the history
  6. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/common/destination/reference.py
    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5fadeeb View commit details
    Browse the repository at this point in the history
  7. clean up compose files a bit

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9a1752d View commit details
    Browse the repository at this point in the history
  8. fix sqlalchemy

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a77192f View commit details
    Browse the repository at this point in the history
  9. add mysql docker compose file

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    420eaf1 View commit details
    Browse the repository at this point in the history
  10. fix linting

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    97e2757 View commit details
    Browse the repository at this point in the history
  11. prepare hint checking

    sh-rp committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    df4f6d0 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. disable part of state test

    sh-rp committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    6b27b98 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. enable hint check

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    ffba901 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fab5232 View commit details
    Browse the repository at this point in the history
  3. rename dataset implementation to DBAPI

    remove dataset specific code from destination client
    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    0de4a6c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    077a25a View commit details
    Browse the repository at this point in the history
  5. remove example pipeline

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    13a759b View commit details
    Browse the repository at this point in the history
  6. rename test_decimal_name

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    10e04d6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    5077ce1 View commit details
    Browse the repository at this point in the history
  8. rename df methods to pandas

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    1025560 View commit details
    Browse the repository at this point in the history
  9. fix bug in default columns

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f8927d3 View commit details
    Browse the repository at this point in the history
  10. fix hints test and columns bug

    removes some uneeded code
    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    7fd3c62 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    3a76178 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    27104e3 View commit details
    Browse the repository at this point in the history
  13. fix docs

    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    1c06d11 View commit details
    Browse the repository at this point in the history
  14. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	docs/website/docs/dlt-ecosystem/transformations/pandas.md
    sh-rp committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e5b3688 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Configuration menu
    Copy the full SHA
    7d09bdb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dbe4baa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f4e0099 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/destinations/impl/filesystem/filesystem.py
    #	poetry.lock
    sh-rp committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    80fe898 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	dlt/pipeline/pipeline.py
    sh-rp committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    d698cd5 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. rename readablerelation methods

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    857803c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8055529 View commit details
    Browse the repository at this point in the history
  3. update dependencies

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    24c7308 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    76759cf View commit details
    Browse the repository at this point in the history
  5. update types

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    d3d8381 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f9a766d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b6c7fbc View commit details
    Browse the repository at this point in the history
  8. fix sql client

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    86fc914 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    58380ec View commit details
    Browse the repository at this point in the history
  10. some typing

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0a24b3a View commit details
    Browse the repository at this point in the history
  11. add dataframes tests back in

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    bef50d7 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    b13e492 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    92ea515 View commit details
    Browse the repository at this point in the history
  14. fix snippets

    sh-rp committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    e1fa308 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    a7958d5 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Configuration menu
    Copy the full SHA
    ed197ea View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ec1656 View commit details
    Browse the repository at this point in the history
  3. add support for delta tables

    sh-rp committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9cd4173 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7dba771 View commit details
    Browse the repository at this point in the history
  5. fix delta tests

    sh-rp committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    3e96a6c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    355f5b6 View commit details
    Browse the repository at this point in the history
  7. try fix azure tests again

    sh-rp committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9002f02 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. fix df access tests

    sh-rp committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    c3050d4 View commit details
    Browse the repository at this point in the history
  2. PR fixes

    sh-rp committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    bbc0525 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ef148c3 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'devel' into exp/1095-expose-readable-datasets

    # Conflicts:
    #	docs/website/docs/dlt-ecosystem/visualizations/exploring-the-data.md
    sh-rp committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    a99e987 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. correct internal table access

    sh-rp committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    eaf1cd8 View commit details
    Browse the repository at this point in the history
  2. allow datasets without schema

    sh-rp committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    6bb7117 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2024

  1. Configuration menu
    Copy the full SHA
    6648b86 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Configuration menu
    Copy the full SHA
    89a9861 View commit details
    Browse the repository at this point in the history
  2. fix sql client tests

    sh-rp committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    631d50b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8e2e37c View commit details
    Browse the repository at this point in the history
  4. fix gs test

    sh-rp committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    dc383fc View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    41926ae View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9b8437a View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. clear duckdb secrets

    sh-rp committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    5d14045 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fb9a445 View commit details
    Browse the repository at this point in the history