Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCS Object Store Interface #88

Merged
merged 13 commits into from
Jan 22, 2022
Merged

GCS Object Store Interface #88

merged 13 commits into from
Jan 22, 2022

Conversation

sarahwooders
Copy link
Contributor

Partially implemented GCSInterface added as an option for testing the replicator, and test script test_gcs_interface.py.

Note: Not all the object store interface functions have been implemented, and the gateway daemon still only works with S3. We can merge now or wait until the gateway integration is done.

@sarahwooders sarahwooders changed the title [WIP] GCS Object Store Interface [DRAFT] GCS Object Store Interface Jan 22, 2022
parasj
parasj approved these changes Jan 22, 2022
View reviewed changes
Copy link
Contributor

@parasj parasj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Can you add another if clause to the skylark cp command to copy objects GCS -> disk and disk -> GCS?

https://github.com/parasj/skylark/blob/e14a1b738a59cf4d2ccb54bd04693f351fb9b2fd/skylark/cli/cli.py#L67-L82

skylark/obj_store/gcs_interface.py Outdated


class GCSInterface(ObjectStoreInterface):
def __init__(self, gcp_region, bucket_name, use_tls=True, part_size=None, throughput_target_gbps=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since use_tls, part_size and throughput_target_gbps aren't used with GCS, take them out.

skylark/obj_store/gcs_interface.py Outdated Show resolved Hide resolved
@sarahwooders
Copy link
Contributor Author

I added the conditions and methods for skylark cp but left them unimplemented - can include that in the next commit.

@sarahwooders sarahwooders changed the title [DRAFT] GCS Object Store Interface GCS Object Store Interface Jan 22, 2022
@sarahwooders sarahwooders merged commit eda87ba into main Jan 22, 2022
@sarahwooders sarahwooders deleted the dev/gcs-support branch January 22, 2022 20:49
@sarahwooders sarahwooders mentioned this pull request Jan 24, 2022
Merged
sarahwooders added a commit that referenced this pull request Jan 24, 2022
@sarahwooders
GCSInterface integration with gateway + replicator (#91)
de86b77 
Integrate gateway/replicator with GCSInterface from #88, and complete implementing GCSInterface methods (e.g. returning futures for uploading/downloading objects).
parasj added a commit that referenced this pull request Jan 24, 2022
@parasj
Squashed commit of the following:
c8f6160 
commit 137d538
Author: Paras Jain <[email protected]>
Date:   Mon Jan 24 23:03:37 2022 +0000

    clean notebook

commit 481f5bc
Author: Paras Jain <[email protected]>
Date:   Mon Jan 24 20:34:27 2022 +0000

    more changes

commit 84cfc28
Author: Paras Jain <[email protected]>
Date:   Sat Jan 22 20:10:18 2022 +0000

    TODOs for solver

commit 8c7b2cf
Author: Paras Jain <[email protected]>
Date:   Sat Jan 22 04:38:31 2022 +0000

    update

commit 473f088
Author: Paras Jain <[email protected]>
Date:   Sat Jan 22 04:28:31 2022 +0000

    Fixes

commit 755dd74
Author: Paras Jain <[email protected]>
Date:   Fri Jan 21 22:30:34 2022 -0500

    Add missing eu-central-1 pair

commit 7ec967f
Author: Paras Jain <[email protected]>
Date:   Fri Jan 21 22:21:47 2022 -0500

    Add AWS transfer costs

commit c49bca1
Author: Paras Jain <[email protected]>
Date:   Fri Jan 21 15:49:43 2022 +0000

    reformat

commit 301e13f
Author: Paras Jain <[email protected]>
Date:   Fri Jan 21 10:04:21 2022 -0500

    Scale costs by number of instances

commit de86b77
Author: Sarah Wooders <[email protected]>
Date:   Mon Jan 24 14:46:44 2022 -0800

    `GCSInterface` integration with gateway + replicator  (#91)

    Integrate gateway/replicator with GCSInterface from #88, and complete implementing GCSInterface methods (e.g. returning futures for uploading/downloading objects).

commit 0d6e63a
Author: Paras Jain <[email protected]>
Date:   Mon Jan 24 14:32:41 2022 -0800

    Revert GCS interface changes due to merge conflict
parasj added a commit that referenced this pull request Jan 24, 2022
@parasj
Squashed commit of the following:
19f2002 
commit c8f6160
Author: Paras Jain <[email protected]>
Date:   Mon Jan 24 23:05:05 2022 +0000

    Squashed commit of the following:

    commit 137d538
    Author: Paras Jain <[email protected]>
    Date:   Mon Jan 24 23:03:37 2022 +0000

        clean notebook

    commit 481f5bc
    Author: Paras Jain <[email protected]>
    Date:   Mon Jan 24 20:34:27 2022 +0000

        more changes

    commit 84cfc28
    Author: Paras Jain <[email protected]>
    Date:   Sat Jan 22 20:10:18 2022 +0000

        TODOs for solver

    commit 8c7b2cf
    Author: Paras Jain <[email protected]>
    Date:   Sat Jan 22 04:38:31 2022 +0000

        update

    commit 473f088
    Author: Paras Jain <[email protected]>
    Date:   Sat Jan 22 04:28:31 2022 +0000

        Fixes

    commit 755dd74
    Author: Paras Jain <[email protected]>
    Date:   Fri Jan 21 22:30:34 2022 -0500

        Add missing eu-central-1 pair

    commit 7ec967f
    Author: Paras Jain <[email protected]>
    Date:   Fri Jan 21 22:21:47 2022 -0500

        Add AWS transfer costs

    commit c49bca1
    Author: Paras Jain <[email protected]>
    Date:   Fri Jan 21 15:49:43 2022 +0000

        reformat

    commit 301e13f
    Author: Paras Jain <[email protected]>
    Date:   Fri Jan 21 10:04:21 2022 -0500

        Scale costs by number of instances

    commit de86b77
    Author: Sarah Wooders <[email protected]>
    Date:   Mon Jan 24 14:46:44 2022 -0800

        `GCSInterface` integration with gateway + replicator  (#91)

        Integrate gateway/replicator with GCSInterface from #88, and complete implementing GCSInterface methods (e.g. returning futures for uploading/downloading objects).

    commit 0d6e63a
    Author: Paras Jain <[email protected]>
    Date:   Mon Jan 24 14:32:41 2022 -0800

        Revert GCS interface changes due to merge conflict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@parasj parasj parasj approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sarahwooders @parasj