Bundle store CLI commands #3828
Conversation
Co-authored-by: Ashwin Ramaswami <[email protected]>
This reverts commit eeaccd7.
| @@ -32,12 +44,10 @@ def test_bundle_store_workflow(self): | |||
| bundle_stores = self.bundle_manager._model.get_bundle_stores(self.user_id) | |||
| self.assertEqual(len(bundle_stores), 1) | |||
| self.assertEqual(bundle_stores[0].get("name"), "im-not-a-store") | |||
| # delete the store | |||
| # Deletion should succeed since there are bundle locations associated with the bundle store | |||
There was a problem hiding this comment.
FYI @adiprerepa I think your test had the opposite behavior / reverse logic, I had to reverse the logic here to get it to work correctly.
…ets into bundle-store-cli
| """ | ||
| return local.model.delete_bundle_store(request.user.user_id, uuid) | ||
| uuids = get_resource_ids(request.json, 'bundle_stores') |
There was a problem hiding this comment.
FYI @adiprerepa @andyjin2000 I had to change these REST endpoints to all accept data / return data in bulk, because that is the kind of REST endpoint that json_api_client.py uses. I'm happy to have a call later to discuss these changes further -- because I think it will help us to know how to best make REST endpoints in the future!
|
|
||
| class Meta: | ||
| type_ = 'bundle-store' | ||
| type_ = 'bundle_stores' |
There was a problem hiding this comment.
The type name must match the endpoint URL (/bundle_stores) as per how our json api client works
| ## Steps to set up GCP | ||
|
|
||
| ``` | ||
| cl store add --name gcp --storage-type gcp --url gcs://bucket1 |
|
@epicfaace Looks like some of the CLI tests are failing so I can't merge this right now. I think they are related to |
Flow is documented here: https://github.com/codalab/codalab-worksheets/blob/f864bb2f1303bcf722e1c204d35f0c4741bc716d/docs/Multiple-Bundle-Stores.md
Reasons for making this change
Included in this PR:
cl store add --name [store name]
Creates a new bundle store, allows one to configure credentials
cl store ls
Lists all bundle stores
cl store rm [store id]
Deletes bundle store with given id
cl upload --bundle-store [store name]
Uploads a bundle and specifies the bundle store. If no bundle store is specified, the CLI will pick the optimal available bundle store.
Not included in this PR:
cl replicate [bundle uuid] --bundle-store [store name]
Replicates the contents of the specified bundle to the specified bundle store.
cl store work [store name]
All bundles will by default be uploaded to this bundle store.
cl run --bundle-store [store name]
Runs a bundle and specifies the bundle store where results are uploaded. If no bundle store is specified, the worker will pick the optimal available bundle store.
cl rm -d --bundle-store [store name]
Updates existing command “cl rm -d”; keeps the bundle, but removes the bundle contents from the specified bundle store.
Related issues
#3803
#3827
Screenshots
Checklist