Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/3911-cl-store infer storage type from url #4061

Merged
merged 27 commits into from
Apr 8, 2022
Merged

Conversation

wwwjn
Copy link
Contributor

@wwwjn wwwjn commented Apr 5, 2022

Reasons for making this change

Fix issue #3911

Design choices

Infer storage type from url in bundle_cli.py rather than bundles.py line 514, because the field storage_type database model BundleStoreSchema can not be NULL, so we need to infer the storage type before insert into database. And we need to know the storage type as soon as possible so I put it in cl command.

Changes

  1. Add auto test cases in test_cli.py
  2. Infer storage type from url in bundle_cli.py

Related issues

#3911

Screenshots

N/A

Checklist

  • I've added a screenshot of the changes, if this is a frontend change
  • I've added and/or updated tests, if this is a backend change
  • I've run the pre-commit.sh script
  • I've updated docs, if needed

@wwwjn wwwjn changed the title Fix/3911 cl store Fix/3911-cl-store infer storage type from url Apr 5, 2022
tests/cli/test_cli.py Outdated Show resolved Hide resolved
Co-authored-by: Ashwin Ramaswami <[email protected]>
@epicfaace
Copy link
Member

@wwwjn thanks, can you fix format?

@wwwjn
Copy link
Contributor Author

wwwjn commented Apr 5, 2022

@wwwjn thanks, can you fix format?

Yes! It seems that after run pre-commit.sh, I mistakenly change a file. Fixed now!

@wwwjn wwwjn requested a review from epicfaace April 5, 2022 20:48
Copy link
Member

@epicfaace epicfaace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Small update to make the code cleaner

codalab/lib/bundle_cli.py Show resolved Hide resolved
tests/cli/test_cli.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
@wwwjn wwwjn requested a review from epicfaace April 6, 2022 17:49
codalab/common.py Outdated Show resolved Hide resolved
codalab/common.py Outdated Show resolved Hide resolved
@wwwjn wwwjn requested a review from epicfaace April 7, 2022 22:38
codalab/common.py Outdated Show resolved Hide resolved
codalab/lib/bundle_cli.py Show resolved Hide resolved
codalab/lib/bundle_cli.py Outdated Show resolved Hide resolved
if args.url is not None:
inferred_type = parse_linked_bundle_url(args.url).storage_type
if args.storage_type is None:
bundle_store_info["storage_type"] = inferred_type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is bundle_store_info["storage_type"] set when args.storage_type is specified?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think No? When storage_type is specified, it should have higher priority than inferred_type. And if a user use a wrong command like "--storage_type azure_blob --url gs://bucket", we will raise Error. If the args.storage_type is specified and equals inferred_type, it's ok not to set it.

tests/cli/test_cli.py Outdated Show resolved Hide resolved
tests/cli/test_cli.py Outdated Show resolved Hide resolved
tests/cli/test_cli.py Show resolved Hide resolved
tests/cli/test_cli.py Outdated Show resolved Hide resolved
tests/cli/test_cli.py Outdated Show resolved Hide resolved
tests/cli/test_cli.py Outdated Show resolved Hide resolved
@wwwjn wwwjn merged commit 24cf1cb into master Apr 8, 2022
@wwwjn wwwjn deleted the fix/3911-cl-store branch April 8, 2022 03:41
@epicfaace epicfaace mentioned this pull request Apr 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants