Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache relpath on stage #4053

Merged
merged 1 commit into from
Jun 17, 2020
Merged

cache relpath on stage #4053

merged 1 commit into from
Jun 17, 2020

Conversation

skshetry
Copy link
Member

@skshetry skshetry commented Jun 16, 2020

On top of #4052

A lot of relpath calls were being used by networkx indirectly through Stage::__eq__() and Stage::__hash__(). They would call relpath and abspath calls. This PR caches them.

Compare with #4052 with following:

Screenshot from 2020-06-16 20-26-29

Related #3653

  • ❗ I have followed the Contributing to DVC checklist.

  • 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here. If the CLI API is changed, I have updated tab completion scripts.

  • ❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

@skshetry skshetry added enhancement Enhances DVC performance improvement over resource / time consuming tasks labels Jun 16, 2020
@skshetry skshetry self-assigned this Jun 16, 2020
@skshetry skshetry changed the title [WIP]: Cache relpath [WIP]: Cache relpath on stage Jun 16, 2020
@skshetry skshetry changed the title [WIP]: Cache relpath on stage cache relpath on stage Jun 16, 2020
@skshetry skshetry marked this pull request as ready for review June 16, 2020 17:32
@skshetry skshetry requested a review from efiop June 16, 2020 17:32
Comment on lines 222 to +226
with folder.chdir():
# `Stage` caches `relpath` results, forcing it to reset
stage1.path = stage1.path
stage2.path = stage2.path

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugly hack, chdir is now not supported as we cache the relpath. It's fine for CLI use-case, but for API usecases, it sucks.

@efiop efiop merged commit 5d957b1 into iterative:master Jun 17, 2020
@skshetry skshetry deleted the cache-relpath branch June 17, 2020 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhances DVC performance improvement over resource / time consuming tasks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants