Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set rwlock on all dvcfiles when load/dump-ing #3789

Closed
skshetry opened this issue May 13, 2020 · 3 comments
Closed

set rwlock on all dvcfiles when load/dump-ing #3789

skshetry opened this issue May 13, 2020 · 3 comments
Labels
p3-nice-to-have It should be done this or next sprint

Comments

@skshetry
Copy link
Member

skshetry commented May 13, 2020

We support running repro/run in parallel, so, there might be some chance of things getting overwritten between the load-serialize-dump process, creating a "last writer wins" situation.

@skshetry skshetry added the p0-critical Critical issue. Needs to be fixed ASAP. label May 13, 2020
@skshetry skshetry added this to the 1.0 milestone May 13, 2020
@efiop
Copy link
Contributor

efiop commented May 13, 2020

Related #3653

@skshetry skshetry added p3-nice-to-have It should be done this or next sprint and removed p0-critical Critical issue. Needs to be fixed ASAP. labels May 18, 2020
@skshetry
Copy link
Member Author

skshetry commented May 18, 2020

Looks like we don't completely allow running parallel execution, as we lock the repo most of the times and only unlock when the command is really running.

dvc/dvc/stage/run.py

Lines 37 to 38 in afbb3a0

@unlocked_repo
def cmd_run(stage, *args, **kwargs):

As the collection of stages happen when the repo is locked in, only one process of dvc can really
be running at that point of time. So, I'm reducing it's priority for now and removing from 1.0 milestone.

But, if we later come to remove that lock or make amends to it to allow parallel execution at all times, this will be p0-critical.

@skshetry skshetry removed this from the 1.0 milestone May 18, 2020
@skshetry skshetry removed the dvc.yaml label Aug 22, 2020
@skshetry
Copy link
Member Author

skshetry commented Jan 3, 2023

Closing as stale.

@skshetry skshetry closed this as not planned Won't fix, can't repro, duplicate, stale Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p3-nice-to-have It should be done this or next sprint
Projects
None yet
Development

No branches or pull requests

2 participants