Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow skipping certain git revisions/commits in DVC #5066

Closed
skshetry opened this issue Dec 9, 2020 · 6 comments
Closed

allow skipping certain git revisions/commits in DVC #5066

skshetry opened this issue Dec 9, 2020 · 6 comments
Labels
feature request Requesting a new feature p2-medium Medium priority, should be done, but less important

Comments

@skshetry
Copy link
Member

skshetry commented Dec 9, 2020

If the dvc.yaml files are corrupted in certain revisions, dvc just fails and there's no way to skip that.
@courentin raised #3885 for this and again on #5037 to specifically address this.

For most parts of DVC, it might be okay and we can also teach params/metrics/plots to cope with failures in certain revisions.
But, for other commands such as push/pull/gc, dvc might just fail, even if we skip those failing commits, it'd be better not to skip and ask users to make their intentions clearer to provide stronger "data versioning" requirements.

And, for gc, it's even dangerous, and for push/pull/gc`, it might just be confusing.

So, coming to my proposal, it'd be better if we could provide a way for users to tell DVC to skip certain commits.
One of the ways would be to provide --ignore-revs or similarly named flags in push/pull/gc, etc.

But, it might not be always easier to use those flags and might be cumbersome especially in CI.
To solve this, we can provide a config (eg: ignore_revsFile) where you could provide a file with list of revisions to ignore:

# .dvc/config
[core]
ignore_revsFile = .dvc-ignore-revs

# .dvc-ignore-revs
# list of revisions for the DVC to ignore
a12ef
12345

The file will always be read from the latest checked out commit.

If the file is cumbersome, we could instead provide a way of adding a list of revs in .dvc/config as they are always read from the workspace.

@skshetry skshetry added the feature request Requesting a new feature label Dec 9, 2020
@efiop
Copy link
Contributor

efiop commented Dec 9, 2020

Could consider creating a gc-specific section for it and for future gc-related config options.

@efiop efiop added the p1-important Important, aka current backlog of things to do label Dec 9, 2020
@efiop
Copy link
Contributor

efiop commented Dec 9, 2020

Viewer team might find this interesting too CC @Suor

@Suor
Copy link
Contributor

Suor commented Dec 9, 2020

We simply skip broken commits, not sure this will affect viewer in any way.

@courentin
Copy link
Contributor

I feel having a .dvc-ignore-revs is a good idea indeed!

@Suor
Copy link
Contributor

Suor commented Dec 28, 2020

Might be used to silent errors in viewer too, I guess. And skipping is faster than parsing of cause.

@dberenbaum
Copy link
Collaborator

Some related work was done in #5984 for certain commands to skip bad revisions. Specifying revisions to ignore would still be a nice feature, but I'm lowering the priority for now since it's not in our current roadmap.

@dberenbaum dberenbaum added p2-medium Medium priority, should be done, but less important and removed p1-important Important, aka current backlog of things to do labels Feb 18, 2022
daavoo added a commit that referenced this issue Aug 5, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
@daavoo daavoo mentioned this issue Aug 5, 2022
2 tasks
daavoo added a commit that referenced this issue Aug 5, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
daavoo added a commit that referenced this issue Aug 5, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
daavoo added a commit that referenced this issue Aug 8, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
daavoo added a commit that referenced this issue Aug 8, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
daavoo added a commit that referenced this issue Aug 8, 2022
Support to write a list of line-separated revisions to `.dvc/ignore_revs`.
If `.dvc/ignore_revs` exists, parse the revisions and use them in `brancher` to skip.

Allows to skip broken commits in `gc`, `pull` and `push`.

Closes #5037
Closes #5066
Closes #7585
@mattseddon mattseddon closed this as not planned Won't fix, can't repro, duplicate, stale Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Requesting a new feature p2-medium Medium priority, should be done, but less important
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants