-
Notifications
You must be signed in to change notification settings - Fork 386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
replace dvc run in cmd-ref #3223
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,8 +15,8 @@ does not change directories in your terminal). | |
- Copy data files or dataset directories for modeling into the repository, and | ||
track them with DVC using the `dvc add` command. | ||
- Process the data with your own source code, using `dvc.yaml` and/or the | ||
`dvc run` command, specifying further <abbr>outputs</abbr> that should also be | ||
tracked by DVC after the code is executed. | ||
`dvc stage add` command to specify further <abbr>outputs</abbr> that should | ||
also be tracked by DVC, and executing the code using `dvc repro`. | ||
Comment on lines
17
to
+19
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we should keep this simpler by not mentioning |
||
- Sharing a <abbr>DVC repository</abbr> with the codified data | ||
[pipeline](/doc/command-reference/dag) will not include the project's | ||
<abbr>cache</abbr>. Use [remote storage](/doc/command-reference/remote) and | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -27,8 +27,8 @@ plots are generated with `dvc plot show` or `dvc plot diff`. This command sets | |||||
(or unsets) default display properties for a specific metrics file. | ||||||
|
||||||
The path to the metrics file `target` is required. It must be listed in a | ||||||
`dvc.yaml` file (see the `--plots` option of `dvc run`). `dvc plots modify` adds | ||||||
the display properties to `dvc.yaml`. | ||||||
`dvc.yaml` file (see the `--plots` option of `dvc stage add`). | ||||||
`dvc plots modify` adds the display properties to `dvc.yaml`. | ||||||
|
||||||
Property names are passed as [options](#options) to this command (prefixed with | ||||||
`--`). These are based on the [Vega-Lite](https://vega.github.io/vega-lite/) | ||||||
|
@@ -134,8 +134,8 @@ plots: | |||||
|
||||||
## Example: Template change | ||||||
|
||||||
_dvc run --plots file.csv ..._ command assign the default template that needs to | ||||||
be changed in many cases. A simple command changes the template: | ||||||
_dvc stage add --plots file.csv ..._ command assign the default template that | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should render correctly now:
Suggested change
It's just a bit unclear why we're referring to such a specific |
||||||
needs to be changed in many cases. A simple command changes the template: | ||||||
|
||||||
```dvc | ||||||
$ dvc plots modify classes.csv --template confusion | ||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -30,7 +30,8 @@ are run one after the other in the order they are defined. The failure of any | |
command will halt the remaining stage execution, and raises an error. | ||
|
||
> Pipeline stages are defined in `dvc.yaml` (either manually or by using | ||
> `dvc run`) while initial data dependencies can be registered with `dvc add`. | ||
> `dvc stage add`) while initial data dependencies can be registered with | ||
> `dvc add`. | ||
|
||
`dvc repro` is similar to [Make](https://www.gnu.org/software/make/) in software | ||
build automation, but DVC captures build requirements | ||
|
@@ -137,8 +138,8 @@ up-to-date and only execute the final stage. | |
`dvc commit` to finish the operation. | ||
|
||
- `-m`, `--metrics` - show metrics after reproduction. The target pipelines must | ||
have at least one metrics file defined either with `dvc metrics` or by the | ||
`-M` or `-m` options of `dvc run` | ||
have at least one [metrics](/doc/command-reference/metrics) file defined in | ||
`dvc.yaml`. | ||
jorgeorpinel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- `--dry` - only print the commands that would be executed without actually | ||
executing the commands. | ||
|
@@ -170,10 +171,10 @@ up-to-date and only execute the final stage. | |
stages (`A` and below) depend on `requirements.txt`, we can specify it in `A`, | ||
and omit it in `B` and `C`. | ||
|
||
Like with the `--force` option on `dvc run`, this is a way to force-execute | ||
stages without changes. This can also be useful for pipelines containing | ||
stages that produce non-deterministic (semi-random) outputs, where outputs can | ||
vary on each execution, meaning the cache cannot be trusted for such stages. | ||
This is a way to force-execute stages without changes. This can also be useful | ||
for pipelines containing stages that produce non-deterministic (semi-random) | ||
outputs, where outputs can vary on each execution, meaning the cache cannot be | ||
trusted for such stages. | ||
|
||
- `--downstream` - only execute the stages after the given `targets` in their | ||
corresponding pipelines, including the target stages themselves. This option | ||
|
@@ -213,11 +214,13 @@ best | |
And runs a few simple transformations to filter and count numbers: | ||
|
||
```dvc | ||
$ dvc run -n filter -d text.txt -o numbers.txt \ | ||
$ dvc stage add -n filter -d text.txt -o numbers.txt \ | ||
"cat text.txt | egrep '[0-9]+' > numbers.txt" | ||
|
||
$ dvc run -n count -d numbers.txt -d process.py -M count.txt \ | ||
$ dvc stage add -n count -d numbers.txt -d process.py -M count.txt \ | ||
"python process.py numbers.txt > count.txt" | ||
|
||
$ dvc repro | ||
``` | ||
|
||
Where `process.py` is a script that, for simplicity, just prints the number of | ||
|
@@ -232,7 +235,7 @@ with open(sys.argv[1], 'r') as f: | |
print(num_lines) | ||
``` | ||
|
||
The result of executing these `dvc run` commands should look like this: | ||
The result of executing `dvc repro` should look like this: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The script is about There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @iesahin! PTAL. |
||
|
||
```dvc | ||
$ tree | ||
|
@@ -248,18 +251,8 @@ $ tree | |
You may want to check the contents of `dvc.lock` and `count.txt` for later | ||
reference. | ||
|
||
Ok, now let's run `dvc repro`: | ||
|
||
```dvc | ||
$ dvc repro | ||
Stage 'filter' didn't change, skipping | ||
Stage 'count' didn't change, skipping | ||
Data and pipelines are up to date. | ||
``` | ||
|
||
It makes sense, since we haven't changed any of the dependencies of this | ||
pipeline (`text.txt` and `process.py`). Now, let's imagine we want to print a | ||
description and we add this line to the `process.py`: | ||
Now, let's imagine we want to print a description and we add this line to the | ||
`process.py`: | ||
|
||
```python | ||
... | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -57,7 +57,7 @@ description_, as detailed below: | |||||
|
||||||
- _always changed_ means that this is a `.dvc` file with no dependencies (see | ||||||
`dvc add`) or that the stage in `dvc.yaml` has the `always_changed: true` | ||||||
value set (see `--always-changed` option in `dvc run`). | ||||||
value set (see `--always-changed` option in `dvc stage add`). | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually these can now link directly to the option anchor like this 🙂
Suggested change
But we prob. need a separate issue to update these There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Extracted to #3236 |
||||||
|
||||||
- _changed deps_ or _changed outs_ means that there are changes in dependencies | ||||||
or outputs tracked by the stage or `.dvc` file. Depending on the use case, | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💅 and -> while ?