guide: DVC Experiments Overview

content/docs/command-reference/exp/apply.md

@@ -15,21 +15,25 @@ positional arguments:
 ## Description
 
 Restores an `experiment` into the workspace as long as no more Git commits have
-been made after the target experiment (`HEAD` hasn't moved). The `experiment`
-can be referenced by name or hash (see `dvc exp run` for details). This changes
-any files (code, data, <abbr>parameters</abbr>, <abbr>metrics</abbr>, etc.)
-needed to reflect the experiment conditions and results in the workspace.
+been made after the target experiment (`HEAD` hasn't moved). The experiment can
+be referenced by name or hash (see `dvc exp run` for details).
 
-⚠️ Conflicting changes in the workspace are overwritten unless `--no-force` is
-used.
+Specifically, `dvc exp apply` changes any files (code, data,
+<abbr>parameters</abbr>, <abbr>metrics</abbr>, etc.) needed to reflect the
+experiment conditions and results in the workspace. Current changes to the
+workspace are preserved except if they conflict with the experiment in question.
+
+⚠️ Conflicting changes in the workspace are overwritten unless unless
+`--no-force` is used.
 
 This is typically used after choosing a target `experiment` with `dvc exp show`
-or `dvc exp diff`, and before committing it to Git (making it
-[persistent](/doc/user-guide/experiment-management#persistent-experiments)).
+or `dvc exp diff`, and before committing it to Git (making it [persistent].
+
+> Note that if a history of [checkpoints] is found in the `experiment`, it will
+> **not** be preserved when applying and committing it.
 
-Note that the history of
-[checkpoints](/doc/command-reference/exp/run#checkpoints) found in the
-`experiment` is **not** preserved when applying and committing it.
+[persistent]: /doc/user-guide/experiment-management/persisting-experiments
+[checkpoints]: /doc/user-guide/experiment-management/checkpoints
 
 ## Options
 

content/docs/command-reference/exp/branch.md

@@ -15,26 +15,28 @@ positional arguments:
 
 ## Description
 
-Makes a named Git
-[`branch`](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging)
-containing the target `experiment` (making it
-[persistent](/doc/user-guide/experiment-management#persistent-experiments)). For
-[checkpoint experiments](/doc/command-reference/exp/run#checkpoints), the new
-branch will contain multiple commits (the checkpoints).
+Makes a named Git [`branch`] containing the target `experiment` (making it
+[persistent]. For [checkpoint experiments], the new branch will contain multiple
+commits (the checkpoints).
 
 The new `branch` will be based on the experiment's parent commit (`HEAD` at the
 time that the experiment was run). Note that DVC **does not** switch into the
 new `branch` automatically.
 
 `dvc exp branch` is useful to make an experiment persistent without modifying
-the workspace, so they can be continued,
-[stored, and shared](https://dvc.org/doc/use-cases/sharing-data-and-model-files)
-in a normal Git + DVC workflow.
+the workspace, so they can be continued, [stored and shared] in a normal Git +
+DVC workflow.
 
 To switch into the new branch, use `git checkout branch` and `dvc checkout`. Or
 use `git merge branch` and `dvc repro` to combine it with your current project
 version.
 
+[`branch`]:
+  https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging
+[persistent]: /doc/user-guide/experiment-management/persisting-experiments
+[checkpoint experiments]: /doc/command-reference/exp/run#checkpoints
+[stored and shared]: /doc/use-cases/sharing-data-and-model-files
+
 ## Options
 
 - `-h`, `--help` - shows the help message and exit.

content/docs/command-reference/exp/init.md

@@ -3,6 +3,9 @@
 Codify project using [DVC metafiles](/doc/user-guide/project-structure) to run
 [experiments](/doc/user-guide/experiment-management).
 
+> Requires a <abbr>DVC repository</abbr>, created with `git init` and
+> `dvc init`.
+
 ## Synopsis
 
 ```usage
@@ -32,6 +35,11 @@ training of machine learning models.
 This command is intended to be a quick way to start running experiments. To
 create more complex stages and pipeliens, use `dvc stage add`.
 
+> 📖 More context in [Experiments Overview].
+
+[experiments overview]:
+  /doc/user-guide/experiment-management/experiments-overview
+
 ### The `command` argument
 
 The `command` argument is optional, if you are using `--interactive` mode. The

content/docs/command-reference/exp/run.md

@@ -22,45 +22,40 @@ Provides a way to execute and track <abbr>experiments</abbr> in your
 <abbr>project</abbr> without polluting it with unnecessary commits, branches,
 directories, etc.
 
-> `dvc exp run` is equivalent to `dvc repro` for experiments. It has the same
-> behavior when it comes to `targets` and stage execution (restores the
-> dependency graph, etc.). See the command [options](#options) for more on the
-> differences.
+> `dvc exp run` is equivalent to `dvc repro` for <abbr>experiments</abbr>. It
+> has the same behavior when it comes to `targets` and stage execution (restores
+> the dependency graph, etc.). See the command [options](#options) for more on
+> the differences.
 
 Before running an experiment, you'll probably want to make modifications such as
 data and code updates, or <abbr>hyperparameter</abbr> tuning. For the latter,
 you can use the `--set-param` (`-S`) option of this command to change
 `dvc param` values on-the fly.
 
+📖 See [DVC Experiments](/doc/user-guide/experiment-management) for more
+information.
+
 Each experiment creates and tracks a project variation based on your
 <abbr>workspace</abbr> changes. Experiments will have a unique, auto-generated
 name like `exp-bfe64` by default, which can be customized using the `--name`
 (`-n`) option.
 
-<details>
-
-### ⚙️ How does DVC track experiments?
-
-Experiments are custom
-[Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References)
-(found in `.git/refs/exps`) with a single commit based on `HEAD` (not checked
-out by DVC). Note that these commits are not pushed to the Git remote by default
-(see `dvc exp push`).
+Each experiment creates and tracks a project variation based on the changes in
+your <abbr>workspace</abbr>. The results of the last `dvc exp run` will be
+reflected in the workspace. Experiments will have an auto-generated ID like
+`exp-bfe64` by default. A custom name can be given instead, using the `--name`
+(`-n`) option
 
-</details>
-
-The results of the last `dvc exp run` can be seen in the workspace. To display
-and compare multiple experiments, use `dvc exp show` or `dvc exp diff`
-(`plots diff` also accepts experiment names as `revisions`). Use `dvc exp apply`
-to restore the results of any other experiment instead.
-
-Successful experiments can be made
-[persistent](/doc/user-guide/experiment-management#persistent-experiments) by
-committing them to the Git repo. Unnecessary ones can be removed with
+To display and compare multiple experiments, use `dvc exp show` or
+`dvc exp diff` (`plots diff` also accepts experiment names as `revisions`). Use
+`dvc exp apply` to restore the results of any experiment, for example to [commit
+them][persisting] to Git. Unnecessary experiments can be removed with
 `dvc exp remove`or `dvc exp gc` (or abandoned).
 
-> Note that experiment data will remain in the <abbr>cache</abbr> until you use
-> regular `dvc gc` to clean it up.
+> Note that experiment data will remain in the local <abbr>cache</abbr> until
+> you use regular `dvc gc` to clean it up.
+
+[persisting]: /doc/user-guide/experiment-management/persisting-experiments
 
 ## Checkpoints
 

content/docs/sidebar.json

@@ -148,6 +148,7 @@
         "slug": "experiment-management",
         "source": "experiment-management/index.md",
         "children": [
+          "experiments-overview",
           "running-experiments",
           "comparing-experiments",
           "sharing-experiments",

content/docs/user-guide/experiment-management/cleaning-experiments.md

@@ -2,7 +2,9 @@
 
 Although DVC uses minimal resources to keep track of the experiments, they may
 clutter tables and the workspace. DVC allows to remove specific experiments from
-the workspace or delete all not-yet-persisted experiments at once.
+the workspace or delete all not-yet-[persisted] experiments at once.
+
+[persisted]: /doc/user-guide/experiment-management/persisting-experiments
 
 ## Removing specific experiments
 

content/docs/user-guide/experiment-management/comparing-experiments.md

@@ -384,8 +384,8 @@ params.yaml  train.epochs      10    10         0
 ## Compare an experiment with the workspace
 
 When you want to compare two experiments, either the baseline experiment in a
-commit, branch, tag or an attached experiment with ID, you can supply their
-names to `dvc exp diff`.
+commit, branch, or tag; or an attached experiment by name, you can supply any of
+these references to `dvc exp diff`.
 
 ```
 $ dvc exp diff cnn-128 cnn-64

content/docs/user-guide/experiment-management/experiments-overview.md

@@ -0,0 +1,71 @@
+# DVC Experiments Overview
+
+DVC Experiments are captured automatically by DVC when [run]. Each experiment
+creates and tracks a variation of your data science project based on the changes
+in your <abbr>workspace</abbr>.
+
+Experiments preserve a connection to the latest commit in the current branch
+(Git `HEAD`) as their parent or _baseline_, but do not form part of the regular
+Git tree or workflow (unless you make them [persistent]). This prevents
+polluting Git namespaces and bloating the repo unnecessarily.
+
+[run]: /doc/user-guide/experiment-management/running-experiments
+
+<details>
+
+### ⚙️ How does DVC track experiments?
+
+Experiments are custom [Git references](/blog/experiment-refs) (found in
+`.git/refs/exps`) with one or more commits based on `HEAD`. These commits are
+hidden and not checked out by DVC. Note that these are not pushed to Git remotes
+by default either (see `dvc exp push`).
+
+Note that DVC Experiments require a unique name to identify them. DVC will
+usually auto-generate one by default, such as `exp-bfe64` (based on the
+experiment's hash). A custom name can be set instead, using the `--name`/`-n`
+option of `dvc exp run`. These names can be used to reference experiments in
+other `dvc exp` subcommands.
+
+</details>
+
+## Basic workflow
+
+`dvc exp` commands let you automatically track a variation of a project version
+(the baseline). You can create independent groups of experiments this way, as
+well as review, compare, and restore them later. The basic workflow goes like
+this:
+
+- Modify hyperparameters or other dependencies (input data, source code,
+  commands to execute, etc.). Leave these changes un-committed in Git.
+- [Run experiments][run] with `dvc exp run` (instead of `repro`). The results
+  are reflected in your <abbr>workspace</abbr>, and tracked automatically.
+- Review and [compare] experiments with `dvc exp show` or `dvc exp diff`, using
+  [metrics](/doc/command-reference/metrics) to identify the best one(s). Repeat
+  🔄
+- Make certain experiments [persistent] by committing their results to Git. This
+  lets you repeat the process from that point.
+
+[pipeline]: /doc/user-guide/project-structure/pipelines-files
+[compare]: /doc/user-guide/experiment-management/comparing-experiments
+[persistent]: /doc/user-guide/experiment-management/persisting-experiments
+
+## Initialize DVC Experiments on any project
+
+DVC Experiments build on basic semantics of <abbr>DVC projects</abbr>. This
+means that minimal formalities are required.
+
+`dvc exp init` lets you quickly onboard an existing data science project to use
+DVC Experiments, without having to worry about bootstrapping DVC manually. You
+can either supply a `command` to execute your experiments or use the
+`--interactive` flag (`-i`) to be prompted for that and other optional
+customizations.
+
+This creates a simple `dvc.yaml` file for you. It uses sane default locations
+for your project's <abbr>dependencies</abbr> (data, parameters, source code) and
+<abbr>outputs</abbr> (ML models or other artifacts, <abbr>metrics</abbr>, etc.)
+-- which you can customize via `-i` or other options of `dvc exp init`.
+
+You can review the results (and commit them to Git) to begin using DVC
+Experiments. Now you can move on to [running your experiments][run] (next).
+
+[codify a pipeline]: /doc/user-guide/project-structure/pipelines-files