exp run: make it work more easily in ci

[dberenbaum]

Right now, exp run fails in CI because every Git forge defaults to cloning with a depth of 1 commit, and exp run tries to find previous commits to do a merge operation as explained in #7547 (comment).

To mitigate, CML introduced cml ci --fetch-depth to make it easier to "unshallow." However, this isn't enabled by default, and unshallowing the entire repo may be expensive and unnecessary.

What can we do to make this better? Some ideas:

1. Test how much depth is usually needed and ask cml ci to default to that depth.
2. Think of ways to avoid needing the extra history during exp run.

[dberenbaum]

@pmrowla I'm still not that clear on why this fails.

For example, I tried to reproduce the problem from the Git CLI and found that what I thought was the same merge operation completed successfully:

# Tested with example-get-started
$ git clone --depth=1 [email protected]:iterative/example-get-started.git
Cloning into 'example-get-started'...
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (42/42), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 42 (delta 4), reused 36 (delta 2), pack-reused 0
Receiving objects: 100% (42/42), 79.47 KiB | 5.30 MiB/s, done.
Resolving deltas: 100% (4/4), done.

$ cd example-get-started

$ dvc pull
A       data/data.xml
A       eval/importance.png
A       model.pkl
A       data/prepared/
A       data/features/
5 files added and 7 files fetched

# exp run fails
$ dvc exp run -v
2023-01-19 10:51:39,357 DEBUG: Computed stage: 'data/data.xml.dvc' md5: '22466731a568fe3bce1ce30913573880'
2023-01-19 10:51:39,359 DEBUG: Preparing to transfer data from 'memory://dvc-staging/f6461f7e5d1f7c9ef6979ca64bfe6dde350adf89d0fcbbd9f4b36a25ca121a71' to '/private/tmp/example-get-started/.dvc/cache'
2023-01-19 10:51:39,359 DEBUG: Preparing to collect status from '/private/tmp/example-get-started/.dvc/cache'
2023-01-19 10:51:39,359 DEBUG: Collecting status from '/private/tmp/example-get-started/.dvc/cache'
2023-01-19 10:51:39,360 DEBUG: Removing '/private/tmp/example-get-started/data/.TeW5ke7ZfjcdGqPZwUKwCb.tmp'
2023-01-19 10:51:39,360 DEBUG: Removing '/private/tmp/example-get-started/data/.TeW5ke7ZfjcdGqPZwUKwCb.tmp'
2023-01-19 10:51:39,361 DEBUG: Removing '/private/tmp/example-get-started/.dvc/cache/.FH5VXc4Dr5qmGhTTp5kosi.tmp'
2023-01-19 10:51:39,361 DEBUG: Removing '/private/tmp/example-get-started/data/data.xml'
2023-01-19 10:51:39,363 DEBUG: Saving information to 'data/data.xml.dvc'.
2023-01-19 10:51:39,408 DEBUG: Stashed experiment 'cdb4dce' with baseline '7e4e86c' for future execution.
2023-01-19 10:51:39,412 DEBUG: Init workspace executor in '/private/tmp/example-get-started'
2023-01-19 10:51:39,423 ERROR: Merge analysis failed: `dvc exp` does not work in shallow Git repos. See <https://dvc.org/doc/user-guide/troubleshooting#git-shallow> for more information.
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/scmrepo/git/backend/pygit2.py", line 704, in merge
    analysis, ff_pref = self.repo.merge_analysis(obj.id)
_pygit2.GitError: object not found - no match for id (a1446b9cdad3bd0302bacc05a57504cfaafbbf20)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/repo/experiments/executor/local.py", line 230, in init_git
    self.scm.merge(merge_rev, squash=True, commit=False)
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 289, in _backend_func
    result = func(*args, **kwargs)
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/scmrepo/git/backend/pygit2.py", line 706, in merge
    raise SCMError("Merge analysis failed") from exc
scmrepo.exceptions.SCMError: Merge analysis failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/cli/__init__.py", line 184, in main
    ret = cmd.do_run()
  File "/Users/dave/Code/dvc/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/Users/dave/Code/dvc/dvc/commands/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/Users/dave/Code/dvc/dvc/repo/experiments/__init__.py", line 472, in run
    return run(self.repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 66, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/run.py", line 59, in run
    return repo.experiments.reproduce_one(
  File "/Users/dave/Code/dvc/dvc/repo/experiments/__init__.py", line 114, in reproduce_one
    results = self._reproduce_queue(exp_queue)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/utils.py", line 74, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/__init__.py", line 354, in _reproduce_queue
    exec_results = queue.reproduce()
  File "/Users/dave/Code/dvc/dvc/repo/experiments/queue/workspace.py", line 90, in reproduce
    entry, executor = self.get()
  File "/Users/dave/Code/dvc/dvc/repo/experiments/queue/workspace.py", line 56, in get
    executor = self.init_executor(self.repo.experiments, entry)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/queue/base.py", line 610, in init_executor
    executor.init_git(
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/funcy/flow.py", line 127, in retry
    return call()
  File "/Users/dave/miniforge3/envs/example-get-started/lib/python3.8/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/experiments/executor/local.py", line 232, in init_git
    raise GitMergeError(str(exc), scm=self.scm)  # noqa: B904
dvc.scm.GitMergeError: Merge analysis failed: `dvc exp` does not work in shallow Git repos. See <https://dvc.org/doc/user-guide/troubleshooting#git-shallow> for more information.
------------------------------------------------------------
2023-01-19 10:51:39,429 DEBUG: Analytics is disabled.

# Try the same operation in Git CLI
$ git checkout refs/exps/exec/EXEC_HEAD
Note: switching to 'refs/exps/exec/EXEC_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 7e4e86c Evaluate bigrams model

$ git merge --squash --no-commit refs/exps/exec/EXEC_MERGE
Updating 7e4e86c..cdb4dce
Fast-forward
Squash commit -- not updating HEAD
 .dvc/tmp/repro.dat | Bin 0 -> 296 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 .dvc/tmp/repro.dat

[casperdcl]

also, exp run could auto-detect it's in CI and do something clever (like call cml ci --fetch-depth)?

[dberenbaum]

I like that idea @casperdcl but don't think it needs to be limited to exp run or requires cml ci (we have scmrepo that can fetch as needed) -- can we call it whenever a ref is not found?