exp init: unclear need for --explicit flag

[jorgeorpinel]

Bug Report

UPDATE: Jump to #7143 (comment)

Description

Tool crashes.

Reproduce

$ dvc exp init --explicit "echo hi"
ERROR: unexpected error - 'live'

Expected

Environment information

$ dvc doctor
DVC version: 2.8.1 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-5.10.60.1-microsoft-standard-WSL2-x86_64-with-Ubuntu-18.04-bionic
Supports:
        gdrive (pydrive2 = 1.4.7),
        hdfs (fsspec = 2021.10.0, pyarrow = 0.15.1),
        webhdfs (fsspec = 2021.10.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git

Additional Information (if any):

It's unclear how to use --explicit in general. Does it require --interactive?

[skshetry]

@jorgeorpinel, could you please try upgrading to the latest version?

[jorgeorpinel]

I did. This is the latest version from pip.

[jorgeorpinel]

It's unclear how to use --explicit in general. Does it require --interactive?

Can we use this opportunity to clarify this? I couldn't find any further info in the wiki or internal docs. I'd like to explain it properly in iterative/dvc.org#3071. Thanks

[skshetry]

@jorgeorpinel, latest version is 2.9.2, you have posted version info about 2.8.1.

[jorgeorpinel]

@skshetry again, 2.8.1 is the latest version from pip. Is that normal?

[jorgeorpinel]

I installed the dev version with python3.7 and venv and I see it's not a problem there. In fact the whole -i flow is quite different.

TBH a more helpful answer here would have been "this has been fixed in an upcoming release," instead of arguing about versions. And again, it's not easy to upgrade the version ATM. I'm using a dev env.

And can you please clarify about --explicit or point me to the internal docs @skshetry ? That part of the question hasn't been answered (changing issue title). Thanks

[jorgeorpinel]

p.s. looks like dvc exp init --explicit <command> creates a super simple stage without assuming any paths. Is it basically a wrapper of dvc stage add -n train <command>? I'm not getting the utility here but I'm sure there's a reason for this option. Can someone summarize its use case? Cc @dberenbaum thanks

[skshetry]

exp init was released with 2.9.0 about a week ago. The latest version is 2.9.2.

Regarding --explicit, you are right that it is a wrapper around dvc stage add ..., and will ignore default and/or config values and only takes and build stage from whatever is passed on. In interactive mode, --explicit will not display default values and will make every prompt required (they can still skip with n as input).

[dberenbaum]

I'm not getting the utility here but I'm sure there's a reason for this option.

While dvc exp init is intended to stay much simpler and less flexible than dvc stage add, it is still a bit too inflexible if all paths are required (data, code, params, metrics, models, plots). The typical example is that experiments without plots are pretty common, yet a plots output is required by default. This was the best idea we have for how to exclude paths and only use those which are explicitly provided.

Some background discussion on this is in #6637. --no-defaults and --ignore-missing were alternative names considered.

[jorgeorpinel]

In interactive mode, --explicit will not display default values
best idea we have for how to exclude paths and only use those which are explicitly provided

OK, I see that it changes the behavior of -i. But I'm still unsure of what's the use case since you can always be explicit during regular --interactive mode.

it is still a bit too inflexible if all paths are required (data, code, params, metrics, models, plots).

@dberenbaum you don't need --explicit to omit paths, you just type n in regular interactive mode (which you have to do with --explicit anyway). It does force you to think about each answer but it seems that's not the intended utility.

Anyway, this is minor probably. But so far I don't think the flag is needed TBH. I'll read #6637 and update the docs as best I can for now. Feel free to close this or follow-up at some point if you prefer.

Thanks

[dberenbaum]

OK, I see that it changes the behavior of -i. But I'm still unsure of what's the use case since you can always be explicit during regular --interactive mode.

True, we could ignore --explicit in interactive mode since it's unlikely to be needed in that mode. Seems minor either way IMO.

@dberenbaum you don't need --explicit to omit paths, you just type n in regular interactive mode (which you have to do with --explicit anyway). It does force you to think about each answer but it seems that's not the intended utility.

The intent is to omit paths in non-interactive mode.

See also #7130.

[jorgeorpinel]

The intent is to omit paths in non-interactive mode.

In non-i mode I'd just use stage add.

[skshetry]

@jorgeorpinel, stage add may not be easier for users who are new to dvc. dvc stage add requires understanding lots of things about DVC and stage concepts, whereas with exp init, it is --code/--data/--models which is more easier.

[dberenbaum]

I think we are not likely to prioritize this one. Closing.