Allow users to pass a custom output dir on the command-line

[alexarchambault]

First cut at #3144, for early feedback just in case.

Seems the out directory is used for two purposes:

• output of tasks
• server process files (lock files to ensure only one server starts, server output, …)

A simple CLI option can be used for the former, which this PR adds (-o / --output).

For the latter, as it's used early when Mill starts (mainly in MillClientMain), CLI options aren't parsed yet.

We could parse the first options a bit more from MillClientMain to read -o / --output. For now, as a workaround, I changed the directory used for the latter case, to .mill, but I don't know if that's a direction you're ready to go to right now. I think it makes sense if users are to use a network directory via -o / --output: server-related files are specific to the current machine, and are not meant to be shared, it seems.

I'll add tests if the current choices look fine.

[alexarchambault]

Fixes #3144

[lihaoyi]

How about we use an env var rather than a flag? That way we dont get to parse it and can keep the server files in the same place (relatively) as they are today. I'd rather not split them into a separate folder if it can be avoided

[lihaoyi]

env var isnt perfect either from a UX perspective, but given this is a bit of a niche case I think its a better solution than reorganizing the file layout for this feature due to implementation concerns.

lets go with an envvar for now and at some point we may have a graal native or scalanative client and can have it be smart enough to use flags

[lefou]

As for the cli args, if at all, I'd suggest to just go with --output. It's a bit too early to settle on a short option. Short options are scarce resources. Otherwise, I think using a env var is ok for the start, too.

Here are some more motivations for alternative out dirs:

• read-only project dirs
• project file systems unsuitable to keep run-state of a process
• out dirs in RAM filesystems

As a consequence, I think we should be able to split the two uses of out, Mill target cache vs Mill worker state. I'd really like to be able to keep all Mill worker state (of arbitrary projects) in one single location. e.g. something in $XDG_RUNTIME_DIR. Most code should be already prepared for this, since we made it independent of os.pwd. This would also avoid the issue, that mill currently creates a ./out dir in every dir it is started from.

[lihaoyi]

Let's go with the env var for now and keep the out/ folder intact, at least as part of this ticket. I think that "all generated files in the out/ folder" is a very nice property to have, even though I had different ideas initially (e.g. our integration test fixtures used to separate them build-generated-files and misc-housekeeping-files), and even though today it's not strictly true (we dump some stuff in ~/.cache as well).

I do agree that generating an ./out folder in every dir that Mill starts in isn't great, and we may still want to break it up later, but we don't need to start that right now. e.g. I could imagine that with graal/scala-native allowing a thicker client, and some kind of overhaul of the evaluator command system, the problem may solve itself in a nicer way. But that will take more time to research and experiment