Auto-add of untracked files screws me up every time

[durin42]

Description

Initially I thought the auto-add of files was a neat idea, but in practice I just leave untracked files in my repo all the time, and tools like patch(1) assume they can drop .orig or similar files in the WC without it being a problem. I think every time I've used jj I've ended up getting grumpy at auto-adds and having to rip something out of a commit, sometimes after doing a push (when it was effectively emulating hg import for example).

Steps to Reproduce the Problem

1. patch -p1 < some.diff (or similar)
2. jj describe && jj close && jj git push
3. Look at web view of diff, notice you committed a .orig file again (or similar)

Expected Behavior

I (still) don't expect auto-adds, and it really surprises me every time, plus it's super frustrating to have to ignore every file spec I might create as a temporary scratch file or what have you.

Actual Behavior

As above.

Specifications

• Platform: macOS
• Version: 6c83eb6

[arxanas]

Agreed on that — I think automatically committing tracked files is fine, but untracked is probably bad:

• They could be very big
• They could contain secrets
• Querying the working copy for only tracked files is probably more efficient in practice (like with git status -uno)

[martinvonz]

I mostly find the feature useful, but I also agree that it can be annoying and confusing. The worst case I've noticed is when you - perhaps accidentally - check out the root commit, where there's no .gitignore file containing target/. Then any command you run will try to commit GBs of data. I ran into that again just a few days ago, and was actually thinking of adding a config for the auto-add behavior.

When looking at an old version of the repo, you'll not see untracked files (e.g. jj --at-op=<some old operation> status), but that seems fine. I think the most annoying bit will be to add a way of providing that information to users who want to commit the working copy in the background (like we will probably do at Google). I'll probably skip that bit to start with when I implement this.

[tp-woven]

Just to throw my 2c in: With the exception of new files that are also added to .gitignore, I actually really like the automatic tracking. I also think the same "repro" from the first comment can be used as a reason to auto-track - if you don't jj st before pushing, you're just as likely to miss a file that you should have added as you are a file that you should have ignored. So my preference is that this would be configurable if possible, rather than just removed.

[elasticdog]

Just wanted to say that I really like the auto-tracking feature as well. After using jj for a while, going back to manually having to think about adding files to be tracked seems like a lot of extra work and possibly more error prone. I'm already used to checking jj status to make sure that I'm committing the expected files...that said, I can definitely see how not thinking about the appropriate ignores right away could be troublesome if the files are large, and also acknowledge the the root commit situation (although I don't know how often that would realistically come up in day-to-day usage).

[necauqua]

The thing with accidentally committing GBs of target when you forgot to ignore it or checkout some old commit is that there's no GC at the moment, so those GBs are forever in the history.

Especially if you have something automatically creating it, like direnv, the moment you check out the commit before your update of gitignore, literally happened to me with jj repo more than once.

The only way is to recreate the repo (losing the oplog) this time not forgetting to ignore stuff/not editing old commits - which is kind of meh as well.

Full GC of course means basically the same thing, only automated by a single command, but something like jj undo --harder [--at-op OP] [--and-immediate-gc-too] (are you sure [y/N]) to delete the last op/OP unrecoverably and gc things it referenced could be cool.

[ony]

Maybe it is possible to track ignores in jj too. If you switch away from commit at which you had something ignored but not at new one - that information probably can be used. E.g. sort of in-flight ignores that visible in jj st to inform about either extending explicit ignores or confirming addition of new paths to current change.

P.S. Can think of how git checkout reacts when your untracked file is about to be overwritten by checked out files. In case of jj all files are tracked by default and thus even absence of the file is tracked.

[necauqua]

Hm, so auto-tracking when you are doing changes in a working commit is the main killer feature.

But for my issue with it, how about this:
What if jj considered .gitignore not just from the current commit, but from the entire history - or actually just descendant of the current commit?
And if you needed to actually add some file in the past you'd have to explicitly track it (instead of explicitly untracking it in the common case you don't need that).

I know this is kind of magical, but the more I think about it the more it makes sense, idk - and the autorebase is magical on it's own, this somehow is kind of even consistent in my head

[martinvonz]

It would be expensive to find the gitignores from all commits, but we could probably index that information. I'm more concerned that it would be unexpected behavior. For example, if you check out a sibling commit where target/ is not ignored, then it would still not be ignored if we only consider descendants.

Maybe it's better to check if the gitignores changed between the old and the new commit and if any untracked files according to the new patterns match the old patterns. If that happen, we could just print a warning about it. We could additionally add the ignores to a per-workspace set of ignores (which we don't support yet). (EDIT: I think this is what @ony suggested.)

[necauqua]

If that happen, we could just print a warning about it

The warning being "those differences are implicitly untracked for this WC, in case your direnv caused GBs of files to generate in target and/or .direnv - add them to gitignore here or explicitly track them, moving to another commit without them ignored (e.g. jj new) will cause them to be tracked in that commit"

^ this is a loose idea, could be refined, for example jj st will have that information ofc

[ilyagr]

Following @ony's suggestion, perhaps when the working copy moves to a new commit, we could track "newly unignored" files by comparing the old .gitignore and the new .gitignore. For example, we could store an extra tree of "newly unignored" files.

Then, the UI could provide ways for dealing with these files. E.g. there could be a command like jj ignored --previously that lists them and jj ignored --previously --restore that gets rid of them. We'd also have to decide whether a modification to a "previously ignored" file makes it no longer "previously ignored".

[kevincliao]

I ran into this today when trying to checkout to a different branch that doesn't contain node_modules. As people have mentioned above at that point it's not possible to run any jj commands. I ended up creating a temporary .gitignore file before being able to jj op restore to a previous checkpoint. Is there a better way to recover when running into this? I wonder if it's possible to have jj op commands still work in this scenario.

[kevincliao]

Ahh ignore my comment, I think I got confused - once there is an option to not auto-add untrack files jj op commands will work again.

[yuja]

Is there a better way to recover when running into this? I wonder if it's possible to have jj op commands still work in this scenario.

You can pass --ignore-working-copy to these commands, but we don't have the last bit to reset the working copy without snapshotting yet.

jj op log --ignore-working-copy  # "jj op log" also works with the current main branch
jj op restore --ignore-working-copy @-
jj workspace update-stale --some-option-to-not-snapshot-before-resetting

[HadrienG2]

Overall, it seems there is no good automated way to handle untracked files when creating commits. One one hand, not tracking them leads to incomplete commits. On the other hand, auto-tracking them leads to commiting of unwanted files. So how would you feel about some variation of the following semi-automated design?

1. By default, interactively prompt before auto-adding files, something like This command will add <list of files> to the current commit, proceed? [y/N].
   • Saying yes follows the current behavior.
   • Saying no aborts the command with an error return value and lets you use jj track and .gitignore as appropriate.
2. Have a way to whitelist sets of files (e.g. source files) so that they are auto-added without a prompt, and not mentioned in prompts when they do occur.
   • This could take the form of a .jjadd file that uses the same glob syntax as .gitignore.
   • The aforementioned prompt would mention the possibility of configuring jj for auto-adding and ignoring.

I think this might strike a good balance between the following concerns:

• Files which we want to track (like source files) eventually get auto-added silently as desired, avoiding incomplete commits and replicating the good parts of the current jj auto-add UX.
• Files which we do not want to track (like object files, target/ directories...) eventually get ignored silently as desired, without undesirable creation of commits that will keep them in the history forever.
• After a short initial configuration period, seeing the prompt becomes an exceptional event and thus leads the user to pause and think, as desired in this situation.

For scripted operation, there should be a way to provide a default answer to the prompt via CLI arguments.

[martinvonz]

I'm personally quite happy with the current behavior (except for the behavior when updating to a commit with different .gitignore). It can be a bit annoying in the beginning, but once you've added the appropriate paths, I find that it works pretty well. Maybe others feel differently. But even if they don't, we may want to make it less annoying for new users by doing something like you suggest.

[ilyagr]

I'm not very happy with the idea of the interactive prompt.

I think that if you edit a .gitignore, any subsequent jj command could trigger this prompt, including jj log. I tend to run an analogue of watch jj log in a tmux pane permanently, and I think this would work very badly with the interactive prompt. Firstly, I'll need to adjust the command to use the "scripted mode". In the "scripted" mode, if the default answer to the prompt is "yes", this goes back to users experiencing auto-add of untracked files. If it's "no", jj's view of the workspace could be out of sync with reality for a while (but, if we go with a prompt, I think this is the better option).

Other UIs will also do an analogue of jj log regularly. Every jj UI (e.g. VS Code plugin) would probably need to have a way of giving this prompt to the user, if we made this interactive.

[HadrienG2]

Ah, yes, there's that. I knew that this design decision of having status commands modify the repository was fishy and going to cause problems someday...

[lf-]

This feature has sadly made me bounce off of jj immediately every time I try it, which is really unfortunate because I keep hearing such good things about it, and want to give it a genuine try.

Every single repository I work on regularly has various testing/strace-log/whatever files in its root. I actually don't mind auto tracking in src/, but in the root it just is not compatible with my workflow.

fwiw a workaround for this that I've not yet checked works on jj might be some kind of terrible thing like so in the .git/info/exclude:

/*
!src/
!Cargo.*

This workaround is quite bad indeed, and I would rather not have to reimplement the git index in .git/info/exclude to be able to use jj, though admittedly it would be with wildcards at least.

[ilyagr]

I have no idea whether this would be helpful you, but here's something that helped me a lot. I can't remember who had suggested it originally; it might be in the FAQ.

• I added _ignore/* to ~/.config/git/ignore (~/.gitignore should also work).

  This is possibly not absolutely optimal (I have been wondering whether /_ignore/ would be better), but works well enough. I actually use _ilyaignore to make the name more unique.

• Create an _ignore subdir in my repo

• Save all weird logs and traces to it

[lf-]

Yup it is in the FAQ or something; I've seen it given as advice before. I just don't like it and it doesn't vibe with how I work, since it would be a whole bunch of extra typing. I could have it be i/ or something, I guess, to reduce typing, but I would still have to remember to do it every time, which feels kind of bad?

[ilyagr]

Inspired by more feedback (#3528 (comment)), perhaps @dpc 's suggestion from that post might work. Perhaps we could have a notion of "untracked" files, like Git, and default files to "untracked", while also auto-updating all the tracked files on each command?

jj status would certainly show any untracked files. jj log could too. It's not quite in the spirit of "everything is a commit" (untracked files would show up as a fake commit in some places), but might work.

One question is what jj diff would do. My first instinct would be to have it act on tracked files only, but complain loudly when there are untracked files. Perhaps each command would do that, I'm unsure.

This would be a huge change, so I almost certainly missed some important considerations.

[dpc]

This would be a huge change

Speaking out of ignorance, I'm guessing jj already needs to compare all worktree files against .gitignore. Right after that it could just compare them against files already tracked in the current change and ignore ones that are not. Plus a command to track a file. And that's kind of it, no? Showing untracked files, etc. seems like a nice-to-have. Deleting a tracked file could work as a "untrack", just like it already does.

Hmm... I guess mv <sometrackedfile> <newlocation> now requires explicit calling "track" on a new location, which is a bit breaking the "immersion", but I think it's fine. And again - this behavior would be optional (but I would suggest making it the default for the sake of newcomers). People that figured out everything could just opt-in into current seamless behavior, which I find elegant and I'm sure I would eventually settle into it just fine, after making sure given repo doesn't produce untracked trash, adding some ./tmp/ to .gitignore and remembering to create my debugging stuff inside it.

[martinvonz]

jj status would certainly show any untracked files. jj log could too. It's not quite in the spirit of "everything is a commit" (untracked files would show up as a fake commit in some places), but might work.

One question is what jj diff would do. My first instinct would be to have it act on tracked files only, but complain loudly when there are untracked files. Perhaps each command would do that, I'm unsure.

If we add support for untracked files, I think it should be pretty much only jj status that shows them. They would just be invisible to every other command. Would that work for the untracked-files proponents?

I guess mv <sometrackedfile> <newlocation> now requires explicit calling "track" on a new location

I don't think so. Almost all commands, and probably also the future jj mv work on commits and just update the working copy to match afterwards.

[joyously]

OK, it's not overwritten, but it's tracked and I didn't track it. If I undo after that warning, is the file deleted?

[martinvonz]

Yes. This is the case I was trying to describe earlier

[marc-h38]

Again, this does not always work; not for every project or usage. Scroll up for examples.

I applied my own advice to myself, went through the whole issue again and I admit the examples given so far were pretty vague and possibly difficult to picture for tidy people, especially if you think littering the working copy is a bad habit not worth discussing.
So I crafted a list of actual filenames, all inspired from real world experience. With hindsight I think someone in the club of "dirty" people should have done this much earlier: it could have saved some time, comments and misunderstandings.

Of course these filenames are much shorter and much more cryptic in "real life" but their purpose and name randomness are the same.

less_optimized_build_is_it_slower/
debug-build-g3dbf325/
weekly_stress_test.logs/
stress_test_20240891T1200.logs/
Robert_3rd_workaround_attempt.patch
openssl_test_key_for_X
short_todo_list_for_bug2345.txt
temporary_script_for_git_bisect_run_for_bug2345.py
hack_test_sequence_that_might_reproduce_the_failure_faster_on_suspect_commits.bash
crashdumps_from_bug2345/
firmware_images_from_customer_X/
some_project_generates_concurrent_test_logs_with_PID_in_their_name/
some_source_file.backup_file_extension_from_very_rarely_used_tool

etc.

"Auto-stashing" of such temporary files is not a feature: it's a bug because they're useful ACROSS commits and branches.

A) I admire organized people who are in a similar situation but who 1) can categorize all the types of temporary files they will use 2) decide in advance some naming convention for each type (e.g. "build*", "*logs") and add it to some gitignore file 3) carefully follow their own naming convention. I'm not that organized, rigorous and disciplined. I don't think I'm alone and TBH I'm afraid I don't want to be that organized.

B) As already mentioned by others, I really don't want to move these temporary files out of the working copy and into some ../temp_for_projectX/ or /tmp directory because it's further away, less convenient and because they're 100% related to projectX and the activity happening inside that particular workspace. Such temporary files belong to the working copy - but not to any commit or stash.

C) This last and less important preference may be more personal and less common: I actually don't want "temporary files with random filenames" to be gitignored and hidden from "git status". Because "git status" is the perfect tool to list those files, to remind me their bad filename that I just forgot I wrote, to remind me that I forgot to do something with them and to show the scraps on my kitchen counter that I need to clean up when I'm done. Very different from the regular, predictable build/ directory which is better gitignored and hidden. I don't mind if litter stay hidden in "jj status" if that makes things easier for jj as long as I can still reach to "git status" to show it. That's why i suggested some echo '*' >> .git/info/jjexclude above as a better hack than echo '*' >> .git/info/exclude

Once again: I'm suggesting ZERO change to the current, default behavior. Having something "opt-in" would be awesome enough. I don't think anyone remotely suggested to change any default behavior either so please don't answer any non-existent suggestion to change the default behavior: it just makes this already long thread even longer. For instance I believe that any new fear of forgetting to add new files is void: just keep using jj the tidy way as you always have and unsubscribe from this issue!

Hope these examples help tidy people understand us the dirty ones.

[marc-h38]

... temporary files with random names ...
Because there is a conceptual difference that no tool can automatically figure out; so they must be told.

That's true for jj too, but you currently have to use .gitignore to tell it.

I think pretty much every version control system invented so far besides jj supports has been supporting not just two but three types of files:

1. tracked
2. (git)ignored ( *.o, build/, *.backup, ...)
3. Temporary, dirty files with random names, see examples above. Not (git)ignored.

jj drops type 3. For tidy people this is a non-event because they never used 3. But for dirty people this is a showstopper (for now).

[martinvonz]

@marc-h38: Thanks for explaining with some concrete examples. Hopefully that helped explain it to anyone who was still against the feature. I haven't seen any arguments against the feature (I've only heard some people say that they wouldn't personally use it). If anyone is concerned that it complicates the design or the conceptual model too much, please say so. What do you think, @yuja?

Are you interested in implementing this feature, @marc-h38?

[joyously]

Is this one of those options that makes a command work differently for different users? What is the definition of options that can do that and options that are unwanted because they do that?

Given the explanation of a "dirty" workflow, it seems answered in the FAQ.

For example, you could add scratch/ to ~/.git/ignore and then store arbitrary files in <your-git-repo>/scratch/.

[martinvonz]

Is this one of those options that makes a command work differently for different users? What is the definition of options that can do that and options that are unwanted because they do that?

I don't feel strongly myself. Maybe the main problem with aliases is that you can redefine a command to behave in ways that are not defined by any config option (other than the alias definition itself). For example, if we get a bug report saying that some branch disappeared and the user said they did jj status, it would make no sense until they told us that they had redefined status to be jj fetch && jj status or something.

[martinvonz]

It actually looks pretty easy to add at least a basic version of this feature. I'll probably send a PR for that this weekend or next weekend (trying to focus on work priorities during the week).

[yuja]

I haven't seen any arguments against the feature (I've only heard some people say that they wouldn't personally use it). If anyone is concerned that it complicates the design or the conceptual model too much, please say so. What do you think, @yuja?

I'm not against this feature. I personally wouldn't want to see random junks in git status, so I would try to find some patterns to ignore these, but I understand we need a way to notice "mis"-ignored files.

I think people agreed that jj track (or jj file track) is needed. How would we implement the mechanism to not auto-track?

• binary config knob snapshot.auto-track = false?
• jj-specific version-controlled ignore file .jjignore?
• jj-specific core.excludesFile-like config?
• positive list to auto-track snapshot.auto-track = "glob:'**/*'"?

[martinvonz]

• positive list to auto-track snapshot.auto-track = "glob:'**/*'"?

This is what I'm going for (with a default of all())

[martinvonz]

The draft in #4338 seems to work (though I've only written a single test case so far). I need to add more tests and also make jj status report untracked paths.

[necauqua]

Oh my god I've been busy and missed all of this.

I don't follow this part.

What I've been saying, and I might be wrong as I was working purely on assumptions without actually looking at code, is this:

The snapshotting process at some point looks at the working dir to snapshot the files, right.
With "not autotracking new files" option enabled, it will check with @ to snapshot only the file modifications/removals for files already present in @, with some form of jj track command being there to actually add new files to @.

I'm pretty sure the #4338 does not do what I wanted, unless I'm misunderstanding, as described it seems like it disables tracking entirely for configured paths - which is a feature, of course, just different to what I meant

My point was that we could have a balance between having to track every single thing and the ease of autotracking, by only requiring to manually track adding new files (might've been better described by my little pseudo-algorithm above).

[necauqua]

Looking just a bit more into the #4338 it repeatedly uses wording "track new files that match the pattern", so may be it does exactly what I wanted and I indeed misunderstood 🙃

And I could even have everything in src/ auto-tracked which is even better