Using jj with a flake.nix is a bad experience

[cormacrelf]

Nix flakes are used for auto-installing things. Often it is used with direnv. JJ itself uses this setup.

Nix flakes also require you to stage all the .nix files in git for them to be found. I think this is a problem with Nix itself -- what the hell sort of requirement is that? Ridiculous. But nevertheless JJ wrecks Nix setups when you have active conflicts.

1. Colocated repo
2. Flake.nix must be in the git staging area to be seen
3. If you have a conflict, even if flake.nix has no conflicts, JJ puts you in a state where flake.nix is an untracked file. Along with the rest of your stuff.
4. Press enter in your shell, direnv tries to reload because some files it cares about have changed.
5. Direnv fails. All your softwares are gone!
6. Especially bad if you installed jj using your flake.nix (!)

It would be nice if this were better. Anyone got a workaround?

[ilyagr]

This seems painful, sorry. I'll ping @necauqua @thoughtpolice in case they have some advice.

I'm guessing that the following is the problem's cause. In colocated repos, jj puts Git's HEAD onto the parent of the working copy commit. I'm guessing that Nix uses Git to extract flake.nix from the HEAD as opposed to using the one in the working copy.

If the parent of the working copy is a conflicted commit, then it's stored in Git in a jj-specific way that non-jj-aware tools will not be able to understand. (It basically stores several versions of the repo in one commit, with a top-level dir for each version). So, as Nix tries to read the HEAD commit's contents with plain Git, it fails to find flake.nix in this case.

I'm not sure whether flake.nix is the only problem; it wouldn't be able to find any other files in their expected locations either.

Boom, bad things happen.

(Hopefully, I've just rephrased what you said into what I think is slightly more correct language. I don't think the words "staged" and "tracked/untracked" are correct here. jj never uses Git's staging area, it's always the same as HEAD)

---

I'm not sure what we can do to help.

The ideal outcome (apart from Nix flakes being redesigned to have an option to not expect or require a Git repo) would be if Nix treated jj as a first-class VCS and use jj's primitives to extract the flake.nix from the parent of the working copy (which I think would be acceptable to Nix Flake's notion of "immutability").

If you use a non-colocated repo, these specific problems will also not occur, as Nix will think the flake is not under version control. I think this cause Nix flakes to refuse to work in many cases. Specifying --impure everywhere might help, but I'm not an expert.

---

In principle, jj could store conflicts in a way that tries to minimize this problem, but it would be really messy. We'd have to introduce functionality that jj itself and most of its users never use, and that's likely to introduce weird behaviors, complexity, and bugs eventually.

[cormacrelf]

Well I do it because I'm new to JJ and only have about ten minutes in the day to read the manual for a new tool, sometimes you gotta do what you gotta do. I definitely appreciate that you can switch back to git tools while learning.

At the same time, pedagogically, writing @ out to the index all the time would make it much easier to learn the JJ way and stop relying on habits. So there's that.

You could also have the @-writing behaviour configurable so there's at least a way back to "just let me use git, I only have 4 minutes to put this change in".

[cormacrelf]

(To be clear, I thought about it, and I think you should do it!)

[PhilipMetzger]

Nix is a good candidate for the mode I call Second Hell of Git but that is partially their own fault. This is something we discussed multiple times already in Discord, were @necauqua and I disagreed heavily. I don't think making @ the Git staging area is a good solution, see the moving branches for a previous discussion.

[ilyagr]

In practical terms, making @ the Git staging area would break VS Code's Git diff view, which would be annoying. I don't think it can show git diff --staged. I think many tools expect git diff to show the latest changes, and I also like that jj's behavior of keeping the staging area unchanged (same as HEAD) is simple and predictable.

I was tempted for a moment to consider doing something like this for conflicted commits: store the view of the repo with conflict markers in the staging area. This would help a few things, for example that same VS Code's Git diff view. However, apart from the general messiness, this has the (fatal, I think) problem that one can really mess up the repo by running git commit (as opposed to jj git commit) if the view with the conflict markers was in the staging area.

[emilazy]

I just tested with VS Code and I see the same stuff after clicking the sidebar Source Control thing after staging all changes in a native Git repository; the files are listed under “Staged Changes” and clicking on one shows a tab named filename (Index). So it seems like it might work fine? Is there some other VS Code thing you had issues with?

The more I think about it the more I like the idea of all @ changes going in the staging area, in terms of mapping Jujutsu concepts to Git tooling. The fact that it’d make Nix work better than with actual Git repositories is admittedly a bonus for me, but it seems like it would be sensible in general as long as there aren’t weird issues like you mentioned with VS Code that would get in the way.

[emilazy]

5. Direnv fails. All your softwares are gone!

By the way, nix-direnv at least falls back to the previous environment on failure these days. Although that has its own pitfalls.

[necauqua]

3. If you have a conflict

Um, do you mean that you are in a conflicted state, and instead of the conflict being materialized, the sides are put into those top-level folders?.
This is a separate issue then, happens to me in vscode

I can't reproduce your entire situation (checked out commit is conflicted -> flake.nix is untracked) in my head atm

In my head, you check out a conflicted commit - the conflicts get materialized, and if they're not in flake.nix itself - it looks like an unchanged file that's committed and everything works, how does flake.nix (or the rest of your stuff) become untracked..

[cormacrelf]

Um, do you mean that you are in a conflicted state, and instead of the conflict being materialized, the sides are put into those top-level folders?. This is a separate issue then, happens to me in vscode

Yes, exactly that.

I can't reproduce your entire situation (checked out commit is conflicted -> flake.nix is untracked) in my head atm

This is weird -- I can't reproduce it in a toy example. I thought the top-level folders was always how it worked but it seems that's not normal. Not sure what it is about the medium-sized repo I hit this issue with.

; cd /tmp
; jj git init --colocate repo
; cd repo
; echo '...' > flake.nix
; jj branch c flake
; jj new
; echo hi > a.txt
; jj branch c hi
; jj new flake
; echo hello > a.txt
; jj branch c hello
; jj new hi hello

; jj log
@    mlolxsmm ... 2024-06-26 20:08:10 7e6f3d4d conflict
├─╮  (empty) (no description set)
│ ◉  qxxxzwor ... 2024-06-26 20:07:27 hello 8a14a55d
│ │  wrote hello
◉ │  zsvntvox ... 2024-06-26 20:07:48 hi HEAD@git 7a761777
├─╯  wrote hi
◉  nsxqztvp ... 2024-06-26 20:06:53 c9a49043
│  add flake.nix
◉  zzzzzzzz root() 00000000
# /tmp/repo (7a76177) [!]
; jj status
The working copy is clean
There are unresolved conflicts at these paths:
a.txt    2-sided conflict
Working copy : mlolxsmm 7e6f3d4d (conflict) (empty) (no description set)
Parent commit: zsvntvox 7a761777 hi | wrote hi
Parent commit: qxxxzwor 8a14a55d hello | wrote hello
# /tmp/repo (7a76177) [!]
; git status -sb
## HEAD (no branch)
 M a.txt
# /tmp/repo (7a76177) [!]
; ls
a.txt  flake.nix
# /tmp/repo (7a76177) [!]
; git ls-tree HEAD
100644 blob 45b983be36b73c0788dc9cbcb76cbb80fc7bb057    a.txt
100644 blob eb1ae458f8ee6e88c072a9989724d7685c56beb1    flake.nix

[cormacrelf]

See below, I did manage to reproduce it.

[necauqua]

Well, in your example, it looks like the conflict got materialized - no changes to flake.nix, and a.txt has conflict markers - everything is as it's supposed to be

And in that state, Nix does not pick up the flake? If so, then that's strange and definitely something to investigate ig

If you check out a conflicted commit with git (or some vscode background process like it happens to me) the conflict doesn't get materialized and you see the conflicting trees themself as top-level dirs - you probably know that already, sorry if reiterating - and then ofc Nix wont find the flake

[necauqua]

Ah wait I can't read lol

your example here is where it works like it should, yeah, and in your reply to Martin below is how you get the toplevel trees through jj only, ok ok

[martinvonz]

This happens when you use git to check out a commit with conflicts. Use jj instead.

[cormacrelf]

I got there using jj new only. Doesn't happen if you jj new first second to create the merge commit, but it does if you do that first, switch away, and then jj new x with x = merge commit.

If I instead create a fresh merge commit, jj leaves the HEAD@git at the first-listed branch name, and things work as you described.

# two branches hi & hello, with conflicts

# /tmp/repo (65835d0) [✘?]
; jj new hi hello
Working copy now at: ywrknwmr fc283379 (conflict) (empty) (no description set)
Parent commit      : zsvntvox 7a761777 hi | wrote hi
Parent commit      : qxxxzwor 8a14a55d hello | wrote hello
There are unresolved conflicts at these paths:
a.txt    2-sided conflict

# /tmp/repo (7a76177) [!]
; git status -sb
## HEAD (no branch)
 M a.txt

# /tmp/repo (7a76177) [!]
; ls
a.txt  flake.nix

But checking out an existing one with jj new has the issue:

# /tmp/repo (7a76177)
; jj --version
jj 0.18.0-9fb5307b7886e390c02817af7c31b403f0279144

# /tmp/repo (7a76177)
; jj log
@  pvsluplv ... 2024-06-26 22:39:25 d0430620
│  (empty) (no description set)
│ ◉  kszpxwyw ... 2024-06-26 22:31:10 9b2d08ac
│ │  (no description set)
│ ◉  xmsqrpvl ... 2024-06-26 22:26:17 mm 65835d08 conflict
╭─┤  (empty) (no description set)
│ ◉  qxxxzwor ... 2024-06-26 20:07:27 hello 8a14a55d
│ │  wrote hello
◉ │  zsvntvox ... 2024-06-26 20:07:48 hi HEAD@git 7a761777
├─╯  wrote hi
◉  nsxqztvp ... 2024-06-26 20:06:53 c9a49043
│  add flake.nix
◉  zzzzzzzz root() 00000000

# /tmp/repo (7a76177)
; jj st
The working copy is clean
Working copy : pvsluplv d0430620 (empty) (no description set)
Parent commit: zsvntvox 7a761777 hi | wrote hi

# /tmp/repo (7a76177)
; git status -sb
## HEAD (no branch)

# /tmp/repo (7a76177)
; jj new xmsqrpvl # conflicted commit
Working copy now at: xopokxsk d4821854 (conflict) (empty) (no description set)
Parent commit      : xmsqrpvl 65835d08 mm | (conflict) (empty) (no description set)
Added 0 files, modified 1 files, removed 0 files
There are unresolved conflicts at these paths:
a.txt    2-sided conflict

# /tmp/repo (65835d0) [✘?]
; git status -sb
## HEAD (no branch)
 D .jjconflict-base-0/flake.nix
 D .jjconflict-side-0/a.txt
 D .jjconflict-side-0/flake.nix
 D .jjconflict-side-1/a.txt
 D .jjconflict-side-1/flake.nix
 D README
?? a.txt
?? flake.nix

# /tmp/repo (65835d0) [✘?]
; ls
a.txt  flake.nix

[necauqua]

Yeah, a bit of an XY problem with the Nix tangent, but that's just a bug in jj it seems.
You're never supposed to see the .jjconflict trees when using the jj commands only, conflicts should materialize into markers, although git colocation complicates matters ofc (with the whole "where to put HEAD if @ has multiple parents etc")

[ilyagr]

My sense is that the problem occurs when the parent commit is conflicted. Even if you use jj new to get to it, Nix uses some version of git ls-files to find the "staged" version (which really means the HEAD version in this case) of flake.nix and the files in the flake. Nix flakes have the crazy idea of enforcing that files should be tracked by the VCS in this way AFAIU.

So, Nix sees the internal way jj stores conflicts, cannot find flake.nix, and acts as though flake.nix isn't stored in the VCS. Emily explained some details of the crazy Nix behavior in #3970 (reply in thread).

[martinvonz]

Maybe we could write the .jjconflict-* directories when in a colocate repo so git doesn't think that they've been deleted. That can be a bit expensive if the repo (working copy) is large, however. I'm also not sure how to do it cleanly without spreading knowledge about colocated repos to the working-copy code (which doesn't currently care about it).

[amiryal]

Sorry I don’t have time for a thorough response to all the sub-threads. As a user of Nix myself, I would love to see this resolved. My 2¢: Maybe the internal representation of conflicted commits in the Git backend of jj should change, to make it more friendly towards the colocated use case? I can think of a few ideas how to do that, but maybe a proper issue/PR would be a better place for them.

[martinvonz]

It used to be more friendly to colocated repos until commit 006c764. I don't think there's much harm in changing it to better match the files in the working copy, as long as we also include the .jj-conflict-* subdirectories (they're needed in order to prevent GC).

[amiryal]

Thanks! I opened #3979 to track this proposal.