Add flake.lock & workflow to update it #1934

i077 · 2021-04-16T16:36:05Z

Description

This PR implements suggestions from #1913, adding a flake.lock file so we can nix run github:nix-community/home-manager without any hiccups. The nixpkgs input is currently set to track nixos-unstable. If you want to test this out, you can run nix run github:nix-community/home-manager/pull/1934/head, which should give you the help text.

To keep this file up to date, the PR also adds a GitHub workflow that

Runs every Sunday and Wednesday at 03:51 (I just picked a random time)
Runs nix flake update --commit-lock-file, committing as a nonexistent user called flakebot
Pushes this commit ~~directly to master~~ to a new branch called flake-update with a custom token FLAKEBOT_TOKEN (this needs to be added to the repo)
Opens a PR (or updates an existing PR) to merge this commit to master. This is done with the same custom token so that other workflows can be run against the PR.

Also updates the README to change the section on a flake-based config a bit.

I also wanted to modify the testing workflow to actually run home-manager's tests, but was having a bit of trouble exposing the tests under the checks output of the flake, so I will leave that out for now. ~~This means that the update workflow will just regularly push updates to master, whether they pass tests or not.~~

cc @berbiche since we were talking about this in the other PR.

Checklist

Change is backwards compatible.
Code formatted with ./format.
Code tested through nix-shell --pure tests -A run.all.
Test cases updated/added. See example.
Commit messages are formatted like
```
{component}: {description}

{long description}
```
See CONTRIBUTING for more information and recent commit messages for examples.
If this PR adds a new module
- Added myself as module maintainer. See example.
- Added myself and the module files to .github/CODEOWNERS.

.github/workflows/update.yml

teto · 2021-04-18T23:55:11Z

at some point it was discussed to open a PR instead of an automatic merge. I preferred that scenario, is that doable through github actions ? nevermind the checks, they can be added later

i077 · 2021-04-19T00:06:03Z

I can have the workflow checkout -B a new branch and open a new PR, that's what @kclejeune and I do on our nixos configs. My only concern was that I didn't want the repo to just have a bunch of flake update PRs, but if you're okay with that I can implement it.

edit: I should clarify: no more than 1 flake update PR will be open at a time, but it will require manual merging each time it gets opened (at most twice a week).

teto · 2021-04-19T07:14:09Z

edit: I should clarify: no more than 1 flake update PR will be open at a time, but it will require manual merging each time it gets opened (at most twice a week).

That seems the best option, would it close and open a new PR or force-push to the current one ?

i077 · 2021-04-19T13:40:04Z

If the update action runs and there's already an existing PR, it will just force-push that branch. The -B switch in the checkout command basically "force creates" a new branch, even if it already existed in the remote, so the previous update commit just gets overwritten.

Also remember that a new secret containing a github token with push access to the repo needs to be made so we can trigger other workflows on the PR. I'd like to add the checks attribute to the flake with all of home-manager's tests so we can actually run them against the nixpkgs input in the update PR, but like I said I was having trouble with that so it can come later.

I'll update the workflow to push to a new branch later today.

teto · 2021-04-21T22:11:09Z

README.md

-    home-manager.url = "github:nix-community/home-manager";
+    home-manager = {
+      url = "github:nix-community/home-manager";
+      inputs.nixpkgs.follows = "nixpkgs";


this is potentially harmful, advanced users can use it but we should default to the mainstream method.

could you elaborate on how this is potentially harmful?

home-manager is guaranteed to work with the nixpkgs in its lock file, it has not been tested with the overriden version so there is a risk of failure no ?

IMO users typically expect their entire configuration to be evaluated against a single instance of nixpkgs unless they specify otherwise, so I figured overriding the input would be a sensible thing to include here.

I disagree, that's not how flakes are conceived (it doesn't default to a single nixpkgs) and I think users prefer something that work rather than shaving off a few MB on the disk. It's still possbile but shouldn't be the default I think.

@kclejeune, maybe I'm confused. I don't see any way to create a token for an organization. What I can see is a way to create a secret, but then I have to provide a value. Perhaps I don't have sufficient access to the nix-community organization? In the documentation I found https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization but it seems to be limited to GITHUB_TOKEN.

@kclejeune, maybe I'm confused. I don't see any way to create a token for an organization. What I can see is a way to create a secret, but then I have to provide a value. Perhaps I don't have sufficient access to the nix-community organization? In the documentation I found https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization but it seems to be limited to GITHUB_TOKEN.

edit: it's not possible the dummy account is probably the way to go until I can finish the flakebot app in a week or two

I don't think this is possible, personal access tokens are per-user only, not per-organization. I am an owner of another GitHub org and I see no such option to create a token at the org level.

The org can have secrets that are maintained at the org level, but their values (in our case: access tokens) would be generated per-user.

@rycee Yes. Please open a ticket at https://github.com/nix-community/infra

can we fix this ? that's the last issue before merging I think ? @Mic92 you seem to have the necessary rights, haven't you ?

.github/workflows/update.yml

asymmetric · 2021-05-07T11:01:09Z

.github/workflows/update.yml

+          PULL_REQUEST_FROM_BRANCH: flake-update
+          PULL_REQUEST_BRANCH: master
+          PULL_REQUEST_UPDATE: true
+          PASS_ON_ERROR: true


Why do we need this?

This will open a new pull request against the master branch, or, if one exists already, update the existing one. PASS_ON_ERROR is set to true since this action will always be run, regardless of whether the flake was updated during the workflow. So if there were no changes, this step would fail, but we don't want the workflow to fail because of that.

Can we instead check if there was any update, and optionally skip this step? Because I'm not sure if we should have a blanked ignore of all errors. Could be hard to notice when there are actual issues.

One way to do this is checking the commit_info declared above.

Hmm, this will be a lot harder than just checking commit_info, since that is literally the commit info of HEAD, regardless of whether a flake update was committed or not. I suppose I can check if a commit was made by comparing against github.sha, though.

After thinking about it some more, I think a better way to do this would be to ask the author of the action to let us specify to pass just if there are no new commits, that way I don't have to "reinvent the wheel". I can open an issue for that.

Action author has suggested that I just implement this logic in this workflow's step, so I have pushed 0be2c37. @asymmetric can you check if this looks right?

.github/workflows/update.yml

README.md

asymmetric · 2021-05-07T11:03:46Z

flake.nix

@@ -29,10 +31,9 @@
        hm = import ./modules/lib { lib = nixpkgs.lib; };
        homeManagerConfiguration = { configuration, system, homeDirectory
          , username, extraModules ? [ ], extraSpecialArgs ? { }
-          , pkgs ? builtins.getAttr system nixpkgs.outputs.legacyPackages
-          , check ? true, stateVersion ? "20.09" }@args:
+          , pkgs ? nixpkgsFor.${system}, check ? true, stateVersion ? "20.09"


This is a nit, but why do we need to use a custom function when nixpkgs.legacyPackages.${system} would suffice?

Figured it was a more succinct expression that was being used elsewhere anyways. I can revert this particular change, though.

asymmetric · 2021-05-07T11:06:32Z

This PR implements suggestions from #1913, adding a flake.lock file so we can nix run github:nix-community/home-manager without any hiccups.

If this is the only reason, wouldn't it make more sense to fix it on nix's side? Why does nix run try to save a lock-file in the first place? And shouldn't this be configurable somewhere?

i077 · 2021-05-07T14:49:40Z

This PR implements suggestions from #1913, adding a flake.lock file so we can nix run github:nix-community/home-manager without any hiccups.

If this is the only reason, wouldn't it make more sense to fix it on nix's side? Why does nix run try to save a lock-file in the first place? And shouldn't this be configurable somewhere?

See the discussion in NixOS/nix#4699.

asymmetric · 2021-05-11T09:18:45Z

.github/workflows/update.yml

+          git checkout -B flake-update
+      - name: Update & commit lockfile
+        run: |
+          nix flake update --commit-lock-file


nix flake update will update all inputs. At the moment it's only nixpkgs, but if we added more, that could be surprising.

Suggested change

nix flake update --commit-lock-file

nix flake lock --update-input nixpkgs --commit-lock-file

The PR body will list the updated inputs (actually, it just inserts the auto-generated commit message), so it won't be as surprising. I think it's better that we update all inputs if more get added so we don't have a situation where inputs haven't been updated in weeks or months.

teto · 2021-06-14T09:52:11Z

what is blocking ? looking forward to this to make lockstep updates in home-manager. Right now, when the nixos-unstable channel updates it can very well break home-manager. It would be better if we could choose what version of nixos-unstable works well via the flake.

Mic92 · 2021-06-16T20:32:11Z

what is blocking ? looking forward to this to make lockstep updates in home-manager. Right now, when the nixos-unstable channel updates it can very well break home-manager. It would be better if we could choose what version of nixos-unstable works well via the flake.

From my understanding it requires a github token still? I can add one on every level.

.github/workflows/update.yml

Mic92 · 2021-06-26T09:51:35Z

If you need any organisational changes please drop a message in #nix-community:bethselamin.de or
#tmp-nix-community:numtide.com on matrix

i077 · 2021-09-08T22:02:31Z

Has a token been added as a secret to this repo? What else needs to be done to move this PR forward?

Mic92 · 2021-11-04T06:26:46Z

Has a token been added as a secret to this repo? What else needs to be done to move this PR forward?

No. I have received no token so far. I think the best solution would be if one of the home-manager maintainer create a bot account that we add to the nix-community organisation.

stale · 2022-02-02T08:57:21Z

Thank you for your contribution! I marked this pull request as stale due to inactivity. If this remains inactive for another 7 days, I will close this PR. Please read the relevant sections below before commenting.

If you are the original author of the PR

GitHub sometimes doesn't notify people who commented / reviewed a PR previously when you (force) push commits. If you have addressed the reviews you can officially ask for a review from those who commented to you or anyone else.
If it is unfinished but you plan to finish it, please mark it as a draft.
If you don't expect to work on it any time soon, please consider closing it with a short comment encouraging someone else to pick up your work.
To get things rolling again, rebase the PR against the target branch and address valid comments.

If you are not the original author of the PR

If you want to pick up the work on this PR, please create a new PR and indicate that it supercedes and closes this PR.

Atry · 2022-05-02T01:49:01Z

When I tried to execute:

nix run github:nix-community/home-manager --no-write-lock-file

I got:

warning: not writing modified lock file of flake 'github:nix-community/home-manager':
• Added input 'nixpkgs':
    'github:NixOS/nixpkgs/b283b64580d1872333a99af2b4cef91bb84580cf' (2022-05-01)
error: attribute 'defaultApp.x86_64-linux' should have type 'derivation'

However, if I execute:

nix shell github:nix-community/home-manager --no-write-lock-file -c home-manager

It works well.

I think we might also want to add a defaultApp to flake.nix

terlar · 2022-05-02T08:40:09Z

This is already provided (defaultApp). What you are seeing is a recent change in Nix 2.8 where the defaultApp needs to be a derivation as opposed to an attribute set that was the convention before.

i077 · 2022-05-03T20:48:18Z

@DeterminateSystems have since released a GitHub Action that does what we would want, and it's probably better for home-manager to use that rather than maintain its own workflow. I can update this workflow to using that action instead.

It will use a default automated token for opening update PRs, but we'll still need to provide a personal auth token if we want to run tests against these PRs.

ncfavier · 2022-05-22T00:00:14Z

#2971 adds a flake.lock, so I guess it's blocked on this.

@DeterminateSystems have since released a GitHub Action that does what we would want, and it's probably better for home-manager to use that rather than maintain its own workflow. I can update this workflow to using that action instead.

Yes, I think that's better.

It will use a default automated token for opening update PRs, but we'll still need to provide a personal auth token if we want to run tests against these PRs.

This is currently not needed as the CI tests don't use the flake at all (maybe they should...).

This is already provided (defaultApp). What you are seeing is a recent change in Nix 2.8 where the defaultApp needs to be a derivation as opposed to an attribute set that was the convention before.

Fixed in #2971, see #2442 (comment).

About the interval, I think twice a week is excessive considering that the updates require human intervention. Once every one or two weeks seems about right, maybe less often for release branches if we choose to update them (maybe release branches should follow the appropriate nixos- release branch instead of unstable?).

i077 · 2022-05-23T13:47:31Z

This is currently not needed as the CI tests don't use the flake at all (maybe they should...).

My hope is that we'd later port rycee's home-manager testing framework to support flakes in a later PR. I was working on that very briefly, but stopped a while ago because of schoolwork.

About the interval, I think twice a week is excessive considering that the updates require human intervention. Once every one or two weeks seems about right, maybe less often for release branches if we choose to update them (maybe release branches should follow the appropriate nixos- release branch instead of unstable?).

I agree that release branches should follow the corresponding branch in nixpkgs. In particular, I think

master should follow nixpkgs-unstable since home-manager is developed against that branch anyway, and
release-xx.yy should follow nixos-xx.yy.

I suppose that means, going forward from release-21.11, we should change the nixpkgs input in flake.nix to follow the appropriate branch and cherry-pick backports on top of that as usual. (Cutting new release branches would involve updating flake.nix at that point as well.)

However, that would mean maintainers would have to review 2 PRs per run of this workflow (for master and current release branch).

Mic92 · 2022-05-29T11:56:37Z

I added a secret for a github bot in this repo. If you use the GH_TOKEN_FOR_UPDATES secret than you will get a token of this bot https://github.com/home-manager-bot
This bot can create branches on this repo

Mic92 · 2022-05-29T11:57:52Z

The access of this bot is managed with terraform here: https://github.com/Mic92/dotfiles/blob/master/terraform/modules/github-push-bot/main.tf and here https://github.com/Mic92/dotfiles/blob/master/terraform/github-bots/main.tf

ncfavier · 2022-06-07T18:57:32Z

#2971 was merged, we have a flake.lock now.

i077 · 2022-06-09T14:10:58Z

Thank you @ncfavier and @Mic92 for your help. A lot has happened since I opened this PR, so later today I will rebase my branch onto the tip of the master branch and redo most of the work so that it only adds the update workflow, since we now have a flake.lock.

As I mentioned somewhere earlier in this PR, future work will involve porting home-manager's tests to the checks output of flake.nix, so we can run CI tests against flake updates.

i077 · 2022-06-10T00:25:00Z

Alright, branch rebased. Since most of the work in the rest of the old commits was done elsewhere, this PR now just implements that update workflow. PRs will be tagged with the "dependencies" label (like dependabot does) to distinguish it from other PRs.

.github/workflows/update-flake.yml

ShamrockLee · 2022-06-16T16:14:55Z

The tests would be available through the legacyPackages output after ~~#3024~~ #3082 is merged.
It can be run automatically by

$ nix develop .#tests.run.all

README.md

teto · 2022-08-22T10:54:49Z

tempted to merge this (and if it breaks stuff we'll deal with it). Any objection ?

i077 · 2022-08-22T19:36:54Z

I'm fine with that. I will just draw attention to #1934 (comment), in particular whether release branches should receive a similar update workflow, perhaps in a later PR:

I suppose that means, going forward from release-21.11, we should change the nixpkgs input in flake.nix to follow the appropriate branch and cherry-pick backports on top of that as usual. (Cutting new release branches would involve updating flake.nix at that point as well.)

However, that would mean maintainers would have to review 2 PRs per run of this workflow (for master and current release branch).

We don't have to address this now, but perhaps if we merge this PR we should consider doing the above on the next NixOS release.

i077 requested a review from rycee as a code owner April 16, 2021 16:36

kclejeune reviewed Apr 16, 2021

View reviewed changes

.github/workflows/update.yml Outdated Show resolved Hide resolved

i077 force-pushed the flake-lock branch 3 times, most recently from 1c5b7e8 to 29daba6 Compare April 20, 2021 20:26

teto reviewed Apr 21, 2021

View reviewed changes

rycee reviewed May 1, 2021

View reviewed changes

.github/workflows/update.yml Outdated Show resolved Hide resolved

asymmetric reviewed May 7, 2021

View reviewed changes

asymmetric reviewed May 11, 2021

View reviewed changes

i077 force-pushed the flake-lock branch from 5a9008e to a389454 Compare May 13, 2021 17:06

teto mentioned this pull request May 27, 2021

Home.nix doesn't seem to be respecting neovim extraConfig anymore #2040

Closed

i077 mentioned this pull request Jun 16, 2021

Add an option to pass only if no commits between source and target vsoch/pull-request-action#73

Closed

asymmetric reviewed Jun 25, 2021

View reviewed changes

.github/workflows/update.yml Outdated Show resolved Hide resolved

asymmetric reviewed Jun 25, 2021

View reviewed changes

.github/workflows/update.yml Outdated Show resolved Hide resolved

asymmetric reviewed Jun 25, 2021

View reviewed changes

.github/workflows/update.yml Outdated Show resolved Hide resolved

i077 force-pushed the flake-lock branch from 1568de7 to 0be2c37 Compare June 28, 2021 23:07

i077 mentioned this pull request Aug 13, 2021

Add home-manager NixOS/flake-registry#11

Merged

berbiche mentioned this pull request Nov 23, 2021

add flake attribute apps to make it easier to run #2442

Merged

7 tasks

ncfavier mentioned this pull request May 20, 2022

Prepare inclusion in nixos-search #2971

Merged

5 tasks

ci: Add workflow to update flake inputs

4b713d6

i077 force-pushed the flake-lock branch from 0be2c37 to 4b713d6 Compare June 10, 2022 00:20

ncfavier reviewed Jun 10, 2022

View reviewed changes

.github/workflows/update-flake.yml Show resolved Hide resolved

.github/workflows/update-flake.yml Outdated Show resolved Hide resolved

ncfavier reviewed Jun 10, 2022

View reviewed changes

.github/workflows/update-flake.yml Outdated Show resolved Hide resolved

i077 force-pushed the flake-lock branch from 49ba8e5 to 9f44989 Compare June 10, 2022 15:57

ci: Don't use token in update workflow

c7aea0b

i077 force-pushed the flake-lock branch from 9f44989 to c7aea0b Compare June 10, 2022 15:58

ncfavier approved these changes Jun 14, 2022

View reviewed changes

This was referenced Jun 15, 2022

flake.nix: provide checks (tests) and docs and related fixes #3024

Closed

Questions about the update and release model of the flake.lock-included pattern #3026

Closed

berbiche approved these changes Jul 8, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

ShamrockLee mentioned this pull request Jul 10, 2022

flake.nix: provide checks (tests) and docs and related fixes (cont'd) #3082

Open

7 tasks

teto merged commit 0160a0c into nix-community:master Aug 22, 2022

GaetanLepage pushed a commit to GaetanLepage/home-manager that referenced this pull request Aug 23, 2022

Add flake.lock & workflow to update it (nix-community#1934)

5f35a1b

austinharris pushed a commit to austinharris/home-manager that referenced this pull request Dec 23, 2022

Add flake.lock & workflow to update it (nix-community#1934)

ea92191

spacekookie pushed a commit to spacekookie/home-manager that referenced this pull request Feb 10, 2023

Add flake.lock & workflow to update it (nix-community#1934)

1041199

	nix flake update --commit-lock-file
	nix flake lock --update-input nixpkgs --commit-lock-file

Add flake.lock & workflow to update it #1934

Add flake.lock & workflow to update it #1934

Conversation

i077 commented Apr 16, 2021 • edited Loading

Description

Checklist

teto commented Apr 18, 2021

i077 commented Apr 19, 2021 • edited Loading

teto commented Apr 19, 2021

i077 commented Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kclejeune May 2, 2021 • edited Loading

Choose a reason for hiding this comment

i077 May 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asymmetric May 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asymmetric commented May 7, 2021

i077 commented May 7, 2021

Choose a reason for hiding this comment

i077 May 12, 2021 • edited Loading

Choose a reason for hiding this comment

teto commented Jun 14, 2021

Mic92 commented Jun 16, 2021

Mic92 commented Jun 26, 2021 • edited Loading

i077 commented Sep 8, 2021 • edited Loading

Mic92 commented Nov 4, 2021

stale bot commented Feb 2, 2022

Atry commented May 2, 2022

terlar commented May 2, 2022

i077 commented May 3, 2022

ncfavier commented May 22, 2022

i077 commented May 23, 2022

Mic92 commented May 29, 2022 • edited Loading

Mic92 commented May 29, 2022

ncfavier commented Jun 7, 2022

i077 commented Jun 9, 2022

i077 commented Jun 10, 2022

ShamrockLee commented Jun 16, 2022 • edited Loading

teto commented Aug 22, 2022

i077 commented Aug 22, 2022 • edited Loading

i077 commented Apr 16, 2021 •

edited

Loading

i077 commented Apr 19, 2021 •

edited

Loading

i077 commented Apr 19, 2021 •

edited

Loading

kclejeune May 2, 2021 •

edited

Loading

i077 May 2, 2021 •

edited

Loading

asymmetric May 11, 2021 •

edited

Loading

i077 May 12, 2021 •

edited

Loading

Mic92 commented Jun 26, 2021 •

edited

Loading

i077 commented Sep 8, 2021 •

edited

Loading

Mic92 commented May 29, 2022 •

edited

Loading

ShamrockLee commented Jun 16, 2022 •

edited

Loading

i077 commented Aug 22, 2022 •

edited

Loading