Run test suite on NixOS #7778

roberth · 2023-02-07T16:37:08Z

Motivation

This one was a rabbit hole.

Original goal: try to fix structuredAttrs $NIX_ATTRS_JSON_FILE points to the wrong location in sandbox #6736
So then find a reproducer
The reproducer needs a proper sandbox
We don't usually run tests with a proper sandbox. That's not good.
Add NixOS vm test that runs the test suite
Add a whole bunch of stuff to make that work

TODO

remove the bad and/or controversial patches to
document a matrix containing outcomes of test helper functions for the axes
- which function
- which test environment
  - devshell build
  - pure build
  - vm test as user
  - vm test as trusted user
  - vm test as root
clean up the test helper functions (insofar as they are new, perhaps; scope creep)
actually fix the bug structuredAttrs $NIX_ATTRS_JSON_FILE points to the wrong location in sandbox #6736
consider Test everything before merging with merge queue + hercules-ci #7674; running this async may turn out to be more painful than necessary
maybe incorporate Clean up daemon handling in the tests #5753
- maybe split out logic functions, setup functions

Context

Testing is how we make sure that we

don't break too many things
can trust contributions to do the right thing
can ask contributors to write good tests
Test everything before merging with merge queue + hercules-ci #7674

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
code and comments are self-explanatory
commit message explains why the change was made
new feature or bug fix: updated release notes

This also makes the path ctrl+clickable in vscode.

roberth · 2023-02-07T16:40:40Z

flake.nix

+            src = lib.cleanSourceWith {
+              src = ./.;
+              filter = path: type:
+                toString path != toString ./tests/nixos
+                && toString path != toString ./flake.nix
+                ;
+            };


This change will be removed.
An improved change can be made after

simplify flake #6538

Ideally the filter is additive - not subtractive - as subtractive filter don't scale well to projects with multiple derivation, which is increasingly the case for NixOS/nix.

roberth · 2023-02-07T16:42:21Z

src/libexpr/primops.cc

+    if (evalSettings.pureEval) {
+        // TODO feature flag?
+        //     state.debugThrowLastTrace(EvalError({
+        //         .msg = hintfmt("'%s' is not allowed in pure evaluation mode", "builtins.storePath"),
+        //         .errPos = state.positions[pos]
+        // }));'
+        state.allowPath(uncheckedPath);
+    }


I'll look for a workaround, as I know this is controversial. builtins.storePath is referentially transparent, and only undesirable for some. If I can find an alternative for this change, I'll use that instead.

roberth · 2023-02-07T16:44:20Z

tests/binary-cache-build-remote.sh

@@ -1,5 +1,8 @@
 source common.sh

+enableFeatures nix-command


I've had to add a lot of these enableFeatures, as I could not find the supposed central place where they should be enabled. Doing it on a case by case basis made more sense to me, but if the in-build tests don't enforce these calls that's going to be annoying. The in-build tests should cover almost everything, so that the VM tests are almost guarnateed to succeed after that.

roberth · 2023-02-07T16:44:59Z

tests/binary-cache-build-remote.sh

@@ -1,5 +1,8 @@
 source common.sh

+enableFeatures nix-command
+unknownFailsOnNixOS # fails because of path may have been built and not gc-ed; generate unique derivation?


We'll want to resolve some of these later, so that we don't have to block this PR for too long.

roberth · 2023-02-07T16:47:28Z

tests/common.sh.in

+isTestOnSystemNix() {
+  [[ "${TEST_SYSTEM_NIX:-}" = 1 ]]
+}
+


These functions should be made a bit more accurate, and "integrate" better with the existing helper functions, which should also be made a bit more accurate or precise.

roberth · 2023-02-07T16:51:48Z

tests/nixos/testsOnNixOS.nix

+          # Tests we don't need
+          echo >tests/plugins/local.mk
+          sed -i tests/local.mk \
+            -e 's^plugins\.sh^^' \
+            -e "s^test-deps += tests/plugins/libplugintest.\$(SO_EXT)^^" \
+            ;
+
+          export TEST_SYSTEM_NIX=1
+          export version=${config.nix.package.version}
+          export NIX_REMOTE_=daemon
+          export NIX_REMOTE=daemon
+          make -j1 tests/eval-store.sh.test installcheck --keep-going


These adaptation should be in an entrypoint that can be run outside of the VM tests. Benefits

Quicker iteration on reproducers and improvements in test coverage that don't require main code changes

Easier troubleshooting of test suite issues

roberth · 2023-02-07T17:05:55Z

tests/nixos/testsOnNixOS.nix

+    nix.package = (builtins.getFlake "git+file:///home/user/h/nix?rev=a1c7cb8fb6c6511daf4884e140aba8ae1788cd84").packages.x86_64-linux.default;
+    nix.settings.substituters = lib.mkForce [];
+
+    nixpkgs.overlays = [
+      (final: prev: {
+        thisNix = prev.nix;
+        nix = config.nix.package;
+      })
+    ];


The purpose of the nix.package and overlay hacks here is to not rebuild the nix derivation every time the test suite is changed during local development. This hack should be moved to a convenience attribute and not run on CI. It should use HEAD instead of a specific hash.
Alternatively, we could have a mode that adds an impurely built (devshell) nix into the store somehow, or we could build Nix in such a way that it does not depend on its test suite.

Possibly helpful is Track test derivations and parallelize building and testing #7662, to recover the testedness guarantee (or work around it by adding an indirection that runs both types of test)

roberth · 2023-04-10T14:54:52Z

Will re-do with fewer changes (and probably less coverage at first)

roberth · 2024-06-16T16:01:59Z

Closing in favor of Run the functional tests in a NixOS environment #10922

roberth added 4 commits February 6, 2023 15:13

run-test.sh: Remove ellipsis; test has already run

bd4ee88

This also makes the path ctrl+clickable in vscode.

WIP allow builtins.storePath in tests

a1c7cb8

TMP filter source

f1a68d9

WIP tests

04b896e

roberth commented Feb 7, 2023

View reviewed changes

roberth mentioned this pull request Feb 7, 2023

structuredAttrs $NIX_ATTRS_JSON_FILE points to the wrong location in sandbox #6736

Closed

roberth changed the title ~~Runs test suite on NixOS, work towards fixing issue #6736~~ Run test suite on NixOS, work towards fixing issue #6736 Feb 7, 2023

roberth commented Feb 7, 2023

View reviewed changes

roberth mentioned this pull request Feb 20, 2023

Source filtering and disabling tests (to later run them externally) #7870

Open

7 tasks

roberth added bug with-tests Issues related to testing. PRs with tests have some priority labels Feb 20, 2023

Artturin mentioned this pull request Mar 16, 2023

LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds #8053

Merged

7 tasks

roberth mentioned this pull request Mar 21, 2023

Fix builtins.storePath in pure mode #8090

Closed

7 tasks

roberth mentioned this pull request Apr 10, 2023

More installer tests #8178

Open

8 tasks

roberth closed this Apr 10, 2023

fricklerhandwerk mentioned this pull request Apr 14, 2023

Reorganize this repo #7876

Open

roberth reopened this Oct 13, 2023

github-actions bot removed the with-tests Issues related to testing. PRs with tests have some priority label Oct 13, 2023

fricklerhandwerk added the tests label Jun 12, 2024

fricklerhandwerk changed the title ~~Run test suite on NixOS, work towards fixing issue #6736~~ Run test suite on NixOS Jun 12, 2024

fricklerhandwerk removed the bug label Jun 12, 2024

roberth mentioned this pull request Jun 16, 2024

Run the functional tests in a NixOS environment #10922

Merged

roberth closed this Jun 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run test suite on NixOS #7778

Run test suite on NixOS #7778

roberth commented Feb 7, 2023 •

edited

Loading

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth Feb 7, 2023

roberth commented Apr 10, 2023

roberth commented Jun 16, 2024

Run test suite on NixOS #7778

Run test suite on NixOS #7778

Conversation

roberth commented Feb 7, 2023 • edited Loading

Motivation

Context

Checklist for maintainers

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth Feb 7, 2023

Choose a reason for hiding this comment

roberth commented Apr 10, 2023

roberth commented Jun 16, 2024

roberth commented Feb 7, 2023 •

edited

Loading