Passing arguments to test suites run via new-test is unsupported

[cocreature]

Currently it is impossible to pass any arguments to a test suite that is run via new-test since they are interpreted as targets (regardless of how many -- you add in between).

There is the question of what should happen if multiple test suites are run. Personally, I would prefer if the arguments are just passed to all of them in that case but I’m also fine with an error or whatever else as long as I have the possibility to specify the test suite explicitly.

[fgaz]

This probably also affects new-bench

[23Skidoo]

Just bumped into this the other day. I think that none of the flags specific to old-test (TestFlags) are supported by new-test right now.

[fgaz]

Apparently some are set to mempty.
See setupHsTestFlags and setupHsTestArgs in Distribution/Client/ProjectPlanning.hs

[lspitzner]

in the meantime..

cabal new-build TESTNAME && `find dist-newstyle/build -type f -name TESTNAME` --color --my --other --args

which probably is brittle and breaks if you build tests with and without optimizations at some point..

[hvr]

@lspitzner less brittle: use cabal-plan list-bin :-)

[fgaz]

Continuing on #3638 (comment),

That's because @hvr's example uses (currently) incorrect syntax.

As with new-run, the full syntax seems to be:

cabal new-test --cabal-flags -- test-name other-test -- --test-flags and test args

The first -- separates cabal flags and targets, and the second one separates targets and targets' args.

This correctly parses as

["test-name","other-test","--","--test-flags","and","test","args"]

Edit: well, maybe the current syntax wasn't intended to be used like this?

[ezyang]

Let me offer one more unsolicited suggestion: maybe cabal new-test should NOT support running multiple tests. If you want multiple tests, make a new command or something.

[hvr]

I somewhat sympathize with @ezyang's suggestion; it would probably make things simpler and less surprising. However, we still want the ability to say cabal new-test all or cabal new-test :pkg:cabal-install. Maybe just disallow running multiple tests and passing flags at the same time; or expressed dually: allow flags only when the target resolves to a single test program?

[phadej]

Yes, there should be a way to run all tests (in project/package) without knowing the test-suite names.

[fgaz]

Partially fixed with #4861 (new-run can now run tests/benchs and it can pass arguments).

new-test still cannot pass arguments.

[sjakobi]

I'd love to see a fix for this! I think the new-run workaround doesn't work for me because it doesn't support --allow-newer.

[l29ah]

new-test and new-bench still don't accept args via --, as seen with cabal-3.6.2.0

[ulysses4ever]

@l29ah unfortunately, this is correct. The corresponding issue is #6198. There's a draft PR attached to it (#7454) with a simple and straightforward design, but it's abandoned, as far as I can see. Do you want to try to pull it out? We're happy to advise.