Can I pass command line arguments to a test suite when using new-test?

[kindaro]

There is an option cabal test --test-options. Is there a corresponding cabal new-test option? If not, can it be introduced?

This is what I do with test v1:

% cabal test echo-args --test-options "We are test options."
...
Resolving dependencies...
Configuring x-0...
Preprocessing test suite 'echo-args' for x-0..
Building test suite 'echo-args' for x-0..
Running 1 test suites...
Test suite echo-args: RUNNING...
Test suite echo-args: PASS
Test suite logged to: dist/test/x-0-echo-args.log
1 of 1 test suites (1 of 1 test cases) passed.
% cat dist/test/x-0-echo-args.log
Test suite echo-args: RUNNING...
### Getting arguments. ###
"We"
"are"
"test"
"options."
Test suite echo-args: PASS
Test suite logged to: dist/test/x-0-echo-args.log

This is what happens when I replace test with new-test:

% cabal new-test echo-args --test-options "We are test options."
cabal: unrecognized 'new-test' option `--test-options'

I looked through cabal new-test --help two times, but I was not able to identify an option corresponding to --test-options. There are many, maybe I missed something.

The relevant section of x.cabal:

test-suite echo-args
    type: exitcode-stdio-1.0
    main-is: EchoArgs.hs
    ghc-options: -main-is EchoArgs.main
    build-depends: base
    hs-source-dirs: test

The testing module:

module EchoArgs where

import System.Environment
import Data.Foldable

main = do
    putStrLn "### Getting arguments. ###"
    getArgs >>= traverse_ print

Cabal version:

% cabal --version
cabal-install version 2.3.0.0
compiled using version 2.3.0.0 of the Cabal library 

[quasicomputational]

You can new-run the test suite and pass arguments like that, e.g., cabal new-run echo-args -- These will be echoed. I agree that that's not obvious so I'll write a patch to add it to --help.

[23Skidoo]

IMO --test-options should also be supported.

[quasicomputational]

That would be applying to every test suite being run by new-test, right?

[hvr]

@23Skidoo didn't we have a discussion that passing the same flags to all testsuites is of modest benefit as soon as you have more than one testsuite, as it'd imply that all testsuites have the same CLI, and which is also where we got stuck w/ passing args to new-test? IMO, we should rather promote the use of cabal new-run test:... -- extraflags as users will imho run into the limitation of something like --test-options pretty soon anyway. Or put differently, why would anyone ever want to use --test-options over new-test test:... -- extra-options?

[harpocrates]

Or put differently, why would anyone ever want to use --test-options over new-test test:... -- extra-options?

Perhaps because (at least for now) the build-tool-depends stuff applies to new-test, but not to new-run? Also, running tests with new-test seems more intuitive than with new-run.

I'm in favor of --test-options working at least in the case when there is one test suite. Perhaps we could print out an error message (or a warning) if --test-options is specified and there are multiple test suites?

[kindaro]

Or put differently, why would anyone ever want to use --test-options over new-test test:... -- extra-options?

I would be very glad to be able to say:

% cabal new-test echo-args -- "We are test options."

— But, unfortunately, it does not work:

% cabal new-test echo-args -- "We are test options."
cabal: Unrecognised target syntax for 'We are test options.'.

If the support for passing arguments like this is added to test, I think it would be nice. Of course, we may print a warning if arguments are specified, but target is not.

---

For now, I can live with passing information through environment, like this:

% X=meow cabal new-test echo-env | grep -w X
("X","meow")

— But it feels a bit hackish.

[hvr]

@kindaro that's the exact discussion we had already somewhere and it's not going to happen any time soon (because of technical CLI issues)

PS: Found some of the discussions:

• [nix-local-build] new-run, new-test, new-bench, new-exec #3638 (comment)
• Passing arguments to test suites run via new-test is unsupported #4643 (comment)

[quasicomputational]

I can imagine a use case for --test-options being passed to multiple test suites: a project with multiple packages, each of which has a test suite with a specific interface (e.g., many tasty-golden test suites that you want to pass --accept to). But I think it's a bit of a niche because I think this fails if any of those packages have a test suite with a different interface. There might be room here for a nice solution but it needs some design work.

[moll]

Is it me or does using cabal v2-run test fail to notice changes in test files? I'm using Hspec with hspec-discover and, while cabal v2-test picks changes up and recompiles the affected files nicely, cabal v2-run test doesn't.

[parsonsmatt]

This feature would be extremely useful to me in CI runners. I use hspec for all of my test suites and I want to pass a --fail-on-focused flag to hspec for all tests. With stack this is trivial - stack test --test-argumetns "--fail-on-focused". CI is currently using cabal-install, and, if I'm reading this right, I'd need to add a line for every single test suite in the repository.

I don't expect it'll be implemented, but I do want to register the request along with a use-case.

[emilypi]

Fwiw, I think this issue should be re-opened, if only to track cabal run caching problems, and to track along with #6198.

[emilypi]

@parsonsmatt i saw from your tweet that this was labeled WONTFIX. But it looks like @harpocrates actually implemented this in #5455 and that's why it was closed. You can call cabal test --test-options="..." in cabal, and cabal v2-run --test-options as well:

    --test-options=TEMPLATES                             give extra options to test
                                                         executables (name templates
                                                         can use $pkgid, $compiler,
                                                         $os, $arch, $test-suite)

Could you describe what you're seeing in more depth, and where you saw WONTFIX? The reason being that Cabal is under new management, and we want to revisit past decisions that perhaps weren't as friendly as they could be.

[parsonsmatt]

Ah, I misunderstood this:

kindaro that's the exact discussion we had already somewhere and it's not going to happen any time soon (because of technical CLI issues)

I thought this meant "`--test-options" is not going to happen any time soon", but it's something else.

[jneira]

The issue title says: Can I pass command line arguments to a test suite when using new-test?
And i think nowadays the answer is "yes, you can with --test-options". I think cabal run caching problems are mentioned in other issues (or a new one could be opened)