Port 'TestFlags' to 'new-test' #5455

harpocrates · 2018-07-24T07:30:47Z

Make all of the old-style test flags available to new-test/new-build/new-install/etc. This should include:

--log (as --test-log)
--machine-log (as --test-machine-log)
--show-details (as --test-show-details)
--keep-tix-files (as --test-keep-tix-files)
--test-options (as --test-options)
--test-option (as --test-option)

Please include the following checklist in your PR:

Patches conform to the coding conventions.
Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.

Please also shortly describe how you tested your change. Bonus points for added tests!

Fixes #4803
Fixes #4643
Fixes #4766
Fixes #5416

harpocrates · 2018-07-24T07:33:03Z

As it stands, I’m sure this PR is broken in all sorts of small ways. However, I would love if someone with some experience could look over the general approach - I’m not very knowledgeable in cabal, so this whole PR might be fundamentally flawed and it would be great to know that before I start fixing the smaller bugs.

This scratches a longstanding itch I’ve had for the Haddock test suite:

haddock$ cabal new-test --test-option="--accept" html-test latex-test hoogle-test

harpocrates · 2018-07-25T15:43:07Z

This is ready for review.

Assuming everything that passed before I rebased continues to pass, there will be one CI failure. Specifically, GHCVER=7.10.3 SCRIPT=script USE_GOLD=YES crashes when GHC runs out of memory (relevant log line). If this is indeed my responsibility, I would appreciate some pointers on where to start debugging this.

EDIT

Looks like Travis is going to be all green, but now Appveyor is failing where it wasn't before. I'm going to assume this is all due to flakiness in the tests.

fgaz · 2018-08-20T15:00:32Z

cabal-install/tests/UnitTests/Distribution/Client/ProjectConfig.hs

@@ -696,6 +713,9 @@ instance Arbitrary PackageConfig where
 instance Arbitrary HaddockTarget where
    arbitrary = elements [ForHackage, ForDevelopment]

+instance Arbitrary TestShowDetails where
+    arbitrary = elements [Never, Failures, Always, Streaming, Direct]


elements [minBound..maxBound] in case we add/remove some constructors

Now that you say it, why not arbitraryBoundedEnum :: (Bounded a, Enum a) => Gen a?

Naively threads 'TestFlags' through the 'new-test' argument parsing through to the backend.

This means that test flags can be passed in to 'new-build', 'new-install', 'new-configure', etc. I've mostly just followed the lead of 'HaddockFlags' here.

Paths with escaped characters would, instead of shrinking, grow.

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-7.8.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-7.6.3" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.2.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.0.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.4.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.6.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.4.4-fdebug-expensive-assertions" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"osx-7.8.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"osx-8.0.2" }

quasicomputational · 2018-11-12T15:29:45Z

cabal-install/changelog

@@ -15,6 +15,9 @@
 	  no longer ignores all arguments except the last one (#5512).
 	* Add the following option aliases for '-dir'-suffixed options:
 	  'storedir', 'logsdir', 'packagedir', 'sourcedir', 'outputdir' (#5484).
+	* Ported old-style test options to the new-style commands (#5455).
+
+2.4.0.0


Looks like a mis-merge here.

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.8.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.6.3" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.10.3" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.2.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.0.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.4.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.6.2" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-7.8.4" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.4.4-fdebug-expensive-assertions" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-7.10.3" }

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-8.0.2" }

harpocrates · 2018-11-12T18:38:24Z

@quasicomputational Could we merge this soon? I'm getting tired of rebasing and of having to direct people to a workaround for Haddock. I really want to just tell folks to run

$ cabal new-test -w ghc-head html-test --test-options="--accept"

instead of explaining why we need

$ cabal new-run -w ghc-head html-test -- \
       --accept \
       --ghc-path ghc-head \
       --haddock-path "$(pwd)/dist-newstyle/build/x86_64-osx/ghc-8.7.20181106/haddock-2.22.0/x/haddock/build/haddock/haddock

The only CI failure looks network related.

quasicomputational · 2018-11-12T19:06:32Z

LGTM. I'd say to go ahead and merge whenever - I can't see any problems and nor has anyone else, evidently.

harpocrates · 2018-11-12T20:04:33Z

Thanks for the review @quasicomputational - I've merged.

harendra-kumar · 2019-01-13T17:38:03Z

@harpocrates is there a reason for the --test-show-details to not work with ghcjs build. I am noticing that it seems to work fine with ghc builds but not ghcjs.

harpocrates · 2019-01-13T17:39:39Z

@harendra-kumar Not that I know of. What are you seeing? Does cabal v1-test work?

harendra-kumar · 2019-01-13T23:00:52Z

@harpocrates I think there is no difference in the two cases, I had the wrong idea about that. However, earlier when I used --show-details=always in the CI tests, the output came on console as the tests were running. Now it does not seem to work that way, but --test-show-details=streaming does that job for me. So I am good.

Some caveats: * Cabal version in CI has been bumped to 3.0+ due to need of test flags support: haskell/cabal#5455 * v1-copy was dropped, since it requires v1 build artifacts. There is no v2-copy command AFAICT. * I don't think there is an equivalent for the v1-install command for libraries?

fgaz added the cabal-install: nix-local-build label Jul 24, 2018

harpocrates force-pushed the feature/new-test-flags branch 3 times, most recently from 8bea237 to 1a634ba Compare July 25, 2018 15:18

harpocrates changed the title ~~[WIP] Port 'TestFlags' to 'new-test'~~ Port 'TestFlags' to 'new-test' Jul 25, 2018

harpocrates force-pushed the feature/new-test-flags branch from 1a634ba to 0ff2a99 Compare July 25, 2018 16:24

harpocrates added the attention: needs-review label Jul 26, 2018

harpocrates mentioned this pull request Aug 10, 2018

cabal test is dropping --test-option flags #920

Closed

harpocrates force-pushed the feature/new-test-flags branch 4 times, most recently from 18004fd to d0d3415 Compare August 13, 2018 16:20

fgaz added this to the 3.0 milestone Aug 20, 2018

fgaz reviewed Aug 20, 2018

View reviewed changes

harpocrates added 5 commits November 11, 2018 18:27

Port 'TestFlags' to 'new-test'

11060c6

Naively threads 'TestFlags' through the 'new-test' argument parsing through to the backend.

Add 'TestFlags' to 'installCommand'

7db8d71

This means that test flags can be passed in to 'new-build', 'new-install', 'new-configure', etc. I've mostly just followed the lead of 'HaddockFlags' here.

Fix non-shrinking 'Arbitrary PathTemplate'

e32b9d1

Paths with escaped characters would, instead of shrinking, grow.

Fix remaining tests and update changelog

a06b3e3

Use 'arbitraryBoundedEnum'

b5858c6

harpocrates force-pushed the feature/new-test-flags branch from 9a42cee to b5858c6 Compare November 12, 2018 02:39

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

ca727f1

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-7.8.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

30c7ba9

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-7.6.3" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

bfe38a5

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.2.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

d8b574a

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.0.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

7416388

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.4.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

0856f50

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"linux-8.6.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

58ae027

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"osx-7.8.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

5fa7f83

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "b5858c66b1ae318e2288a79e531a30a488b89b5c", "tag":"osx-8.0.2" }

Remove redundant import

3cad549

quasicomputational reviewed Nov 12, 2018

View reviewed changes

Fix a merge error in changelog

c713430

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

a34ae38

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.8.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

6f4989f

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.6.3" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

35392e8

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-7.10.3" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

025c0b9

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.2.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

4fd0524

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.0.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

607435b

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.4.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

c3d409a

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"linux-8.6.2" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

766aa87

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-7.8.4" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

44a1149

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-7.10.3" }

haskell-pushbot pushed a commit to haskell-pushbot/cabal-binaries that referenced this pull request Nov 12, 2018

{"origin":"haskell/cabal/pull/5455",

b537455

"url":"pull/5455", "account":"haskell", "repo":"cabal", "commit": "c71343083987a725f2b393b8f30793eada7325ee", "tag":"osx-8.0.2" }

harpocrates merged commit de5d2c5 into haskell:master Nov 12, 2018

harpocrates deleted the feature/new-test-flags branch November 12, 2018 20:04

gwils mentioned this pull request Feb 1, 2019

Instructions for new-style cabal builds system-f/fp-course#317

Closed

emilypi mentioned this pull request May 21, 2021

Can I pass command line arguments to a test suite when using new-test? #5416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port 'TestFlags' to 'new-test' #5455

Port 'TestFlags' to 'new-test' #5455

harpocrates commented Jul 24, 2018 •

edited

Loading

harpocrates commented Jul 24, 2018

harpocrates commented Jul 25, 2018 •

edited

Loading

fgaz Aug 20, 2018

harpocrates Aug 20, 2018

fgaz Aug 20, 2018

quasicomputational Nov 12, 2018

harpocrates Nov 12, 2018

harpocrates commented Nov 12, 2018

quasicomputational commented Nov 12, 2018

harpocrates commented Nov 12, 2018

harendra-kumar commented Jan 13, 2019

harpocrates commented Jan 13, 2019

harendra-kumar commented Jan 13, 2019

Port 'TestFlags' to 'new-test' #5455

Port 'TestFlags' to 'new-test' #5455

Conversation

harpocrates commented Jul 24, 2018 • edited Loading

harpocrates commented Jul 24, 2018

harpocrates commented Jul 25, 2018 • edited Loading

EDIT

fgaz Aug 20, 2018

Choose a reason for hiding this comment

harpocrates Aug 20, 2018

Choose a reason for hiding this comment

fgaz Aug 20, 2018

Choose a reason for hiding this comment

quasicomputational Nov 12, 2018

Choose a reason for hiding this comment

harpocrates Nov 12, 2018

Choose a reason for hiding this comment

harpocrates commented Nov 12, 2018

quasicomputational commented Nov 12, 2018

harpocrates commented Nov 12, 2018

harendra-kumar commented Jan 13, 2019

harpocrates commented Jan 13, 2019

harendra-kumar commented Jan 13, 2019

harpocrates commented Jul 24, 2018 •

edited

Loading

harpocrates commented Jul 25, 2018 •

edited

Loading