Test suite overhaul #3398
Test suite overhaul #3398
Conversation
|
I tried to break up the |
|
There's some residual classloader leakage in the test suite; |
|
@lefou this should be ready to look at if you have some time. The most important stuff to review here is the code in I'm planning on adding some examples/documentation for what the "happy path" for someone writing a Mill plugin looks like, using the new APIs. But that can come as a follow up as this PR is big enough already |
| val sv = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) | ||
| val sjsv = sys.props.getOrElse("TEST_SCALAJS_VERSION", ???) | ||
| val sv = sys.props("TEST_SCALA_2_13_VERSION") | ||
| val sjsv = sys.props("TEST_SCALAJS_VERSION") |
There was a problem hiding this comment.
This change means, we silently go with a null, making any test failure hard to trace back to a missing sysprop. The explicit ??? ensured we'd see the cause, the missing sysprop, early. Sysprops are an easy way to configure the test suite from the build tool, but they can be easily broken, too. Can you revert to the sys.prop.getOrElse notation in all places?
There was a problem hiding this comment.
Huh I didn't realize sys.props("doesntexist") returns null. I thought it would throw since sys.props is a Map, but I guess there's a default value. I can revert those changes
|
I just removed testing for scala 2.11 native 0.4. The code didn't change, but those two significantly slowed down the test suite by bloating the test matrix, so I wanted to trim it down where possible. The code may or may not work, but by removing the tests we are stopping the guarantee that it should |
We could have just disabled the Scala 2.11 test in the test matrix, but run it occasionally before a release. It's a bit unfortunate to reduce coverage. I always found it a (unique) selling point for Mill, that all our main build chains are covered by tests. As a consequence, we should mark Scala 2.11 as (officially) unsupported, since we no longer have easy ways to detect or reproduce issues. |
* Added `Kotlin_Intro_to_Mill.adoc`, `Kotlin_Installation_IDE_Support.adoc`, `Kotlin_Builtin_Commands.adoc` pages * Fixed an issue with inherited example adoc pages not rendering correctly since #3398
Mill's own test fixtures are pretty decent, and could definitely benefit external plugin authors. But the implementation, API, and usage within Mill's own test suite has always been a mess. This PR takes the first step at centralizing all of them in
mill.testkitand tries to clean them up as much as possible. This improves our own experience working in the Mill codebase, and ensures the test fixtures are in decent shape to be published and used by downstream plugin authors.We traditionally never guaranteed binary compatibility for
mill.testkit, probably for good reason since it was never as polished or designed as the rest of Mill's APIs, and this PR should get us much closer to the point at which we can start giving compatibility guarantees.This PR is definitely too big to properly review in detail, but the test suite passing should give us some confidence I didn't break everything. It only also only touches test code, and should not affect the packaged Mill distribution at all
Major Changes
ExampleTestSuite,IntegrationTestSuite, andTestEvaluatorhave been moved intomill.testkitTestEvaluatorhas been renamedUnitTester, andExampleTestSuiteandIntegrationTestSuitehave hadExampleTesterandIntegrationTesterclasses broken out of them. These*Testerclasses contain all the test-framework-agnostic functionality, so it can easily be used independently of Mill or uTest.There are now
IntegrationTesterTestsandExampleTesterTeststo go along withUnitTesterTests(previouslyUnitTestTests), illustrating how to use all these helpers in standalone scenariosUnitTesterChangesConsolidated most of the
def workspaceTestboilerplate functions into theUnitTesterclassA lot of test code and test data needed to be slightly tweaked to accommodate the now standardized
UnitTesterfolder setupUnitTesterno longer tries to pick two nicely-named folders on disk to work with, one for the project source code and one for the out folder, and instead just does all its work inside anos.temp.dir()without/as a sub-folderDropped support for Scala 2.11, 3.0, 3.1 in the test suite, as well as Scala-Native 0.4, following the rest of the com-lihaoyi projects.
Replaced the long-deprecated utest
"foo" - {syntax withtest("foo"){UnitTester#applynow returns aUnitTester.Resultobject, allowing us to fetch fields via.valueand.evalCountrather than fiddling with tuplesMillTestKit.scalahas been removed, withUnitTesterandTestBaseModulebroken out into their own source files, and all the helper methods to generate nice folder paths are no longer used and have been removedIntegrationTesterChangesThe various
eval*methods have been consolidated into a singlevalmethod that follows OS-Lib's newos.callAPIThe various
meta*methods have been consolidated into a single.meta("...").*APIinitWorkspaceno longer returns the workspace path, since it is already available throughIntegrationTester#workspacePathExampleTesterChangesMoved the logic for parsing example test
build.scfiles intomill-build/src/ExampleParser.scala, so we can share it between Mill's ownbuild.scas well as the application code inmill.testkitvia the meta-buildExampleTestercan now be used standalone vianew ExampleTester(clientServerMode: Boolean, workspaceSourcePath: os.Path, millExecutable: os.Path).run(), without needing any special integration with the Mill build. Downstream users can still do build-level stuff if they wish, and Mill's own build does so, but it is no longer mandatory andExampleTestercan be used as part of any vanilla test suite.