Parallelize test run (extending xUnit)

[jdom]

Hi, there's something I wanted to do for a long time, but never find the actual time to do so. Maybe somebody already has the expertise and would like to help out.
Today all of our tests run serially and take a long time. Ideally, once we start an in-memory test cluster, we should be able to run all of the tests that use that test cluster in parallel. Nevertheless, we cannot have 2 test clusters running tests at the same time, mainly because GrainClient relies on statics, so the tests would conflict with each other.

xUnit by default provides the ability to run tests in different test collections in parallel (effectively run all tests within 1 collection serially, but parallelize with running multiple collections concurrently). This is the exact opposite of what we need, as each of our collections are the ones that set up and tear down the cluster and client. So we need to run each test collection serially, but parallelize within it.

The work to support this test execution is not completely trivial, although there are some samples in xUnit's github repo that show how to make changes to it. I have not made an assestment of how much work that is, but here are the hints I got on where to look at (from a Slack chat with them):

jdom [2:03 PM] is there a way to invert the condition for how to run tests in parallel? I would like to have all the tests in a certain fixture (being that a collection or class fixture) to run in parallel between each other, and once that entire fixture ran, then move to the next collection or class fixture
bradwilson [3:25 PM] Sorry, there's no simple way to do this today (you could define your own unit test framework, but that's not trivial).
jdom [3:30 PM] @bradwilson: ok. By that should I assume that the current extensibility points in xunit are not enough to accomplish this parallelization? I haven't tried yet, but if it is possible then we might give it a shot
bradwilson [3:37 PM] @jdom No, the extensibility points are sufficient, but you would be extending an awful lot of infrastructure to make it happy.
For example, take a look at what this does: https://github.com/xunit/samples.xunit/tree/master/ObservationExample

Another option is that if we could force each collection to be ran in a separate AppDomain, then at that point we could leverage xUnit's built in parallelization, given that each client will not be conflicting with other collections' usages of GrainClient (and its statics).

[amccool]

Is this the true issue re: #467 ?

mainly because GrainClient relies on statics, so the tests would conflict with each other.

but does this imply that pure grain code should execute cleanly in unit tests without a limit on parallelization?

[jdom]

Not sure I understand what you are asking. But yes, assuming a single grain client and silo cluster, most tests should be able to run in parallel, at least the ones that are within a shared fixture.

[veikkoeeva]

In addition to description at #1682 (comment), here's some more explanation on one potential way to parallalize tests (especially on one way to parallelize tests involving silos).

[shlomiw]

@jdom - this one is important. I'm heavily using tests on Orleans with TestClusters, mainly for integration tests. And yeah, it takes long time because currently I can't parallelize them.
Maybe #2779 would help in 1.5, to run few clusters in parallel (and then the internal tests would run sequentially, but still it would be all-in-all faster).
What do you suggest?

Another issue is that the bootstrapping of the TestCluster is quite slow, and it's important mainly when debugging a specific test, then you need to wait long time every time you start the debugging process.
Any suggestions here to make it faster?

Thanks

[jdom]

I believe that now that we have non-static clients, it should be easier to run clusters in parallel, so this might be less of an issue if we can't parallelize the "right way".
I believe there are still some statics that might interfere in some cases, but I'm not really sure. Logging is one of the few remaining pieces that is still static, but will eventually go away with #2814, and shouldn't be a huge concern for testing.

Slowness of startup is an issue that we are aware, and we would like to provide in the future a way to avoid a lot of ad-hoc scanning of assemblies that are in the deployment directory. In the meantime, one thing that does help a little bit is if you use build-time codegen instead of run-time codegen.

[shlomiw]

In the meantime, one thing that does help a little bit is if you use build-time codegen instead of run-time codegen.

@jdom - thanks - this helped to reduce my start time from 20 seconds to 16 seconds. This will speed up a bit the debugging process.

I'll try installing 1.5.0-rc and play with parallel TestClusters, this can help a lot when running full tests process. Will let you know.

[jdom]

Awesome, please do let us know. We briefly tried to run our full set of tests, but we got them to stall, but most likely because some of the tests are using statics themselves, as well as messing with Orleans internals that some still use statics. I'm pretty sure you can get better results when testing application level code instead of Orleans internals.

[jdom]

@sergeybykov pinged me for an update on the current state of this:

Currently there is some parallelization in CI via scripts (but not natively in xUnit, so VS or other tooling doesn't take advantage of it). Basically we run each test assembly in a different process in parallel with a small max degree of parallelization. We might easily increase this value by modifying a single line of powershell, given the scheduling changes in Orleans 2.1, where we are no longer consuming so much CPU in the silo while idle.
Nevertheless, given that we got rid of statics AND that the changes in 2.1 make co-hosting so much friendlier to the CPU, we should probably start using native xUnit parallelization, even if it's not in the ideal way (as mentioned before, ideally we should start 1 cluster and run all tests for that cluster in parallel. Here we would do the opposite: start several clusters in parallel, and run the test for each cluster serially but in parallel with all other clusters).
For this we should probably partition in smaller test collections (instead of ~1 per assembly). Now that startup is so much more configurable via DI, we should also try to start up very streamlined test clusters, that don't do any expensive assembly scanning, but instead we provide the list of grain types (or a single assembly if that's too complex) and the least amount of generic services possible. This way we can reduce the startup time of a cluster to some non-concerning amount of (milli)seconds (maybe that already happens in our test, haven't checked in a while, but it used to take more than 10 seconds for a test cluster to be ready for tests to run).

[sergeybykov]

Thank you, @jdom. Sounds like there's still quite a bit of work left here. 😊

[ReubenBond]

We have some level of parallelism now, via scripts as mentioned above, but also via Xunit's internal parallelization. I believe we could further parallelize, but I'll close this now as it's not really actionable anymore.