Provide an API or annotation to specify a class to use to order the test methods

[kretes]

When adding this annotation to a test super class -> it is not taken into account

[marcphilipp]

Sounds like a bug, thanks for reporting. Can you provide a test case or some example code?

[christian-king]

It's simply in org.junit.internal.MethodSorter:52, it's only looking at the current class:

Comparator<Method> comparator = getSorter(clazz.getAnnotation(FixMethodOrder.class));

I think it definitely should be looking at the superclasses.

[dsaff]

I don't really want to enable reading FixMethodOrder from a superclass. In general, test classes shouldn't expect any kind of ordering between their test methods, and if a particular order is necessary, I'd like the annotation to be in the code-reader's face if at all possible.

Open to counter-arguments.

[kretes]

test class shouldn't expect any kind of ordering - that's right - still I would like to be able to have the same behaviour i had for junit 4.10, as this has for example an advantage of having the same ordering in source file and in execution order, which makes it easier to grasp, which test cases failed, as they are written next to each other.

I know the order of jvm methods is not guaranteed, but it works for many environments predictably.

My original problem was the one reported in #560 .
Since this was not possible i tried with second option of having a unified behaviour across the project, which could be 'from the superclass'.

[dsaff]

@kretes, unfortunately (for people who liked the accidental "feature" in JUnit), the number of JVM installations for which there will be any way via reflection to figure out source code order appears doomed to shrink over time.

[bmuskalla]

@dsaff, completely agree with you that tests should not ever rely on method order. But given there are people out there (I'd imagine mostly during migration scenarios) need to use the FixMethodOrder annotation, it would really help a lot to enable in centrally and in the same way, disable it again with a single line of code.

[dsaff]

@bmuskalla, that only works if every test class in the package already inherits from the same base class.

Over at #635, we're discussing adding a --filter command-line option to enable global control of which classes get run. I would be fine taking the same path and adding a --sorter command-line option that could be used for these kinds of purposes, and many others. What do you think?

[noel-yap]

Whatever solution is implemented, I think it ought to be able to handle randomizing the order of the tests.

[odupuy]

As per my comment in #633, yes tests should not depend on the order but legacy applications do exist and they are our bread and butter. Adding a fixed order helps in Eclipse (same order for each execution) and externally in Ant, maven...
+1 to have the annotation to be inherited (not the case currently), this will avoid me and others to fix hundreds of classes instead of fixing a few abstract test cases inherited by the many concrete tests. Of course, if you set the annotation in a subclass, it should override the one coming from the parent class.
+1 too to have an option for a REALLY random test order @FixMethodOrder(MethodSorters.RANDOM) as this will greatly help to detect these non repeatable tests.

[RainerW]

I agree, tests should never expect any order.

BUT: Tester do!!!

By default i arrange tests inside a testclass by an logical group, and let their names begin with the method as prefix. But 4.11 MethodSorters.DEFAULT just creates the worst imaginable order:

@FixMethodOrder has to respected the inheritence like @RunWith. This ticket should therefore be fixed regardless of what you discuss in issue #xyz.

[kcooney]

I changed the title this bug, since the previous title suggested a specific implementation that we decided not to use.

If someone wants to take a stab at either a sorting annotation or a sorting flag, let us know.

[agustisanchez]

test class shouldn't expect any kind of ordering

In general, that may be right, but in some cases it's not practical at all.
For instance, when testing CRUD operations of persistent entities.

It's perfectly OK for me to take advantage of the side effect of the creation test (the persisted entity) to test the retrieval of it, then test the creation of a duplicate, then test an update of the persisted entity and finally testing the deletion of it. I even define an initial pseudo-test as a test fixture.

So my test classes for entity CRUD typically look like:

   void test00_populateDependentObjects();
   void test10_createEntity();
   void test20_retrieveEntity();
   void test30_createDuplicateEntity(); // Expect exception
   void test40_updateEntity(); //Test update timestamp among other things
   void test90_deleteEntity();

I'm perfectly aware of the good practice stating that unit tests should be independent of each other and should not have side effects but, in this particular scenario, that would imply creating a new sample entity for every single test increasing the overall execution time for the test harness for no benefit. To me, the entire test CRUD class is the unit test and I see no point in testing the retrieval case if the creation case has failed in the first place.

So I'm a big fan of the @FixMethodOrder annotation and I would certainly appreciate it if it was made inheritable.

In general, I don't think API designers should bulldoze good practices on to the API users. "Good practices" are scenario-dependent. Besides, what is regarded as good or bad practice tends to change over time.
Applying good practices should be mainly the responsibility of the API user, the API should allow for some flexibility.

I also mentioned that I use a pseudo-test for test fixtures. That's because the @BeforeClass annotation can only be applied to static class methods. That may be a perfectly ok good practice in the times of core Java, but nowadays initialization is performed thru DI frameworks that don't mix well with static methods and prefer instance methods. The result is that I have to resort to tricks or to frameworks outside JUnit for test initialization.

[dsaff]

I'm not against in general allowing a facility for requesting test ordering from source code. (The full quote above was: "In general, test classes shouldn't expect any kind of ordering between their test methods, and if a particular order is necessary, I'd like the annotation to be in the code-reader's face if at all possible.")

However, I'd like to see it using the general Sorter mechanism already baked into JUnit, and not FixMethodOrder, which was a hack designed for a particular quirk in the transition from JUnit 1.6 to 1.7 (if memory serves). Sorter allows general-purpose logic in computing ordering. FixMethodOrder requires code changes to core JUnit for each new ordering someone wants to add.

[kcooney]

I sent out pull #1130 that adds a @SortWith annotation.