Skip to content

Commit

Permalink
Introduce support for ordering the sequence of test methods
Browse files Browse the repository at this point in the history
These commits introduce new support in JUnit Jupiter for ordering the
sequence of test methods.

Specifically, these commits introduce a new MethodOrderer API that can
be registered via a new @TestMethodOrder annotation.

JUnit Jupiter provides the following built-in implementations of this
new API.

- Alphanumeric: sorts test methods alphanumerically based on their
  names and formal parameter lists.
- OrderAnnotation: sorts test methods numerically based on values
  specified via the @order annotation.
- Random: orders test methods pseudo-randomly and supports
  configuration of a custom seed.

Consult the new "Test Execution Order" section of the User Guide for
details and example usage.

Issue: #13
  • Loading branch information
sbrannen committed Nov 9, 2018
2 parents d8c4c90 + 7c1f77a commit 5300e65
Show file tree
Hide file tree
Showing 17 changed files with 1,311 additions and 9 deletions.
6 changes: 6 additions & 0 deletions documentation/src/docs/asciidoc/link-attributes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ endif::[]
:AfterAllCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterAllCallback.html[AfterAllCallback]
:AfterEachCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterEachCallback.html[AfterEachCallback]
:AfterTestExecutionCallback: {javadoc-root}/org/junit/jupiter/api/extension/AfterTestExecutionCallback.html[AfterTestExecutionCallback]
:Alphanumeric: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.Alphanumeric.html[Alphanumeric]
:ArgumentsAccessor: {javadoc-root}/org/junit/jupiter/params/aggregator/ArgumentsAccessor.html[ArgumentsAccessor]
:ArgumentsAggregator: {javadoc-root}/org/junit/jupiter/params/aggregator/ArgumentsAggregator.html[ArgumentsAggregator]
:Assertions: {javadoc-root}/org/junit/jupiter/api/Assertions.html[org.junit.jupiter.api.Assertions]
Expand All @@ -62,10 +63,14 @@ endif::[]
:ExtendWith: {javadoc-root}/org/junit/jupiter/api/extension/ExtendWith.html[@ExtendWith]
:ExtensionContext: {javadoc-root}/org/junit/jupiter/api/extension/ExtensionContext.html[ExtensionContext]
:ExtensionContext_Store: {javadoc-root}/org/junit/jupiter/api/extension/ExtensionContext.Store.html[Store]
:MethodOrderer: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.html[MethodOrderer]
:MethodSource: {javadoc-root}/org/junit/jupiter/params/provider/MethodSource.html[@MethodSource]
:Order: {javadoc-root}/org/junit/jupiter/api/Order.html[@Order]
:OrderAnnotation: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.OrderAnnotation.html[OrderAnnotation]
:ParameterContext: {javadoc-root}/org/junit/jupiter/api/extension/ParameterContext.html[ParameterContext]
:ParameterizedTest: {javadoc-root}/org/junit/jupiter/params/ParameterizedTest.html[@ParameterizedTest]
:ParameterResolver: {javadoc-root}/org/junit/jupiter/api/extension/ParameterResolver.html[ParameterResolver]
:Random: {javadoc-root}/org/junit/jupiter/api/MethodOrderer.Random.html[Random]
:RegisterExtension: {javadoc-root}/org/junit/jupiter/api/extension/RegisterExtension.html[@RegisterExtension]
:RepetitionInfo: {javadoc-root}/org/junit/jupiter/api/RepetitionInfo.html[RepetitionInfo]
:ResourceLock: {javadoc-root}/org/junit/jupiter/api/parallel/ResourceLock.html[@ResourceLock]
Expand All @@ -74,6 +79,7 @@ endif::[]
:TestInfo: {javadoc-root}/org/junit/jupiter/api/TestInfo.html[TestInfo]
:TestInstanceFactory: {javadoc-root}/org/junit/jupiter/api/extension/TestInstanceFactory.html[TestInstanceFactory]
:TestInstancePostProcessor: {javadoc-root}/org/junit/jupiter/api/extension/TestInstancePostProcessor.html[TestInstancePostProcessor]
:TestMethodOrder: {javadoc-root}/org/junit/jupiter/api/TestMethodOrder.html[@TestMethodOrder]
:TestReporter: {javadoc-root}/org/junit/jupiter/api/TestReporter.html[TestReporter]
:TestTemplate: {javadoc-root}/org/junit/jupiter/api/TestTemplate.html[@TestTemplate]
:TestTemplateInvocationContext: {javadoc-root}/org/junit/jupiter/api/extension/TestTemplateInvocationContext.html[TestTemplateInvocationContext]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ repository on GitHub.
* New `LOCALE` and `TIME_ZONE` constants in `org.junit.jupiter.api.parallel.Resources`
for use with `@ResourceLock` to synchronize test execution regarding the default
`Locale` and default `TimeZone`, respectively.
* New `MethodOrderer` API for ordering the sequence of tests with built-in support for
_alphanumeric_, `@Order` annotation based, and _random_ ordering of test methods.
- See <<../user-guide/index.adoc#writing-tests-test-execution-order, Test Execution
Order>> in the User Guide for details.
* New `DisplayNameGenerator` interface and `@DisplayNameGeneration` annotation that allow
declarative configuration of a pre-defined or custom display name generator.
* JUnit 4's `@Ignore` annotation is now supported for disabling test classes and test
Expand Down
40 changes: 39 additions & 1 deletion documentation/src/docs/asciidoc/user-guide/writing-tests.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,9 @@ module.
| `@ParameterizedTest` | Denotes that a method is a <<writing-tests-parameterized-tests, parameterized test>>. Such methods are _inherited_ unless they are _overridden_.
| `@RepeatedTest` | Denotes that a method is a test template for a <<writing-tests-repeated-tests, repeated test>>. Such methods are _inherited_ unless they are _overridden_.
| `@TestFactory` | Denotes that a method is a test factory for <<writing-tests-dynamic-tests, dynamic tests>>. Such methods are _inherited_ unless they are _overridden_.
| `@TestInstance` | Used to configure the <<writing-tests-test-instance-lifecycle, test instance lifecycle>> for the annotated test class. Such annotations are _inherited_.
| `@TestTemplate` | Denotes that a method is a <<writing-tests-test-templates, template for test cases>> designed to be invoked multiple times depending on the number of invocation contexts returned by the registered <<extensions-test-templates, providers>>. Such methods are _inherited_ unless they are _overridden_.
| `@TestMethodOrder` | Used to configure the <<writing-tests-test-execution-order, test method execution order>> for the annotated test class; similar to JUnit 4's `@FixMethodOrder`. Such annotations are _inherited_.
| `@TestInstance` | Used to configure the <<writing-tests-test-instance-lifecycle, test instance lifecycle>> for the annotated test class. Such annotations are _inherited_.
| `@DisplayName` | Declares a custom display name for the test class or test method. Such annotations are not _inherited_.
| `@DisplayNameGeneration` | Declares a custom display name generator for the test class. Such annotations are _inherited_.
| `@BeforeEach` | Denotes that the annotated method should be executed _before_ *each* `@Test`, `@RepeatedTest`, `@ParameterizedTest`, or `@TestFactory` method in the current class; analogous to JUnit 4's `@Before`. Such methods are _inherited_ unless they are _overridden_.
Expand Down Expand Up @@ -363,6 +364,43 @@ characters have been removed.
include::{testDir}/example/TaggingDemo.java[tags=user_guide]
----

[[writing-tests-test-execution-order]]
=== Test Execution Order

By default, test methods will be ordered using a default algorithm that is deterministic
but intentionally nonobvious. This ensures that subsequent runs of a test suite execute
test methods in the same order, thereby allowing for repeatable builds.

NOTE: In this context, a _test method_ is any instance method that is directly or
meta-annotated with `@Test`, `@RepeatedTest`, `@ParameterizedTest`, `@TestFactory`, or
`@TestTemplate`.

Although true _unit tests_ typically should not rely on the order in which they are
executed, there are times when it is necessary to enforce a specific test method
execution order -- for example, when writing _integration tests_ or _functional tests_
where the sequence of the tests is important, especially in conjunction
`@TestInstance(Lifecycle.PER_CLASS)`.

To control the order in which test methods are executed, annotate your test class or test
interface with `{TestMethodOrder}` and specify the desired `{MethodOrderer}`
implementation. You can implement your own custom `MethodOrderer` or use one of the
following built-in `MethodOrderer` implementations.

* `{Alphanumeric}`: sorts test methods _alphanumerically_ based on their names and formal
parameter lists.
* `{OrderAnnotation}`: sorts test methods _numerically_ based on values specified via the
`{Order}` annotation.
* `{Random}`: orders test methods _pseudo-randomly_ and supports configuration of a
custom _seed_.

The following example demonstrates how to guarantee that test methods are executed in the
order specified via the `@Order` annotation.

[source,java,indent=0]
----
include::{testDir}/example/OrderedTestsDemo.java[tags=user_guide]
----

[[writing-tests-test-instance-lifecycle]]
=== Test Instance Lifecycle

Expand Down
37 changes: 37 additions & 0 deletions documentation/src/test/java/example/OrderedTestsDemo.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Copyright 2015-2018 the original author or authors.
*
* All rights reserved. This program and the accompanying materials are
* made available under the terms of the Eclipse Public License v2.0 which
* accompanies this distribution and is available at
*
* http://www.eclipse.org/legal/epl-v20.html
*/

package example;

// tag::user_guide[]
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(OrderAnnotation.class)
class OrderedTestsDemo {

@Test
@Order(1)
void nullValues() {
}

@Test
@Order(2)
void emptyValues() {
}

@Test
@Order(3)
void validValues() {
}
}
// end::user_guide[]
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/*
* Copyright 2015-2018 the original author or authors.
*
* All rights reserved. This program and the accompanying materials are
* made available under the terms of the Eclipse Public License v2.0 which
* accompanies this distribution and is available at
*
* http://www.eclipse.org/legal/epl-v20.html
*/

package org.junit.jupiter.api;

import static org.apiguardian.api.API.Status.EXPERIMENTAL;

import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.util.List;
import java.util.Optional;

import org.apiguardian.api.API;

/**
* {@link MethodDescriptor} encapsulates functionality for a given {@link Method}.
*
* @since 5.4
* @see MethodOrdererContext
*/
@API(status = EXPERIMENTAL, since = "5.4")
public interface MethodDescriptor {

/**
* Get the method for this descriptor.
*
* @return the method; never {@code null}
*/
Method getMethod();

/**
* Determine if an annotation of {@code annotationType} is either
* <em>present</em> or <em>meta-present</em> on the {@link Method} for
* this descriptor.
*
* @param annotationType the annotation type to search for; never {@code null}
* @return {@code true} if the annotation is present or meta-present
* @see #findAnnotation(Class)
* @see #findRepeatableAnnotations(Class)
*/
boolean isAnnotated(Class<? extends Annotation> annotationType);

/**
* Find the first annotation of {@code annotationType} that is either
* <em>present</em> or <em>meta-present</em> on the {@link Method} for
* this descriptor.
*
* @param <A> the annotation type
* @param annotationType the annotation type to search for; never {@code null}
* @return an {@code Optional} containing the annotation; never {@code null} but
* potentially empty
* @see #isAnnotated(Class)
* @see #findRepeatableAnnotations(Class)
*/
<A extends Annotation> Optional<A> findAnnotation(Class<A> annotationType);

/**
* Find all <em>repeatable</em> {@linkplain Annotation annotations} of
* {@code annotationType} that are either <em>present</em> or
* <em>meta-present</em> on the {@link Method} for this descriptor.
*
* @param <A> the annotation type
* @param annotationType the repeatable annotation type to search for; never
* {@code null}
* @return the list of all such annotations found; neither {@code null} nor
* mutable, but potentially empty
* @see #isAnnotated(Class)
* @see #findAnnotation(Class)
* @see java.lang.annotation.Repeatable
*/
<A extends Annotation> List<A> findRepeatableAnnotations(Class<A> annotationType);

}
Loading

0 comments on commit 5300e65

Please sign in to comment.