Automatic discovery of @Category annotations

[mattinger]

It would be great if I did not have to specifically enumerate all the test classes in a test suite in order to run a specific category. It seems counter intuitive. The only use that @category really has at that point is at the method level, not at the class level.

I came up with a spring based solution (doesn't really require much of spring, but it uses some spring utility classes, in particular ClassPathScanningCandidateComponentProvider). Perhaps something similar could be implemented in JUnit. The downside to this of course is that the @category annotation (or some other annotation) has to be at the class level in order for the class to be found. That being said, we might be able to leave out the filter on the scanning provider, though you run the risk of pulling in too much if you do that.

@RunWith(DiscoverCategories.class)
@IncludeCategory(Foo.class)
@ScanPackage
public class MyTestClass { ... }

And in DiscoverCategories class it's relatively easy to find all the classes in the classpath with the @category annotation:

/***
 * Scan for classes for a given test suite.  This will use the {@link ScanPackage}
 * annotation to look for classes with the {@link Category} annotation.
 * 
 * @param suiteClass
 * @return
 * @throws InitializationError
 */
private static Class<?>[] scanClassesWithCategoryAnnotation(Class<?> suiteClass) 
    throws InitializationError {

    String packages[] = null;
    if (suiteClass.isAnnotationPresent(ScanPackage.class)) {
        packages = suiteClass.getAnnotation(ScanPackage.class).value();
    }

    return scanClassesWithCategoryAnnotation(packages);
}

    /***
 * Scan for classes for a given set of packages.  This will look for classes
 * with the {@link Category} annotation.
 * 
 * @param suiteClass
 * @return
 * @throws InitializationError
 */
private static Class<?>[] scanClassesWithCategoryAnnotation(String packages[]) 
    throws InitializationError {

    ClassPathScanningCandidateComponentProvider provider =
        new ClassPathScanningCandidateComponentProvider(false);
    provider.addIncludeFilter(new AnnotationTypeFilter(Category.class));

    Set<BeanDefinition> allDefs = new HashSet<BeanDefinition>();


    if (packages != null) {
        for (String pack : packages) {
            Set<BeanDefinition> defs = provider.findCandidateComponents(pack);
            allDefs.addAll(defs);
        }
    }

    Set<Class<?>> klasses = new HashSet<Class<?>>();        
    for (BeanDefinition def : allDefs) {
        String beanClassName = def.getBeanClassName();
        try {
            Class<?> klass = Thread.currentThread().getContextClassLoader()
                .loadClass(beanClassName);
            klasses.add(klass);
        }
        catch (ClassNotFoundException e) {
            throw new InitializationError(e);
        }
    }

    return klasses.toArray(new Class<?>[klasses.size()]);
}

/***
 * Public constructor for a test suite class.
 * 
 * @param klass
 * @param builder
 * @throws InitializationError
 */
public DiscoverCategories(Class<?> klass, RunnerBuilder builder)
        throws InitializationError {
    super(builder, klass, scanClassesWithCategoryAnnotation(klass));
    try {
        filter(getCategoryFilter(klass));
    }
    catch (NoTestsRemainException e) {
        throw new InitializationError(e);
    }
}

[mattinger]

FYI: After testing, the annotation filter cannot be left out of spring's scanning provider. If it's left out the scanner doesn't find any classes.

[kcooney]

Have you seen ClasspathSuite? http://johanneslink.net/projects/cpsuite.jsp

[kcooney]

If you still wanted to use JUnit4-style categories, you could do:

import org.junit.extensions.cpsuite.ClasspathSuite.*;

@ClassnameFilters({ "!.*Tests" })
public class AllTests {}

And then do:

@RunWith(Categories.class)
@IncludeCategory(Fast.class)
@SuiteClasses({ AllTests.class })
public class FastTests {}

This assumes that your suite classes end in "Tests" ("FastTests", "SmokeTests", etc) while your other tests end in something else ("ServerTest", etc)

[mattinger]

Yes, we want to use categories. We also have a load of legacy tests that we don't want to rename. I'm aware of ClasspathSuite, but it really didn't meet our news for those reasons.

[mattinger]

I have tried your solution and found that it would probably work for us:

@ClassnameFilters({".*Test"})
@IncludeJars(false)
@RunWith(ClasspathSuite.class)
public class AllClassesSuite {

}

However, the one issue I saw was where the test suite classes lived. I wanted to make a seperate module to define the different categories:

@category(FastTestCategory.class)
@category(SlowTestCategory.class)

And then create a shared test suite for running them:

@RunWith(Categories.class)
@IncludeCategory(FastTestCategory.class)
@SuiteClasses({AllClassesSuite.class})
public class FastTestSuite {

}

The problem I encountered was that if FastTestSuite was brought into the module via a maven dependency, and then my individual tests were in the current module, no tests were run when I tried to execute that suite.

I'm not sure if this is a classloading issue or just a maven issue. If i try to run any suite not immediately in the current module, no tests get executed. But if i move that suite to the current module it's fine.

[kcooney]

I suggest asking on the mailing list: http://tech.groups.yahoo.com/group/junit/

My guess is if you have tests in a different module, then they will be in a jar, so will get filtered out by @IncludeJars(false), but that's just a guess.

The general idea is you can use a reflection-based suite builder to find all of your tests, and then use Categories to select which ones to run.

[mattinger]

I tried changing includeJars to true, with no luck.

I think the problem is that the suite is in a jar file, and the tests are not. So the surefire plugin is getting confused due to it's classloader. It works fine in eclipse.

[mattinger]

I was hoping to have the suite in a single place, the groups support in junit is kind of tacked on after the fact, and isn't truly native to junit. I'd love just to be able to say: mvn test -Djunit.groups=FastTest

and be done with it, but alas, i have to create classes to represent the stuff, and create a suite to run it. As a result, in order to be able to just run the FastGroup group for a multi module project, I have to replicate the same suite in every module, and it would have to have the exact same classname. Then i run:

mvn test -Dtest=com.myorg.FastTest

Just a maintenance headache.

[kcooney]

These sound like mvn issues. Have you tried asking on the mailing list?

[Tibor17]

Hi all,

i created an issue with an explanation and a solution.
i suggest the JUnit team to request the source code in order to open a discussion and integrate my changes by them or me (does not really matter who will make the submit).

https://github.com/KentBeck/junit/issues/307

@mattinger
Please have a look in to the example.
You may comment it anyway if it would solve your problem.

[dsaff]

@mattinger, some things are getting fixed with respect to Category support, although we still don't have an internal answer to ClasspathSuite. Is this a continuing issue for you?

[kvanrobbroeck]

Hello,

I too would very much like some automated discovery of categories of some sort in the core JUnit framework.

Other solutions such as the wildcard based classpath scanning are available, but none seem to be part of JUnit itself. I would also prefer a solution that uses annotation metadata (such as these categories), or maybe both would be interesting.

I came across this issue while examining the JUnit categories but I find it inconvenient that one must always enumerate each class using the @SuiteClasses annotation, because this is easily forgotten.

[marcphilipp]

I've started working on integrating the best parts from @dhemery's runtime-suite and @jlink's ClasspathSuite a while ago. Maybe that would be something for 4.13?

[dsaff]

@marcphilipp, that could certainly be useful.

[jlink]

@marc Need help? We could do a session at Socrates...
Johannes

Von meinem iPad gesendet

Am 31.07.2014 um 21:22 schrieb Marc Philipp [email protected]:

I've started working on integrating the best parts from @dhemery's runtime-suite and @jlink's ClasspathSuite a while ago. Maybe that would be something for 4.13?

—
Reply to this email directly or view it on GitHub.

[dhemery]

I regret that I don't have the energy to participate in this right now. But I'll be following along.

[marcphilipp]

@jlink We can take a look at it there for sure. Now I just have to
dig through my workspaces and find it… ;-)