v4 output results in NullPointerException when used in Jenkins

[bgotink]

Caveat: I don't know the junit.xml format at all, so my apologies if this is a bug in Jenkins rather than tap-junit. I opened this ticket because it works when using tap-junit version 3.

Describe the bug

Jenkins builds fail with a NullPointerException when passing output of tap-junit version 4 to the junit plugin.

This is caused by the lack of a name attribute on the testsuite element. Jenkins falls back to that attribute if no classname attribute is present on a testcase (source).

To Reproduce

1. Use Jenkins
2. Have a build output an XML file through tap-junit
3. Run that XML file through Jenkins's junit plugin

17:53:48  Recording test results
17:53:48  null
Error when executing always post condition:
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 1.2.3.4/1.2.3.4:56789
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at hudson.FilePath.act(FilePath.java:1070)
		at hudson.FilePath.act(FilePath.java:1059)
		at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:107)
		at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:149)
		at hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:243)
		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.NullPointerException
	at hudson.tasks.junit.CaseResult.getPackageName(CaseResult.java:399)
	at hudson.tasks.junit.TestResult.tally(TestResult.java:795)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:145)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3112)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:375)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:118)
	at java.lang.Thread.run(Thread.java:748)

Expected behavior

The XML file is parsed by Jenkins

[dhershman1]

This sounds like it might be something on my end, Do you mind posting the command line you pass to run tap-junit?

And if possible 1 or 2 of your tests tap output if you could and I can get a working example on my end. I think it's literally just a naming issue if I'm not putting the name in right then that's something I need to correct right away, or maybe its the arguments being passed in!

[bgotink]

Sure, though I had to redact parts. I just realised we control the TAP output, so it could be we made a mistake there that accidentally worked with tap-junit version 3.

A tap file:

TAP version 13
# @scope/package
ok 1 - Package @scope/package is using registered runtime dependency @angular/core
ok 2 - Package @scope/package is using registered runtime dependency @angular/cdk
ok 3 - Package @scope/package is using registered runtime dependency rxjs
ok 4 - Package @scope/package's dependency @angular/cdk is used
ok 5 - Package @scope/package's dependency @angular/core is used
ok 6 - Package @scope/package's dependency rxjs is used
1..6

The command:

tap-junit -o results -n dependencies.xml <results/dependencies.tap

The output with tap-junit 3:

<?xml version="1.0"?>
<testsuites tests="6" name="Tap-Junit" failures="0" errors="0">
  <testsuite tests="6" failures="0" errors="0" name="@scope/package">
    <testcase name="#1 Package @scope/package is using registered runtime dependency @angular/core"/>
    <testcase name="#2 Package @scope/package is using registered runtime dependency @angular/cdk"/>
    <testcase name="#3 Package @scope/package is using registered runtime dependency rxjs"/>
    <testcase name="#4 Package @scope/package's dependency @angular/cdk is used"/>
    <testcase name="#5 Package @scope/package's dependency @angular/core is used"/>
    <testcase name="#6 Package @scope/package's dependency rxjs is used"/>
  </testsuite>
</testsuites>

The output with tap-junit 4 (pretty-printed):

<?xml version="1.0"?>
<testsuites tests="6" name="Tap-Junit" failures="0">
  <testsuite tests="6" skipped="0" failures="0">
    <testcase name="Package @scope/package is using registered runtime dependency @angular/core" id="1"># @scope/package
</testcase>
    <testcase name="Package @scope/package is using registered runtime dependency @angular/cdk" id="2"></testcase>
    <testcase name="Package @scope/package is using registered runtime dependency rxjs" id="3"></testcase>
    <testcase name="Package @scope/package's dependency @angular/cdk is used" id="4"></testcase>
    <testcase name="Package @scope/package's dependency @angular/core is used" id="5"></testcase>
    <testcase name="Package @scope/package's dependency rxjs is used" id="6"></testcase>
  </testsuite>
</testsuites>

[dhershman1]

It looks like maybe your plugin is expecting a name on testsuite, though I didn't see that in the junit specs, I can probably add it without it affecting much. It won't use your # @scope/package as the name most likely mostly because the new parser counts that as a comment.

This is why it shows up in the first testcase as text. The parser takes it in as a tap comment and thinks it applies to the test that comes next.

The sad part is I can't make out anything to pull that comment as a name instead of just a comment. So we may need to do something fancy there.

[dhershman1]

I released v4.1.0, try using the new -c argument in your command line and see if that fixes your issue. If it doesn't please don't hesitate to reopen this