Generate reports when tests fail? #61

cbaldan · 2017-03-15T16:12:09Z

Hello.

I'm trying to improve my project setup and I couldn't find the answer in the documentation.
We do not use Jenkins, so that's not an option for me.
Also, I want the reports generated every time the maven test target is invoked.

So, this is how my plugin is setup:

<build>
	<plugins>
		<plugin>
			<groupId>net.masterthought</groupId>
			<artifactId>maven-cucumber-reporting</artifactId>
			<version>3.5.0</version>
			<executions>
				<execution>
					<id>cucumber-report</id>
					<phase>test</phase>
					<goals>
						<goal>generate</goal>
					</goals>
					<configuration>
						<projectName>${project.name}</projectName>
						<outputDirectory>${project.build.directory}</outputDirectory>
					</configuration>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

When there are no failures, the plugin is executed as expected:

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-cucumber-reporting:3.5.0:generate (cucumber-report) @ xxxxxxxxxxxxxxxxxxxxxxxx ---
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
[INFO] About to generate Cucumber report.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] XXXXXXXXXXXXXXXXXXXXXX ............................ SUCCESS [ 0.049 s]
[INFO] XXXXXXXXXXXXXXXXXXXXXXXXXXXXX ..................... SUCCESS [ 14.574 s]
[INFO] XXXXXXXXXXXXXXXXXXXXXXXXXXXXX - Test Automation ... SUCCESS [ 31.654 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

But when a test does fails, the report generation is skipped:

Tests run: 6, Failures: 2, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] XXXXXXXXXXXXXXXXXXXXXX ............................ SUCCESS [ 0.829 s]
[INFO] XXXXXXXXXXXXXXXXXXXXXXXXXXXXX ..................... SUCCESS [ 18.122 s]
[INFO] XXXXXXXXXXXXXXXXXXXXXXXXXXXXX - Test Automation ... FAILURE [ 35.968 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

I do want reports generated when there are test failures, the way I currently make it happen is setting this property in my pom:

<maven.test.failure.ignore>true</maven.test.failure.ignore>

Then the report does get generated, but the Test Automation suite is reported as SUCCESS:

Tests run: 6, Failures: 2, Errors: 0, Skipped: 0

[ERROR] There are test failures.

Please refer to C:\xxxxxxxxxxxxxxxxxxxxxxxx\test-automation\target\surefire-reports for the individual test results.
[INFO]
[INFO] --- maven-cucumber-reporting:3.4.0:generate (cucumber-report) @ xxxxxxxxxxxxxxxxxxxxxxxx ---
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
[INFO] About to generate Cucumber report.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] xxxxxxxxxxxxxxxxxxxxxxxx ............................ SUCCESS [ 0.004 s]
[INFO] xxxxxxxxxxxxxxxxxxxxxxxx ..................... SUCCESS [ 9.232 s]
[INFO] xxxxxxxxxxxxxxxxxxxxxxxx - Test Automation ... SUCCESS [ 27.349 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Am I missing something? Is there a better way to configure the project and get reports even when tests fail?

Thanks,
Cleber.

The text was updated successfully, but these errors were encountered:

damianszczepanik · 2017-03-16T21:19:46Z

This is related how you run mvn not the report itself. Try to process the build execution even test fails.

cbaldan · 2017-03-16T21:41:50Z

Thanks Damian, but I don't know how.
Do you mean the way I invoke mvn in the command line?

I currently call:
mvn clean test -Ptest-automation

What would be a better way?

cbaldan · 2017-03-28T14:15:46Z

Hi Damian.
I can't figure this one out on my own and you are the most knowledgeable person about this.
Would you be so kind to shed some light for me?

Thanks,
Cleber.

shohilbagdai · 2017-05-26T10:50:58Z

Hi cbaldan.
I am facing same issue and can not figure it out how to solve it, are you facing same issue or u found solution?

Thanks,
Shohil

cbaldan · 2017-05-26T13:23:12Z

Nope, still using the test failure ignore property.

shohilbagdai · 2017-05-26T13:50:23Z

Thanks for reply, but in my case I m going to use that in Jenkins so if builds success (with failed test) it will trigger next job, which is no good for me From: Cleber Baldan Junior [mailto:[email protected]] Sent: 26 May 2017 14:23 To: damianszczepanik/maven-cucumber-reporting <[email protected]> Cc: Shohil Bagdai ; Comment <[email protected]> Subject: Re: [damianszczepanik/maven-cucumber-reporting] Generate reports when tests fail? (#61) Nope, still using the test failure ignore property. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#61 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AWdlWa8ohUm0SfTEKm9NyxHNYRL4I69kks5r9tJAgaJpZM4MeK0O>.

ptrthomas · 2017-06-24T05:24:28Z

This is certainly an issue and after the last release of Karate where we recommended this plugin, we are getting reports of users running into this issue.

Until this is fixed, for the benefit of others on this thread, here is one possible workaround.

don't use this maven plugin, use a normal surefire config (without the test failure ignore setting), and add the net.masterthought:cucumber-reporting jar as a maven test dependency
introduce a minimal log4j2.properties to keep the cucumber-reporting code happy
with less than 10 lines of code, you can programmatically generate this report

For details, look at this commit: karatelabs/karate@c5968d1

damianszczepanik · 2017-06-24T15:26:27Z

@ptrthomas would you like to prepare pull request for this?

ptrthomas · 2017-06-24T16:40:21Z

@damianszczepanik the solution above is a hack or workaround which is not the right fix for this maven-plugin.

I would like to help here, just that right now I'm busy on Karate and I suspect I will need to dig into the maven life-cycle to understand what the right fix is for this plug-in. any tips from anyone with better ideas would be great.

KLBonn · 2017-08-29T15:06:46Z

I have a similar challenge as shohilbagdai.
We would like to break our pipeline in case of failed tests, but AFTER having generated the beautiful reports (which include essential information to investigate those failed tests, after all)!

Unfortunately, if we include <testFailureIgnore>true</testFailureIgnore> in maven-surefire-plugin config (in order to let report generation happen), then test failures will never be "communicated" back to the pipeline.

For me, the maven-cucumber-reporting would greatly benefit from a "build-breaker" flag/logic/step/whatever.

[Update: Restore XML tags]

kprpalash82 · 2017-08-29T15:38:17Z

Hi, Did anyone found a solution for this. Currently i am also facing the same issue where if a test case has failed the report is not generated and if i add < testFailureIgnore>true< /testFailureIgnore> to Surefire it generates report but build is also success.
The above workaround will too much of an effort for us.

pnakhat · 2017-09-19T15:42:54Z

Same issue. Any solutions ?

damianszczepanik · 2017-09-23T05:39:00Z

If I'm right this is the maven feature not the issue of this project. Think this should help https://stackoverflow.com/questions/1923857/is-there-a-way-to-fail-fast-for-junit-with-the-maven-surefire-plugin

cbaldan · 2017-09-23T22:51:58Z

I can't see how making Maven fail fast helps us get the reports generated.
Our need here is different, because the nature of the tests we are executing is different then the guy in stackoverflow. Cucumber tests are (usually) very high level, whereas unit tests are the most low level tests there are.

I spent some time thinking on this matter and thought of some possible ways to go:

Make the Cucumber Reporting project be executed by Cucumber itself
Make this plugin throw an error when it detects test failures
Ask/Make a PR for Maven to have a new stage for test report generation
Ask/Make a PR for Maven to have a new property to run a plugin even after a test failure

1) Perhaps the most elegant solution would be finding out if Cucumber has a plugin architecture and allows using of an external tool to generate the reports. This would be straight forward, keep a clean and organized architecture but not sure they do have such functionality. I'll investigate in the next days.
Also, if they do have, not sure how we would connect these 2 Maven plugins in runtime. Maybe with some tinkering we can think of a way to make this happen.

2) If I'm not mistaken, older versions of Cucumber-reporting would throw an error when it found out that there were test erros/failures. So what I did back then was: set <testFailureIgnore>true</testFailureIgnore> and rely that this plugin would thrown an exception after all tests were run and the report generated so that the build fails. But I don't recall this behavior in the latest versions.
(I think it's the code reported at issue #45 is the that should break the build)

The last 2 options are a bigger stretch, but they sound reasonable to me, and I think it would improve a lot Maven support for testomg needs.

3) Add a specific phase in Maven's lifecycle to generate test reports, which by default would be executed regardless of the tests result - success or failure.
AFAIK they don't have such phase, and I'm very sure the community would be very hesitating to create a new phase in the lifecycle.

4) Create a new property in Maven, specific to allow any other plugin to be run after the test phase.
I imagine this should be explicitly configured in the plugin <configuration> section, and could be something like <executeRegardlessOfTestFailure>, or <executeIfTests>[passed|failed|passedOrFailed]</executeIfTests>. I think this would be very useful, not only for test report generation, but would open a whole new world of possibilities.

Let's see, I'll get active in this in the next days.

Regards,
Cleber.

cbaldan · 2017-09-23T23:28:51Z

Perhpas Maven already has support for this.

http://maven.apache.org/surefire/maven-failsafe-plugin/
Maven won't fail the build if the tests fail during integration-test phase, still calling post-integration-test phase, and finally, the verify phase will break the build.

So we should make a test. Bind Cucumber-JVM to integration-test and Maven-cucumber-reporting to post-integration-test.
Calling mvn verify should do it!
Not sure what Maven will do with the artifacts/jars that were built, anyhow, the build should fail.

damianszczepanik · 2017-09-24T05:50:45Z

At least it will move report execution to verify phrase. I don't think this is plugin responsibility excep https://github.com/damianszczepanik/maven-cucumber-reporting/blob/master/src/main/java/net/masterthought/cucumber/CucumberReportGeneratorMojo.java#L110

cbaldan · 2017-09-27T06:07:03Z

I made a poc and got it all to work successfully under maven failsafe plugin, but the code is still too messy.
I'll clean it up and push to my repo soon.

cbaldan · 2017-10-01T21:11:58Z

I just pushed to my repo the branch verify-goal-poc, which is working and generating the report as expected, even when there are test failures.

I made the pom configuration as simple as possible:
https://github.com/cbaldan/cucumber-reporting-poc/tree/verify-goal-poc

The only thing is that cucumber-reporting does not like to be in the verify phase, whether there are failed tests or not.
I produced the output below commenting commenting out Then This step fails from sample feature file I provided. If tests fail, the same log4j errors still show up.
But if cucumber-reporting is not run, no log4j errors show up.
@damianszczepanik could you help me clean out those error messages?

Thanks,
Cleber.

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-cucumber-reporting:3.3.0:generate (generate-HTML-report) @ cucumber-reporting-investigation ---
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
[INFO] About to generate Cucumber report.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.136 s
[INFO] Finished at: 2017-10-01T17:57:29-03:00
[INFO] Final Memory: 11M/38M
[INFO] ------------------------------------------------------------------------
Exception in thread "pool-1-thread-1" java.lang.NoClassDefFoundError: org/apache/logging/log4j/message/ParameterizedMessage
        at org.apache.logging.log4j.message.ParameterizedNoReferenceMessageFactory.newMessage(ParameterizedNoReferenceMessageFactory.java:104)
        at org.apache.logging.log4j.message.AbstractMessageFactory.newMessage(AbstractMessageFactory.java:75)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2010)
        at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1884)
        at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:793)
        at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry.run(DefaultShutdownCallbackRegistry.java:76)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.ParameterizedMessage
        at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
        at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
        at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
        at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)

damianszczepanik · 2017-10-02T18:11:56Z

I have seen similar problems and all of them were related to log4j version. Take a look at closed issues https://github.com/damianszczepanik/cucumber-reporting/issues?q=is%3Aissue+is%3Aclosed

cbaldan · 2017-10-03T12:39:20Z

I agree @damianszczepanik , but maven-cucumber-reporting is the only project using log4j:

If it gets commented out, no errors arise.
I`m positively sure it comes from this plugin.

Cleber@DELL-9550 MINGW64 /d/gitProjects/cucumber-reporting-poc (verify-goal-poc)
$ mvn dependency:tree -Dincludes=org.apache.logging.log4j
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building cucumber-reporting-investigation 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ cucumber-reporting-investigation ---
[INFO] cbaldan:cucumber-reporting-investigation:jar:0.0.1-SNAPSHOT
[INFO] \- net.masterthought:maven-cucumber-reporting:jar:3.3.0:compile
[INFO]    \- net.masterthought:cucumber-reporting:jar:3.3.0:compile
[INFO]       +- org.apache.logging.log4j:log4j-api:jar:2.7:compile
[INFO]       \- org.apache.logging.log4j:log4j-core:jar:2.7:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.976 s
[INFO] Finished at: 2017-10-03T09:29:53-03:00
[INFO] Final Memory: 10M/24M
[INFO] ------------------------------------------------------------------------

damianszczepanik · 2017-10-04T21:00:42Z

I'm pretty sure it is not. Through out the dependencies I believe there are many log4j as this is very popular logging library.

cbaldan · 2017-10-06T23:27:14Z

That's not what maven is saying.
You could checkout the project I provided and see with your own eyes that it only happens when maven-cucumber-reporting is executed.

Can you prove me wrong?

AniketVIC · 2017-10-13T08:52:47Z

RESOLVED 👍 mvn install -Dmaven.test.failure.ignore=true
on command line

aistoica · 2018-03-06T09:02:23Z

I found the solution for this. Use failsafe plugin:

       <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>${maven.failsafe.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>net.masterthought</groupId>
            <artifactId>maven-cucumber-reporting</artifactId>
            <version>3.15.0</version>
            <executions>
                <execution>
                    <id>execution</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                    <configuration>
                        <projectName>Tests</projectName>
                        <outputDirectory>target/cucumber-reports/advanced-reports</outputDirectory>
                        <cucumberOutput>target/cucumber-reports/CucumberTestReport.json</cucumberOutput>
                    </configuration>
                </execution>
            </executions>
        </plugin>

Hope it helps

damianszczepanik closed this as completed Mar 16, 2017

damianszczepanik reopened this Sep 19, 2017

damianszczepanik closed this as completed Sep 24, 2017

This was referenced Sep 25, 2017

When tests have failed, this plugin may prevent execution of all the remaining phases #45

Closed

Generate reports when <testFailureIgnore>true</testFailureIgnore> is set to false #54

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate reports when tests fail? #61

Generate reports when tests fail? #61

cbaldan commented Mar 15, 2017

damianszczepanik commented Mar 16, 2017

cbaldan commented Mar 16, 2017

cbaldan commented Mar 28, 2017

shohilbagdai commented May 26, 2017

cbaldan commented May 26, 2017

shohilbagdai commented May 26, 2017 via email •

edited

Loading

ptrthomas commented Jun 24, 2017

damianszczepanik commented Jun 24, 2017

ptrthomas commented Jun 24, 2017

KLBonn commented Aug 29, 2017 •

edited

Loading

kprpalash82 commented Aug 29, 2017 •

edited

Loading

pnakhat commented Sep 19, 2017

damianszczepanik commented Sep 23, 2017

cbaldan commented Sep 23, 2017

cbaldan commented Sep 23, 2017

damianszczepanik commented Sep 24, 2017

cbaldan commented Sep 27, 2017

cbaldan commented Oct 1, 2017

damianszczepanik commented Oct 2, 2017

cbaldan commented Oct 3, 2017

damianszczepanik commented Oct 4, 2017

cbaldan commented Oct 6, 2017 •

edited

Loading

AniketVIC commented Oct 13, 2017

aistoica commented Mar 6, 2018 •

edited

Loading

Generate reports when tests fail? #61

Generate reports when tests fail? #61

Comments

cbaldan commented Mar 15, 2017

damianszczepanik commented Mar 16, 2017

cbaldan commented Mar 16, 2017

cbaldan commented Mar 28, 2017

shohilbagdai commented May 26, 2017

cbaldan commented May 26, 2017

shohilbagdai commented May 26, 2017 via email • edited Loading

ptrthomas commented Jun 24, 2017

damianszczepanik commented Jun 24, 2017

ptrthomas commented Jun 24, 2017

KLBonn commented Aug 29, 2017 • edited Loading

kprpalash82 commented Aug 29, 2017 • edited Loading

pnakhat commented Sep 19, 2017

damianszczepanik commented Sep 23, 2017

cbaldan commented Sep 23, 2017

cbaldan commented Sep 23, 2017

damianszczepanik commented Sep 24, 2017

cbaldan commented Sep 27, 2017

cbaldan commented Oct 1, 2017

damianszczepanik commented Oct 2, 2017

cbaldan commented Oct 3, 2017

damianszczepanik commented Oct 4, 2017

cbaldan commented Oct 6, 2017 • edited Loading

AniketVIC commented Oct 13, 2017

aistoica commented Mar 6, 2018 • edited Loading

shohilbagdai commented May 26, 2017 via email •

edited

Loading

KLBonn commented Aug 29, 2017 •

edited

Loading

kprpalash82 commented Aug 29, 2017 •

edited

Loading

cbaldan commented Oct 6, 2017 •

edited

Loading

aistoica commented Mar 6, 2018 •

edited

Loading