Feature to execute failed tests only

[chun2014]

As a end to end testing framework, it would be nice if Protractor provide a way to execute failed tests only from test suite(s) run.

[juliemr]

Sounds nice - this would require integration with the test runner (Jasmine, Mocha, etc) and I'm not sure how it would know which ones failed before (it would have to store state somewhere). It's not likely that this will be added soon, but it's a neat idea.

[zarroba]

Maybe it could use the previous xunit report or any other report format as input to know which tests failed in the previous execution...

[nuweb]

I have the same problem with some tests, they pass most times, but fail sometimes, so it would be nice feature to have protractor retry those failed tests.

[iamisti]

have the same problem. I'm looking forward this feature!
Do anybody knows when it will be ready-to-use?

[jlin412]

+1

[alecxe]

+1

[hankduan]

Can you clarify which of the following use cases is the one you want?

1. Automatically rerun failed tests as part of the initial protractor run to deflake. i.e.

> protractor config.js --retry-failed
running...A,B,D passed, C,E failed
running C,E again...C passed, E failed
running E again...E passed

2. Manually rerun failed tests after edits. i.e.

> protractor config.js
running...A,B,D passed, C,E failed
> vim test.js // try to fix C and E
> protractor config.js --run-last-failed
running only C,E...C passed, E failed
> vim test.js // try to fix E
> protractor config.js --run-last-failed
running only E...E passed

[iamisti]

the first one, so "1) Automatically rerun failed tests as part of the initial protractor run to deflake"
Also it would be nice if there would be a configurable value for the maximum retry count.
e.g.: in your example there was a 2 re-run, if you set the limit to 1 then the last one won't run.

I hope it was clear.

[galxy25]

@hankduan @iamisti The first one for sure! In my case, when running tests on a CI server, having to redo an entire build/push because of a test timing out is frustrating.

[vladradz]

+1 for first one

[papucho]

+1

[willgm]

+1

[robertoronderosjr]

+1, any updates on this?

[hankduan]

Not right now, but considering the big feedback, I'll see if I can do something about it in the near future

[iamisti]

Please do it as soon as possible. This is almost the most important feature that we're missing from it.

[ruimarques]

+1

[jlin412]

+1

[sanjeevmajgotra]

+1, must have feature. have a suite of 1300+ TCs, 15~20 are failing and in teamcity and causing build failure, all are passing if running individually.

[ericmdantas]

👍

[sanjeevmajgotra]

Hi @hankduan , could you please let me know possible ETA for this feature. I need to decide whether i should have a workaround implemented or wait for this implementation from protractor team.

[hankduan]

Likely some time between 1-2 months. I'm trying to get to it as soon as possible, but there are a few other pressing issues that are ahead of this.

[ericedem]

We desparately need a way to retry tests in our pipeline, as we have reached critical mass of 100+ tests where likely one will fail. Is there an alternative workaround for retries, or is this the only way? @hankduan

[hankduan]

You would need write a script to retry from a top level. (i.e. examine the output which test passed failed, and retry those --using grep). This is probably the approach I'll need to take (well, without the script part) when I do start investigating a way to do this, as there is no way for protractor to tell jasmine or mocha to retry a test directly.

[ericedem]

hmm ok I was actually thinking about doing something like this myself. I might hack something together as a proof of concept to solve our immediate needs.

[gkohen]

@premkh9, are you using gulp? If so I suggest using protractor flake mentioned above:
https://www.npmjs.com/package/protractor-flake

[bvasilchik]

I've actually had to make a script that finds all the failures and re-runs them because there is sometimes one or two that fail but it was just a matter of communication with a service like browserstack or saucelabs taking too long or some other factor that wasn't a real positive failure.

[vcernomschi]

Protractor-flake resolved this issue for me.

[premkh9]

@gkohen, can you please help me how to implement protractor flake in protractor configuration file. I am not using gulp.

[koreyba]

+1

[longlho]

if u guys are using Mocha, it already has retry

[lalau]

@longlho

Mocha's retry is mostly good, but there are still some issues:

1. It continues the test case even it encounters a failure, and retry after the whole test case has finished. It should stop and retry as soon as it encounters a failure to save time. It is especially important to test cases that involve a longer sequence of actions.
2. It doesn't handle failures thrown in beforeEach, including both things like browser.wait timeouts and expect. I probably don't use expect at all in beforeEach, but I use browser.wait a lot in beforeEach to make sure each test starts with the same common conditions. It would be great if it can support that.

[premkh9]

can someone please help me how to implement protractor flake in protractor configuration file. I am not using gulp.

[brunofracalossi]

+1

[pix27]

+1

[premkh9]

@juliemr @NickTomlin - Its really hard to configure protractor flakes from following documentation. Can you please explain in detail, It would be really helpful to many of them

https://www.npmjs.com/package/protractor-flake

[NickTomlin]

@premkh9 if you have an issue please raise it on the protractor flake repository. Protractor flake is not an official angular/protractor project.

[philipb4u]

Re running a failed tests are very useful feature for the frame work perspective. I tried flake but It did not help. Can you please suggest any other way to achieve this.

[meizzhou]

+1

[qualityshepherd]

For folks having trouble setting up protractor-flake, I've added it to my protractor-example tests on GitHub. Take a look at the flake file, which I use to kick off tests... eg. ./flake conf.js.

You could also always use it directly... for example, assuming you have protractor-flake installed as a dev dependency (--save-dev), you could then run it with node_modules/.bin/protractor-flake -- conf.js.

[sjelin]

We could do this in jasminewd - no way to do this directly in protractor though. If you need this for mocha or cucumber you'll have to ask them for this feature.

Jasmine wd issue: angular/jasminewd#73

[irushah]

+1

[thangakumard]

+1

[anudeep-mj]

+1

[ajaysuwalka]

+1

[espekkaya]

Hello,
I have just found a library. I think it solves our problem.
Protractor-retry

[daniell0gda]

Hello,
I have just found a library. I think it solves our problem.
Protractor-retry

"Windows as an environment to launch & use this package is unfortunately not yet supported."

[GYuriy]

+1

[Juani-Canete-zz]

This is an older issue, but if you are looking for this, you can achieve this using different npm packages, I'm using:

https://www.npmjs.com/package/protractor-flake

I've changed some stuff because I'm using typescript, and when a test fails protractor shows the trace for typescript files, and flake looks the trace in order to rerun failed tests, so you need to point to transpiled JS files in order to be able to rerun failed tests, but should be pretty straightforward the implementation.

[Juani-Canete-zz]

Also works out of the box with windows/mac/linux we are using it in a cross-platform development team (=