Running JUnit test multiple times; stop on first failure
Answered
I have some Selenium test cases that fail sporadicaly. It would be nice if I could create a run configuration where I specify the Junit method to run and the number of times I'd like to re-run it, and when it fails it would stop... or I could just have all X number of results in the run pane; with the failures alongside the succesfull runs.
Any ideas? Possible feture request?
Thanks!
Please sign in to leave a comment.
While searching for the answer myself, I stumbled upon this unanswered question. Goto "Run / Debug Configuration" and select the desired test run of choice or create a new test run. Under Configuration you can set it up to repeat "N Times", "Until Failure" or "Until Success".
Is there an way to run only the failed test every time after the first run? Would be useful, because if an Appium test was successful it shouldn't fail the next time.
Therefore I don't want to run already successful tests again
Hello,
Does the "rerun failed tests" solves the issue for you?
Please see: https://www.jetbrains.com/help/idea/performing-tests.html#rerun-failed-tests
I need to run the test automated without user interaction, so it should be started one time and continue running the failed test until a specified number of round has finished.
Say start the test with the definition run 3 Times, first run all test, 2nd round run the failed tests from the first run, and in the third round run the remaining failed test from the 2nd round.
So in the end, only the tests that really don't work should remain red.
The remaining red test should be checked at the end, why they are failing and how to make them green afterwards.
Is ther any solution to do this? Would be happy about some cli commands too.
Hello,
Unfortunately there is no ready tool to rerun the failed tests automatically after the first run.
Please feel free to create a feature request here: https://youtrack.jetbrains.com/issues/IDEA
Thank you
Would be a useful feature to have. Great suggestion Franz.
Franz, Olga Klisho Is there a feature request for this already?
Hello, please see the issue created: https://youtrack.jetbrains.com/issue/IDEA-290035
Thank you
I'm using RubyMine and I don't see this option. Olga Klisho is this only available for intellij?
I just want the simple case of running a cucumber selenium test 10 times for example, not doing anything special on failure, just so I can see common breaking points to resolve
Nicholas Shaw, there's indeed no such possibility in RubyMine. Do you need for all the frameworks or only Cucumber?
Olga Kuvardina cucumber is my only use case right now but I could certainly see it being useful for others.
I'm guessing this is an unlikely change?
Nicholas Shaw we have a request for adding it for Cucumber tests, so could you please add your vote there: https://youtrack.jetbrains.com/issue/RUBY-25056/Allow-running-repeating-cucumber-tests-Until-failure
Olga Kuvardina that seems to relate to intelliJ, it's also from three years ago so I cannot say I have much hope.
Should I raise another request for adding the run x times feature that exists within intelliJ today?
Nicholas Shaw yes, if you'd like to see it added then please submit a request on our tracker: https://youtrack.jetbrains.com/issues/RUBY