Functional tests are unstable
Answered
I'm developing plugin and I have lots of functional tests in my project. Unfortunately, these tests are very unstable: when I run tests on windows machine they work just fine, but when it comes to build in travis CI some tests gets broken, and the set of broken tests vary from build to build. I tried to run tests on Ubuntu and Mac - some tests fail. When I run exact failed test or debug it - it succeeds. Looks like some tests affect others. I must be missing something.
My functional tests are derived from BasePlatformTestCase
JUnit 4.12
project: https://github.com/nyavro/i18nPlugin, extensions-base branch
Errors:
Please sign in to leave a comment.
Without looking at your actual test implementations, these are some common problems:
- Always call
super.tearDown()infinally {...}-block of your test class to avoid leaks/side-effects from previously run (failed) test(s)- Do not use unordered collections when expected results do depend on ordering; or change your test to use non-ordered assertions
- Do not use OS-specific assumptions (e.g. non-case-sensitive filenames, "/" path separator on Windows, ...)
Thanks for reply!
Unfortunately none of the listed hints helped me much.
-Always call
super.tearDown()infinally {...} - all my tests are under BasePlatformTestCase which shoud do tearDown-Do not use unordered collections... - I don't.
-Do not use OS-specific assumptions... - I don't.
All these advises give no answer on main question - instability. Why the same tests failed when run all tests and succeeded when run separately? I've tried to just turn them off. Then I saw that other tests (that were successful before) start failing. I turned them off one by one again and again and this process does not seem to stop. Looks like all off them are failing.
I realized that this total test craziness started when I moved to IDEA-2020.1. Before that I had thousands of tests that were stable and all running successful.
I see this warning:
Maybe the reason is around here?
The WARNING message is not related to test failures in any way. I'll take a look your tests implementations later.
Yann Cebron, thanks for your response! Currently The where I have unstable tests is completely broken, it does not even compile - I'm trying to transform the project into multi module gradle setup, so I have to fix it first. If I still observe such a strange tests behavior when I'm done, I'll continue the thread. Thanks!