Unit tests fail if executed all at once, starting from the 212.x version of Intellij Platform SDK Follow
I develop a language plugin, it has lots of unit tests. Starting from the 212.x version of the SDK some of them fail if executed all at once, but they don't fail if executed individually. The kinds of unit tests failing:
Exactly the same unit tests executed all at once on the 211.x version work fine, i.e. all successful.
What could be the cause? Any idea?
P.S. I had another issue with the 212.x and 213.x versions and the cause was a change in the SDK implementation, therefore, I guess, it could be something similar. See https://intellij-support.jetbrains.com/hc/en-us/community/posts/4410472950418-Custom-action-for-specific-file-type-on-an-existing-shortcut
Please sign in to leave a comment.
A number of common problems leading to such flaky tests are listed here https://plugins.jetbrains.com/docs/intellij/testing-faq.html#issues
Most often, some test doesn't fully restore the state to "full neutral" after running, thus causing unpredictable results when e.g. order of tests is changed.
Thanks for the link, Yann Cebron!
How to understand the following paragraph?
Should I wrap each test block into try/finally? For example:
Yes, I've added a snippet to the page:
In the end, I was able to workaround tests, but I still don't know what has changed in 212.x.
The folding and formatting tests were fixed by adding myFixture.configureByText(...), because the filetype was identified as the plain text, e.g.
One of the parsing tests provided different results for one of the files, e.g.
I had to delete this file from the test case because I couldn't fix or workaround it.
See first entries in https://plugins.jetbrains.com/docs/intellij/api-changes-list-2021.html#20213. Make sure you use latest gradle-intellij-plugin
No difference. The parsing tests fail as before if executed with other test cases. These DUMMY_BLOCK elements appear.
Let me expand on Yann's suggestion. I had a very similar issue where tests started to fail after updating that ran perfectly before. I did a lot of digging but there was one annoying part: The tests did not fail when I ran them under a debugger and also, like yours, they only failed if I ran all tests.
What I found is that my highlighting tests were responsible which was surprising because 1) the actual test code was very small and 2) the highlighting tests always passed but it was other tests that failed after them. When I "ignored" the highlighting tests, all other tests would pass again. That confused me because it meant my highlighting tests left the testing in a different state and I could not explain that because I only called "configureByText" and "checkHighlighting".
I stared at the code for ages (on my code) but then looked into the BasePlatformTestCase and saw that the file created with "configureByText(LanguageInstance, code)" is always created/overwritten the same file name "aaa.extension". Therefore, I simply tried to make the file name unique using
and like magic, all my tests passed again.
Therefore, if you experience the same behavior, please report back because it very likely means we need to look deeper into the IntelliJ Platform code.