Unit tests fail if executed all at once, starting from the 212.x version of Intellij Platform SDK

Answered

Hi there,

I develop a language plugin, it has lots of unit tests. Starting from the 212.x version of the SDK some of them fail if executed all at once, but they don't fail if executed individually. The kinds of unit tests failing:

  • ParsingTestCase.doFileTest
  • BasePlatformTestCase.testFoldingWithCollapseStatus
  • FormatterTestCase.doTextTest

Exactly the same unit tests executed all at once on the 211.x version work fine, i.e. all successful.

What could be the cause? Any idea?

P.S. I had another issue with the 212.x and 213.x versions and the cause was a change in the SDK implementation, therefore, I guess, it could be something similar. See https://intellij-support.jetbrains.com/hc/en-us/community/posts/4410472950418-Custom-action-for-specific-file-type-on-an-existing-shortcut


Regards,
Maksim

0
8 comments

A number of common problems leading to such flaky tests are listed here https://plugins.jetbrains.com/docs/intellij/testing-faq.html#issues

Most often, some test doesn't fully restore the state to "full neutral" after running, thus causing unpredictable results when e.g. order of tests is changed.

0

Thanks for the link, Yann Cebron!

How to understand the following paragraph?

Always call super.tearDown() inside finally {..} block of your test class to avoid leaks and side effects from previously run (failed) tests.

Should I wrap each test block into try/finally? For example:

class TestClass {
  fun test() {
    try {...} finally { super.tearDown() }
  }
}

 

0

Yes, I've added a snippet to the page:

 

void tearDown() {
try {
// test specific tear down calls
}
catch (Exception e) {
addSuppressedException(e);
}
finally {
super.tearDown();
}
}
1

In the end, I was able to workaround tests, but I still don't know what has changed in 212.x.

The folding and formatting tests were fixed by adding myFixture.configureByText(...), because the filetype was identified as the plain text, e.g.

myFixture.configureByText(TurtleFileType, "")
myFixture.testFoldingWithCollapseStatus(
"$testDataPath/$beforeFileName",
"$testDataPath/$afterFileName"
)

One of the parsing tests provided different results for one of the files, e.g.

  PsiErrorElement:',', '.', ';', <qt object> or '{|' expected, got '('(52,53)
    PsiElement(()('(')(52,53)
  PsiElement(STRING_LITERAL_QUOTE)('"abc"')(53,58)
  PsiElement())(')')(58,59)

and

  PsiElement(DUMMY_BLOCK)(52,59)
    PsiErrorElement:',', '.', ';', <qt object> or '{|' expected, got '('(52,53)
      PsiElement(()('(')(52,53)
    PsiElement(DUMMY_BLOCK)(53,58)
      PsiElement(STRING_LITERAL_QUOTE)('"abc"')(53,58)
    PsiElement())(')')(58,59)

I had to delete this file from the test case because I couldn't fix or workaround it.

0

See first entries in https://plugins.jetbrains.com/docs/intellij/api-changes-list-2021.html#20213. Make sure you use latest gradle-intellij-plugin

0
  • I use the latest version of gradle-intelij-plugin always
  • I switched to JUnit 5
  • I applied the toolchain setting to use JDK 11
  • I added systemProp.idea.force.use.core.classloader=true to gradle.properties

No difference. The parsing tests fail as before if executed with other test cases. These DUMMY_BLOCK elements appear.

0

Please try

myFixture.configureByText("uniqueFilename.EXTENSION", ...")

instead of

myFixture.configureByText(TurtleFileType, "")
0

Let me expand on Yann's suggestion. I had a very similar issue where tests started to fail after updating that ran perfectly before. I did a lot of digging but there was one annoying part: The tests did not fail when I ran them under a debugger and also, like yours, they only failed if I ran all tests.

What I found is that my highlighting tests were responsible which was surprising because 1) the actual test code was very small and 2) the highlighting tests always passed but it was other tests that failed after them. When I "ignored" the highlighting tests, all other tests would pass again. That confused me because it meant my highlighting tests left the testing in a different state and I could not explain that because I only called "configureByText" and "checkHighlighting".

I stared at the code for ages (on my code) but then looked into the BasePlatformTestCase and saw that the file created with "configureByText(LanguageInstance, code)" is always created/overwritten the same file name  "aaa.extension". Therefore, I simply tried to make the file name unique using

val tmpFileName = "highlight${getTimeMillis()}.m"
myFixture.configureByText(tmpFileName, code)
val result = myFixture.doHighlighting()

and like magic, all my tests passed again.

Therefore, if you experience the same behavior, please report back because it very likely means we need to look deeper into the IntelliJ Platform code.

0

Please sign in to leave a comment.