"0 test classes found in package" after upgrading to 2017.3

Please file a bug at https://youtrack.jetbrains.com/issues/IDEA with a small sample project to reproduce attached.

See if you have the same problem as https://youtrack.jetbrains.com/v2/issue/IDEA-183169.

Hey thanks for responding. I am pretty sure its not the same issue as the one you linked.

I think, however that i know what's wrong. 

In the project i'm trying to run, the tests are in a classic structure with /java and /test directories. But the test classes are in packages under /java, and the java directory is not marked up as test in the module settings.

Prior to 2017.3 this was not a problem, you could still click on a directory containing tests and do "run tests in..." but now intellij says "no tests found" unless the directory is part of a structured flagged as "test" in module settings. I have jumped back and forth several times to verify

Does this make sense?

Hi Mathias,

yes, you are absolutely right. Thanks for the analysis!

Starting from 2017.3 IDEA detects tests by classpath to get better performance. If production output would be included in roots to process, time to discover tests would be increased (~2 times?). So the change was intentional. 

Why your project doesn't mark tests roots? 

Thanks,

Anna

Well, this is an old integration test module with mixed test and source classes that i have taken over maintenance for. In the short run, its not feasible to change the structure due to external dependencies.

I tried marking the /java dir as test root, but that caused the tests to not find the java classes that are "source code" instead, (the module contains some classes that are kind of "test utils" for common tasks, and when i removed set the directories to test instead of source, the tests couldn't find any of those classes anymore - intellij started marking them red and it didn't compile.

As a sidenote - regardless of what your goal with this for us unfortunate change is, i would argue that it's pretty confusing that i can click "run tests on" a directory that definitely contains unit tests, the process starts and then intellij says "0 tests found".

Either it should work, or the option should not be there in the menu so that you understand that tests are not possible to run.

How do you build your project artifacts and how do you start tests on CI? What build tool do you use?

There are several different modules in the project, all are structured with maven. we have scripts that run maven to create all the wars/jars we need. (about 10 jars, 3 war files)

The tests that lie in the different modules run as part of the build/ci process, and are found in the /test directories as per standard, but this is a separate module where we basically set up the entire production environment (jms queues, db, push servers, sms sending etc.) and run a bunch of end-to-end tests.

It is those tests that previously could be executed from within intellij by simply right clicking on the package under /java that contains the tests and select "run tests in...", which is not possible anymore

So your test classes get into artifacts? and you don't run those tests on CI cause maven doesn't recognize them as well?

You can use the workaround suggested in https://youtrack.jetbrains.com/v2/issue/IDEA-183169.

Though the configuration is unusual and we had some complains that IDEA finds classes in production scope and it was more or less an incident that tests in production scopes are not filtered out (e.g. because build tools such as maven or gradle do not search for tests in production scope).

No tests go into artifacts, and the module i am talking about does not get built at all. This specific module is run manually, either through maven test or from within intellij.

Again, my only problem is that i can nolonger click on a directory containing @Test classes and methods and do "run tests in".

The no-arg constructor ticket is the same that your colleague suggested earlier in this thread. It is another problem, and does not make intellij discover my test classes.

I suggested to use the same workaround:

> To workaround add -Djunit4.search.4.tests.in.classpath=false to vm options (Help | Edit custom VM options).

Hello Anna,

I am using latest IntelliJ (2017.3.3, community edition), and I am seeing this problem in my old OSS project (https://github.com/jmockit/jmockit1) which is all properly configured, and has separate directories for production and test code (both for sources and class files). IDEA always says "Empty test suite." for any package containing JUnit 4 tests. It does not occur with a package containing only TestNG tests, though; so it seems to be JUnit-specific.

BTW, adding "-Djunit4.search.4.tests.in.classpath=false" makes no difference.

However, what *does* avoid the problem is to comment out the "junit-jupiter-engine" dependency in the "main" module's pom.xml file. (This dependency is there, along with JUnit 4.12 and TestNG because the project is a testing library with support for those three different test frameworks.)

A related bug in IDEA is that it insists on running my JUnit 4 test suite with JUnit 5, simply because the pom.xml has an *optional* dependency on JUnit 5. We need to be able to create run configurations for both JUnit 4 and JUnit 5, which are completely different test frameworks (and no, I don't want to run with JUnit 5's "vintage" runner).

Thanks

I tried some tests in your project, works fine here. Could you please tell what tests do you try to run?

Thanks,

Anna

I tried running all tests under the "mockit" package.

I don't see an IDEA's bug here. Though the dependency is optional, your project won't compile without it and the production code provides custom test engine and thus IDEA has to start junit 5 as IDEA can't detect tests from custom engines and must rely on junit 5 detection mechanism (in your case it fails to find anything but IDEA doesn't know that). There are 2 ways to resolve the problem: force every user of IDEA to specify which framework to use (though junit 5 should run all tests on itself) or split the module to keep the dependency on junit 5 in separate module.

Sorry for the inconvenience,

Anna