Test suites are not executed properly and tests can't be found

Answered

Description: I am currently experiencing an issue where my test suite is not being executed. The Maven build process completes without errors, but the tests themselves do not seem to run. This issue is occurring in a Java project where I am using Maven an JUnit5 for build automation and dependency management.

Environment:

  • Java Version: 21
  • Surefire Plugin Version: 3.1.2
  • Testing Framework Used:JUnit5
  • Operating System: Windows 11
  • IDE: Aqua 233.11555.14 ( couldn't find proper place for this IDE so placed in IntelliJ IDEA as connected with java language)

Issue Details: The Maven build completes successfully, and the mvn test command does not throw any errors. However, it appears that the test suite is not being executed while tests itself run just fine. The expected suite do not run, and no test are found.

Troubleshooting Steps Taken:

Test Naming Conventions: Verified that the test classes follow Maven's default naming conventions (*Test, Test*, *TestCase). Also checked for any custom includes and excludes patterns in the Surefire configuration.

Directory Structure: Confirmed that all test classes are located in the src/test/java directory.

Compilation: Ensured there are no compilation errors in the test classes.

Test Framework Compatibility: Checked for compatibility issues with the testing framework (e.g., JUnit 5) and the Surefire plugin.

Annotations: Verified that test methods are properly annotated with @Test or similar annotations as required by the testing framework.

Empty Test Classes: Ensured that the test classes contain valid test methods and are not empty.

Verbose Output: Ran Maven with the -X flag (mvn test -X) for detailed debug output but did not find any clear indication of the issue.

Plugin Version Compatibility: Checked for any compatibility issues with the version of the Surefire plugin being used.

Surefire Reports: Reviewed the Surefire reports in target/surefire-reports but found no clues about the issue.

I have also tried running tests separately with mvn test and it is running just fine. 

Additional Information:

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>ExampleAssert</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.release>21</maven.compiler.release>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.36.0</version>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.10.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite-api</artifactId>
            <version>1.10.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
                <configuration>
                    <release>21</release>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.1.2</version>
                <configuration>
                    <includes>
                        <include>**/*Suite.java</include>
<!--                        <include>**/*Test.java</include>-->
                    </includes>
                    <testFailureIgnore>true</testFailureIgnore>
                    <trimStackTrace>false</trimStackTrace>
                    <useFile>false</useFile>
                    <printSummary>true</printSummary>
                    <redirectTestOutputToFile>false</redirectTestOutputToFile>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
SmokeSuite.java

package suites;

import ecommerce.tests.LoginTest;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.platform.suite.api.Suite;

@Suite
@SelectClasses({LoginTest.class})
public class SmokeSuite {

}

I am looking for assistance in diagnosing and resolving this issue. Any insights or suggestions on what might be causing this problem and how to fix it would be greatly appreciated.

1
6 comments

Hello!

Can you please clarify how you are triggering the tests' execution (of possible, record a brief screenshare)?

Some of the AQUA's workflow may involve its' Native Build System and while it aims to stick as close as possible to Maven, there might be discrepancies in some cases.

 

You can also try enabling “Delegate IDE build/run actions to Maven” option found under Settings / Preferences → Build, Execution, Deployment → Build Tools → Maven → Runner.

1

Sure. Here is a link to screenshare https://imgur.com/a/URTcHFL.
I run it in 2 ways. Running from IDE button in class and by mvn clean test.
When I run *Test classes, these run just fine as you can see here: https://imgur.com/a/nzkyX7Y
https://imgur.com/4WmocgG

I've also tried enabling “Delegate IDE build/run actions to Maven”  as you've suggested but it didn't help.
https://imgur.com/o5XtLpX

1

Thank you for the update!

I've tried reproducing this behavior, but the issue does not seem to occur on my end.

Would you be able to share your Project, so we can dig deeper into it?

1

I can but not exactly as it is. I must change naming.
It is very simple - abstract base class which will be extended in application base classes. 
I want to run app base class via mvn test or run in class file but can run only separate test files not suite.

https://mega.nz/file/2HhWmBQS#ygaUZKE3gaL_lnzzAJ_4m5VPqLEDX0NjbO2IPrIZKMU

0

Thank you for proving the sample Project!

I've managed to reproduce the issue.

There is also a similar or, possibly, the same issue currently being investigated in IDEA-312873.

I would suggest to upvote the Issue and leave a comment briefly describing your case.

 

In my testing I have also discovered that:

Tests seem to run successfully when:

1

Thanks for tracking similar issue :)
As a temporary workaround I've used TestNG and configured test suites in xml files and these run fine.

0

Please sign in to leave a comment.