Not able to hit a breakpoint in IntelliJ when trying to remote debug a executing test

I have a spring boot project (Java 8, JUnit 4). I am using IntelliJ (Ultimate 2018.2) for IDE. My test passes when I run from within IntelliJ but fails when I run it from command line using gradle test command. So I am trying to remote debug my Junit test. I have breakpoints set. But when the test is executing the debugger is not stopping at the breakpoints. This is what I have done so far:

1. I have created a remote debug configuration in IntelliJ with port as 5005
2. I am running this command: gradle test -Dtest.debug on command line
3. I see build output that the build is waiting for the debugger to connect to port 5005 and the process suspends
4. At this point I run my remote debug configuration and it successfully connects to port 5005
5. After this the JUnit test is executed but the debugger does not stop at the breakpoints.

In fact when the test is running the breakpoints on the junit test are changing to lite gray color. This tells me that the JVM IntelliJ debugger got connected to did not have the executable code for my test. Gradle is spinning off multiple JVMs I believe and IntelliJ is connecting to the wrong one. How can I validate this?

How to make debugger stop at the breakpoints? Really appreciate someone's help.