Custom testing framework - Showing tests in console

Answered

I am creating a custom testing framework plugin for a testing framework that I am developing as part of a university project. I have managed to create plugin but I am running into some difficulties passing the tests to the console UI so Intellij can display it properly.

I have implemented BaseJavaApplicationCommandLineState and passed it to my Configuration through getState, but I cannot seem to get it to find my tests when I run the test class, I know that the test methods are called as they print to the text console, but Intellij just tells me that the "test framework quit unexpectedly". Do i need to pass it some special parameters to locate the tests.

I tried to pass it a SMTestLocator but that doesn't seem to help and there is limited documentation on how to implement it properly.

If anyone could help or point me in the right direction that would be great.

Many thanks.

13 comments
Comment actions Permalink

Hi Anna, 

I have viewed that topic and have tried to understand it, the one thing I'm confused about is how I pass the listener to the console, do I need to somehow pass a SMTRunnerEventsListener to listen the events? SMTestProxy is stating that it doensn't have a test reporter which is why it is stating that the test quit, also I have noticed that the test name in the SMTestProxy is "[root]".

Should this be the name of my test file? or is this correct?

Many thanks,

Tom

0
Comment actions Permalink

In addition, after some further poking, I believe SMTestLocator is where the downfall is, my system just can't seem to recognise the tests. Is there an example of how to implement the SMTestLocator? It's not just as simple as checking a PsiMethod for an annotation. Or is there a way to get the  PsiMethod/PsiClass from the path and protocol?

Many thanks,

Tom

0
Comment actions Permalink

https://github.com/JetBrains/intellij-community/blob/master/java/execution/impl/src/com/intellij/execution/testframework/JavaTestLocator.java

it's used when you want to navigate from tests tree to the test source

But ... it looks like that you don't send service messages from your tests to IDEA VM. How would you attach a listener to your tests to send messages depends on the test framework you use

Anna

0
Comment actions Permalink

Hi Anna,
The test framework is a custom one, I don’t rely on another framework. I have created a framework that intellij calls from the command line then passes in the class to framework for testing and any optional methods to test. In my plugin I also extend JavaTestFramework(I think that’s what it’s called) and register it in my plugin.xml so that IntelliJ knows about the framework and I can create a new test for the test generator.
So as I understand it I need to pass service messages from my custom framework? Or from the plugin itself using ServiceMessageBuilder?

Thank you for the help,
Tom

0
Comment actions Permalink

After thinking about you last comment and looking at the topic you linked, I realised that I need to send ##teamcity messages over the console for IntelliJ to pick up the tests.

Now after days of trying I am making progress after sending ##teamcity[enteredTheMatrix] I now have IntelliJ finding my test suite!! More work to go but I see and end in sight.

Just one more question I have seen that from other open source projects and IntelliJ itself that I need to send more messages for my tests to be seen. Is there any documentation on what ##teamcity messages I need to send? I can see that a few of them take extra arguments but I don't know the full specification of these messages.

 

(Sorry for the double comments, and I really appreciate the help)

Tom

0
Comment actions Permalink

you need also to send testStarted/testFinished/etc messages so IDEA will build and update the tree of tests. See e.g. `com.intellij.junit4.JUnit4TestListener`.

 

You also may check `jetbrains.buildServer.messages.serviceMessages.ServiceMessageTypes` for all recognised types of events

Anna

0
Comment actions Permalink

That’s great Anna I’ve made solid progress after looking what you’ve suggested, thanks for the help.
Tom

0
Comment actions Permalink

Hi Anna,

Sorry to bring this back up. There's just one more problem I am facing that I hope you can help with. The testing it working perfectly and IntelliJ is displaying the results of the test correctly.

I am struggling to get the gutter icons to run the tests. IntelliJ is stating "nothing here" when I click them.

When a test class is created there are no initial gutter icons, but once the tests have run it will display the rerun tests icon, which when clicked doesn't work.

Is it advised to supply my own RunLineMarkerContributor with custom actions to run the tests? Or is there an api provided by IntelliJ I can call to tell it where the tests are?

I assume that I need to provide it with a ReRunFailedTests action, I have noticed there is an AbstractReRunFailedTests. Which is why there is nothing to run when I click the rerun gutter icon.

 

Thanks,

Tom

0
Comment actions Permalink

is it java you write your tests in? If so, then registered TestFramework extension should be enough. If not, then you need to register smth similar to `com.intellij.testIntegration.TestRunLineMarkerProvider`

Anna

0
Comment actions Permalink

Hi Anna,

I write my tests in Java and I have a test framework registered but it doesn’t give me the initial run gutter icon, only the rerun icons after I have ran the tests using the main green run triangle on the tool bar? Maybe I have my test framework setup wrong

Tom

0
Comment actions Permalink

Please debug here `com.intellij.testIntegration.TestRunLineMarkerProvider#getInfo(com.intellij.psi.PsiElement)`. It looks like `isTest` does't work for some reason

0
Comment actions Permalink

Thank you Anna I will look into that, I feel it is some wrong setup on my part.
Tom

0

Please sign in to leave a comment.