Showing parallel tests in SMTestRunner's console view with custom test framework

已回答

Permanently deleted user

创建于 2017年06月15日 17:35

Hello!

I'm writing a plugin for a new test framework, the idea being:

Start a process P that runs tests in parallel

Connect to P's stdout, which looks like this

[1] 12:34:56 - *** StartTest *** Fruits.testApple
[2] 12:34:56 - *** StartTest *** Fruits.testBanana
[1] 12:34:57 - Buying apples...
[2] 12:35:01 - Buying bananas...
[2] 12:35:50 - *** EndTest *** Fruits.testBanana
[1] 12:35:51 - *** EndTest *** Fruits.testApple

([1] and [2] are thread ids that allow me to track all output for a certain test)

Using SMTestRunnerConnectionUtil, connect P to the nice tree UI showing test progress
Demultiplex the log lines for each test
???
Show each test's logs when clicking on it in the tree UI

This post has been immensely helpful in getting me started, but I can't figure out step 5. The TeamCity format specifies a "flowId" parameter meant for this situation, but it doesn't seem work.

Does anyone have any insights into how display parallel tests in the SMTestRunner UI properly?

5 条评论

Ilya Kazakevich

创建于 2017年06月15日 20:39

正式评论

Hello.

FlowId is used by TC mostly for output , but Idea simply ignores it. That means you can't output anything somewhere inside of test tree except current running test.

That is reason why issues like https://youtrack.jetbrains.com/issue/PY-22549 still exist.

Besides that, your approach may work, but you need to use so called "IdBased" mode. In regular mode node can be either suite (branch) or test (leaf). Suits can't be empty. And when test failed all its ancestors are marked so. Any started test belongs to currenly opened suite hence only sequental execution is possible.

In "IdBased" mode there is no difference between tests and suites. Any test may have parent and root should have parentNodeId=0, so you can easly start 3 tests then finish second one, then first one etc.

To enable this mode you need to set ``isIdBasedTestTree()`` to true in your ```SMTRunnerConsoleProperties```.

You may also want to subscribe to ```SMTRunnerEventsListener.TEST_STATUS``` because in this mode parent tests are not marked red when child test failed (see ```PythonTRunnerConsoleProperties``` for example).

Here is simple python script (look at ```parentNodeId``` and ```nodeId```, they are important)

# coding=utf-8
from time import sleep

from teamcity import messages

service_messages = messages.TeamcityServiceMessages()
service_messages.testCount(3)
service_messages.message("testSuiteStarted", name="parent", nodeId="parent", parentNodeId="0")


# 3 processes started
service_messages.message("testStarted", name="child1", nodeId="child1", parentNodeId="parent")
service_messages.message("testStarted", name="child2", nodeId="child2", parentNodeId="parent")
service_messages.message("testStarted", name="child3", nodeId="child3", parentNodeId="parent")


sleep(1)
# process 3 finished
service_messages.message("testFinished", name="child3", nodeId="child3")


sleep(2)
# Then, 1 and 2
service_messages.message("testFinished", name="child1", nodeId="child1")
sleep(1)
service_messages.message("testFailed", name="child2", nodeId="child2", message="oops", details=":(")


service_messages.message("testSuiteFinished", nodeId="parent")

Permanently deleted user

创建于 2017年06月15日 23:59

Thank you! The UI tree now works with nodeIds and parentNodeIds, but I can't get stdout for tests to show up for nodes.

Here's how I'm handling stdout from within the plugin:

processHandler.notifyTextAvailable(ServiceMessageBuilder.testStdOut(testName)
                                                        .addAttribute("out", escapeTeamCityString(line))
                                                        .addAttribute("parentNodeId", "0")
                                                        .addAttribute("nodeId", testName) + '\n', 
                                    ProcessOutputTypes.SYSTEM);

Ilya Kazakevich

创建于 2017年06月16日 16:43

Stdout does not work with NodeId. You should use test name and do it sequentally.

>That means you can't output anything somewhere inside of test tree except current running test.

Permanently deleted user

创建于 2017年06月16日 17:15

Unfortunately the test results are coming in parallel from a 3rd party. I'll see if I can queue them up until a test is finished instead,

Is it possible to disable all output from being shown while the tests are running, or create a filter?

Ilya Kazakevich

创建于 2017年06月20日 12:45

Printing takes place in SMTestProxy#printOn . Try to set breakpoint there and you should be able to find a correct place to block it. But is easier to create some kind of proxy to queue and mute tests.

请先登录再写评论。