Issues when running tests (counts wrong, output, ...)

If I run the "test" Rake task (attached Test::Unit) there are some issues with that. The main thing is that the count above is wrong, in my example it shows "Done: 64 of 64" but it really has run 77 tests:

16 tests, 53 assertions, 0 failures, 0 errors
Test suite finished: 1.271 seconds
61 tests, 140 assertions, 0 failures, 0 errors
Test suite finished: 9.04 seconds

Even without changing anything to my tests, the next run may show different numbers.

Looking at the logs shows lines like the following:

2009-06-04 11:58:51,894 [  75739]   WARN - eralToSMTRunnerEventsConvertor - Test wasn't started! TestFinished event: name = {LocationTest.test_truth}. Cant find running test for [LocationTest.test_truth]. Current running tests: {}

Worth to mention that I've counted 13 lines - exactly the number of tests I'm missing. I've also seen similar lines stating "Not all events were processed! [LocationTest.test_truth]"

As the 'missing' tests change it looks like some race conditions occuring.

Just realized that the output tab shows matching lines:

##teamcity[testFinished name='LocationTest.test_truth' duration='2' timestamp='2009-06-04|'T|'12:26:55.785+0200']



Besides that the grouping of the test classes and test cases doesn't work properly if running all tests. The unit tests show up correctly but the functionals show up under the first functional test class (this has its test cases and all other functional test classes as child nodes).

The last thing affects the output:

  • there's no tab to view the output in the test.log (as the server log tab)
  • viewing the test.log in the editor doesn't interpret the color coding
5 comments
Comment actions Permalink
  • there's no tab to view the output in the test.log (as the server log tab)

I've filled an issue - http://www.jetbrains.net/jira/browse/RUBY-4248. We will add the similar log tab as for server.log in rails server run configuration.

2009-06-04 11:58:51,894 [  75739]   WARN - eralToSMTRunnerEventsConvertor - Test wasn't started! TestFinished event: name = {LocationTest.test_truth}. Cant find running test for [LocationTest.test_truth]. Current running tests: {}

Worth to mention that I've counted 13 lines - exactly the number of tests I'm missing. I've also seen similar lines stating "Not all events were processed! [LocationTest.test_truth]"


As the 'missing' tests change it looks like some race conditions occuring.

As far as I know Rake and Test::Unit doesn't run tests simultaneously so race condition should be impossible. RubyMine logs test/suite started/finished events in stdout. In your case for "LocationTest.test_truth" RubyMine received only "testFinished" event and "testStarted" was missed. Maybe some of your tests redirects or caputres $stdout ?

Try to run only LocationTest test suite. Are test runner show results correctly?

Besides that the grouping of the test classes and test cases doesn't work properly if running all tests. The unit tests show up correctly but the functionals show up under the first functional test class (this has its test cases and all other functional test classes as child nodes).

I think it is side effects of missed events.
Also please look through test's output in RubMine's test runner. Does it contains exceptions related to RubyMine's tests runner?

0
Comment actions Permalink

When I run the single tests, all works well. But if I run multiple tests (e.g. 'test' or 'test:units') the issues mentioned before occur. As mentioned, it's not the same set of tests that show up in the logs, this may differ from one run to another, in the same way the counts may differ.

As it's clear to me that this is hard to reproduce I've created a new Rails project with 4 models (all scaffolded) and added some relationships (probably not required). When I run the tests, similar issues occur (well, the grouping in the left pane doesn't break). Attached you'll find the project directory archive and an archive with the log showing some test runs, their output and according screenshots (I forgot to create the latter at first, so you'll have two RubyMine runs in the log). Interestingly even the screenshots differ...



Attachment(s):
Test_Project-results.tar.gz
Test_Project.tar.gz
0
Comment actions Permalink

Today I realized, that this is not only a - more or less - minor display issue affecting the counts, some output and the 'test tree'. It may even happen, that either tests are not executed or failures are not reported.
I don't know exactly what is the case but I assume it is the first - the test did not appear in the tree on the left and I've seen one of the "##teamcity" messages in the output.
If I run the tests from the 'test' directory or directly from the according file, the failure is correctly reported.

0
Comment actions Permalink

upetschke wrote:

Today I realized, that this is not only a - more or less - minor display issue affecting the counts, some output and the 'test tree'. It may even happen, that either tests are not executed or failures are not reported. :(
I don't know exactly what is the case but I assume it is the first - the test did not appear in the tree on the left and I've seen one of the "##teamcity" messages in the output.
If I run the tests from the 'test' directory or directly from the according file, the failure is correctly reported.

Thank you for your logs and test project. I've reproduce the bug and we've fixed it in RubyMine 1.1.1! (new version is available on http://www.jetbrains.com/ruby/download/index.html) Actually it was bug in console output parsing which is necessary for building test tree in GUI.

0
Comment actions Permalink

Quick test shows: it works

0

Please sign in to leave a comment.