RCov only works for the application, not tests

Hi,

I'm using 3.0 and have created and run a few basic RSpec and Cucumber tests (just to make sure the plumbing was hooked up correctly). I think I'm having another plumbing problem and hope someone can help. (I didn't see this problem described in the Forum.)

I successfully installed the 'rcov' gem in the 'test' group in my Gemfile. I then edited the run configuration settings for my Rails Development Server and Rake 'spec' and 'cucumber' tasks.

The Dev Server works fine. When I run the app it displays the following RCov warning: ** WARNING: Ruby 1.9 Support is experimental at best. Don't expect correct results! **. When I stop the server the RCov percentages are displayed in the Project window. This leads me to believe a large portion of the plumbing is in place.

However, I can't see any results for 'spec' and 'cucumber' tests. I enabled both in the Run Config dialog box the same way I did for the Dev Server. I didn't add any patterns for Dev Server and that worked, so I assume that I don't have to add any for my test apps, i.e. I assume something like *.rb and *.haml are included by default. (Is this a correct assumption?) (Also, I tried adding *.rb as a pattern and that didn't help.)

The 'Help' says something about adding appropriate options to the Rakefile when running RSpec. That part confused me. Do I have to actually edit a Rakefile somewhere? If so, where is it? I found the Rakefile for the RSpec gem that I installed, but it doesn't have a section in it that resembled what was in the Help. So I simply added the following to the 'Task Arguments' field in the 'Configuration' tab for RSpec: "rcov_opts = ENV["RCOV_OPTS"]. I don't know if this accomplishes the same thing or not.

That might be one problem, but even if it is, it seems that the cucumber tests should work. I have enabled RCov for them in the same way, but I don't see any coverage stats. Not only do I not see any stats, but I don't see the "Show Coverage" button ( the little rake icon) in either my spec or cucumber test windows after running my tests.

Another confusing thing: When I run my spec and/or cucumber tests nothing happens RCov-wise. However, I notice that if I stop my Development Server and then run either my spec or cucumber tests a dialog box pops up when the tests finish running. It says "Do you want to display coverage data for 'cucumber Coverage results'?" (Similarly for rspec.) When I click 'yes' I don't see any stats anywhere. Then if I run the tests again I don't see that dialog box; it only pops up right after stopping my Development Server.

This is all pretty screwy, but it's probably just a result of some missing plumbing or setting.

Thanks,
Larry

5 comments
Comment actions Permalink

I then edited the run configuration settings for my Rails Development Server and Rake 'spec' and 'cucumber' tasks.

...

The 'Help' says something about adding appropriate options to the Rakefile when running RSpec. That part confused me. Do I have to actually edit a Rakefile somewhere? If so,

Hi Larry,

First of all which run configuration do you use for running spec/cucumber tests? A Rake task run configuration or just a Cucumber or RSpec one?

  • In case of Rake run configuration you need configure you rakefile to collect rcov data, because cucumber/spec rake tasks launche external process and collects arguments for its command line. RubyMine cannot influence on this step.


E.g:

require "rake"
require "rspec/core/rake_task"

desc "Run all specs with rcov"
RSpec::Core::RakeTask.new("test_cov") do |t|
  t.rcov      = true
  t.rcov_opts = ENV["RCOV_OPTS"]
end

  • As for Cucumber/Test::Unit/RSpec/Ruby/Rails Server run configurations - RubyMine generates correct command line which launches tests with attached rcov engine, so no additional tuning is needed.


Please download my sample project - http://dl.dropbox.com/u/749580/work/rcov_rspec2.zip. I recommend to create new RVM sdk with empty gemset, configure project to use the sdk, then install gems from Gemfile using bundler. Check that rspec and rcov gems are automatically attached in project settings. Now you can play with "foo_spec" RSpec 2.0 configuration and "test_cov(bundler)" rake run configuration.

It says "Do you want to display coverage data for 'cucumber Coverage results'?" (Similarly for rspec.) When I click 'yes' I don't see any stats anywhere. Then if I run the tests again I don't see that dialog box; it only pops up right after stopping my Development Server.

The message appears when RubyMine suggest switch coverage obtained from one run configuration to coverage provided by some other run configuration. E.g. server coverage to rspec rake task coverage. If you rerun your run configuration coverage data will be updated without questions because it is related to the same run configuration.
"RubyMine | Tools | Show Coverage Data" action allow to switch before cached coverage results. For current coverage result you can generate standart rcov html coverage report.

Not only do I not see any stats, but I don't see the "Show Coverage" button ( the little rake icon) in either my spec or cucumber test windows after running my tests.


I don't understand. Could you attach a screenshot.
0
Comment actions Permalink

Hey Roman,

Thanks a lot for your quick and thorough reply. (As always.) (It's a wonder you guys have time to get any real work done! ;))

Before I forget, I'm using the latest versions of RubyMine, Ruby, Rails, Bundler, RVM, RSpec, Cucumber, Capybara, and RCov. (I Just downloaded and installed all of this stuff during the last few weeks. It amazes me how you can get RubyMine to correctly orchestrate all of these incredibly complex pieces - way to go!)

I was running my Cuke and RSpec tests via Rake tasks; when I switched over to RubyMine's implementation (as per your recommendation) they both started working fine, as did RCov... almost.

The Help says that the default Working Directory is the project directory. That may be true, but neither Cuke nor RSpec tests ran until I actually entered my project directory into the Working Directory text field in the Run Configuration dialog box. It seems like you shouldn't have to do this if the project directory is indeed the default.

Another problem (I think): When I run RSpec tests I see Console and RSpec Log windows within the Test Window. I do not see these windows when I run Cucumber tests. Maybe they're not supposed to be there, however there is a Logs tab in the Cucumber Run Configuration dialog box that specifies a "Cucumber log", just like there is a Logs tab in the RSpec Configuration dialog box that specifies an "RSpec log". Since RSpec shows an "RSpec Log" window I would think that Cucumber should as well; if it's not supposed to then its Logs tab is confusing and probably should be removed as it serves no purpose.

Finally, the Help has a "Monitoring Code Coverage" page that talks about a "Show Coverage" button that is supposed to be visible in the Testing Toolbar - that is the button I am talking about that I do not see. I have attached a screen shot of the Help page that mentions this button. I assume it's supposed to show up somwhere in the bottom Test Results window.

Thanks again,
Larry



Attachment(s):
ShowCoverageButton.png
0
Comment actions Permalink

Hi Larry,

Thanks a lot for your quick and thorough reply. (As always.) (It's a wonder you guys have time to get any real work done! ;))


We glad to hear it, thanks!

I was running my Cuke and RSpec tests via Rake tasks; when I switched over to RubyMine's implementation (as per your recommendation) they both started working fine, as did RCov... almost.


Rake tasks way is also possible. In my previous answer I attached an example of rspec rake task with enabled rcov + attached a sample project which supports boths types of test launching - using rake and rspec run configurations

I was running my Cuke and RSpec tests via Rake tasks; when I switched over to RubyMine's implementation (as per your recommendation) they both started working fine, as did RCov... almost.


Sorry I didn't understand "almost". Do you see coverage info in project view and editor?

The Help says that the default Working Directory is the project directory. That may be true, but neither Cuke nor RSpec tests ran until I actually entered my project directory into the Working Directory text field in the Run Configuration dialog box. It seems like you shouldn't have to do this if the project directory is indeed the default.


Did you create  & configure run configuration manually (e.g. you clicked on '+' button in Run|Edit Configurations dialog) or it was automatically generated by right click popup menu in editor? RubyMine fills "working directory" field only if configuration was automatically generated and configured by RubyMine.

Another problem (I think): When I run RSpec tests I see Console and RSpec Log windows within the Test Window. I do not see these windows when I run Cucumber tests. Maybe they're not supposed to be there, however there is a Logs tab in the Cucumber Run Configuration dialog box that specifies a "Cucumber log", just like there is a Logs tab in the RSpec Configuration dialog box that specifies an "RSpec log". Since RSpec shows an "RSpec Log" window I would think that Cucumber should as well; if it's not supposed to then its Logs tab is confusing and probably should be removed as it serves no purpose.


"Cucumer log" tab is related to ./logs/cucumber.log file. And "RSpec log" shows ./logs/test.log file. I assume "cucumber.log" file doesn't exist in your project.

Finally, the Help has a "Monitoring Code Coverage" page that talks about a "Show Coverage" button that is supposed to be visible in the Testing Toolbar - that is the button I am talking about that I do not see. I have attached a screen shot of the Help page that mentions this button. I assume it's supposed to show up somwhere in the bottom Test Results window.

It is a bug in our help. "Show Coverage" button is related to per test coverage, which isn't supported by RCov. Per-test coverage means that coverage engine allows to track coverage per each tests, not only per whole test suite. Our Help shares some parts amoung all our products and this parts relates to Java coverage. Sorry for confusing, I submitted a documentation update request - http://youtrack.jetbrains.net/issue/RUBY-7374. So in your case coverage for tests run configuration should be shown automatically.

0
Comment actions Permalink

Hi Roman,

Thanks for the answers. When I said "almost" I was referring to the things I was about to write, i.e. no default path, no log-file tab, etc. It was a sloppy and unclear way to say that, so don't feel badly about not knowing what the heck I was talking about. :)

Regarding your other comments:

1) I created my Cucumber Run Config manually (i.e. by clicking on the '+' button), so your comment about it not automatically filling in the "Working Directory" field was correct.

2) I did not have a 'cucumber.log' file. After I created one in the '/log' the "Console" and "Cucumber Log" tabs did appear in the test window, however there is no output in the 'cucumber.log' when I run the tests. Should anything be in that window?

Thanks,
Larry

0
Comment actions Permalink

2) I did not have a 'cucumber.log' file. After I created one in the '/log' the "Console" and "Cucumber Log" tabs did appear in the test window, however there is no output in the 'cucumber.log' when I run the tests. Should anything be in that window?

As far as I remember Cucumber framework creates and do some logging in that file, but I don't know exactly how to enable it, probably it has beed already deprecated.

0

Please sign in to leave a comment.