cucumber tests fail when using drb, succeed when not.

Using RM 3.1.1RC2

  • cucumber (0.10.2)
  • cucumber-rails (0.4.0 5fa1bd0)
  • rspec (2.5.0)
  • spork (0.9.0.rc4)

My Spork server starts fine.

when I run all features "with --drb"


/home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --format Teamcity::Cucumber::Formatter --expand --drb --color
Testing started at 2:42 AM ...
Disabling profiles...
3 scenarios (3 passed)
12 steps (12 passed)
0m4.815s
Empty test suite.

Process finished with exit code 0
And the test side reports
No tests found

When I run it "without --drb"
/home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --format Teamcity::Cucumber::Formatter --expand --color
Testing started at 2:49 AM ...
3 scenarios (3 passed)
12 steps (12 passed)
0m2.990s

Process finished with exit code 0

And the test side reports

"All tests passed"
The only difference is the "--drb" option.
Anyone have any idea why that would be?

8 comments
Comment actions Permalink

Did you start spork server from RubyMine using "Tools | Run Spork Drb.." ?

0
Comment actions Permalink

Yes I am starting spork from inside RubyMine.  Output from Spork when running Cucumber tests is attached.



Attachment(s):
spork.out.zip
0
Comment actions Permalink

Could you show your cucumber run configuration options? Please take a screenshot.

0
Comment actions Permalink

Sure thing, Cucumber Run All Features config attached to this reply.



Attachment(s):
Screenshot-Run-Cucumber All Features.png
0
Comment actions Permalink

Hi Roman,

Thank you for your replies. Your help in this issue is highly appreciated!
This seems to be a never ending story though ;(
Me too, I've never managed to get spork up and running.
In the meantime, I work again with other solutions, but this bothers me. I have to give up many other important features.
But Testing with RubyMine is unpractical and not to recommend, because the testing without drb is to slow, its better to use terminal + textmate again.?:|
I attach my screens, too. Hopefully, this issue is gonna be fixed soon, or when me (or others) are doing something wrong have a hint,
what we can do actually.

Thanks a lot!



Attachment(s):
Bildschirmfoto 2011-04-06 um 23.11.49.png
Bildschirmfoto 2011-04-06 um 23.10.12.png
0
Comment actions Permalink

it didn't make a difference for the issue I'm having, though it might point the way toward it.

My problem is best illustrated through a 2x2 grid of . With and Without the DRB checkbox checked, With and without "--format Teamcity::Cucumber::Formatter" included in the Runner options dialog.

With DRB and with "--format Teamcity::Cucumber::Formatter"  
There is no test GUI structure. (see attached picture)
It launches with the commandline: /home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --drb --verbose --color --format Teamcity::Cucumber::Formatter

it is filled with lines like...
##teamcity[customProgressStatus testsCategory='Scenarios' count='0']

##teamcity[testSuiteStarted name='Feature: Authentication' locationHint='file:///home/scottp/workspace/lexigenic/features/manage_authentications.feature:1' timestamp='2011-04-13T17:36:23.328-0600']

##teamcity[testSuiteStarted name='Scenario: Log in user' locationHint='file:///home/scottp/workspace/lexigenic/features/manage_authentications.feature:3' timestamp='2011-04-13T17:36:23.330-0600']

##teamcity[customProgressStatus type='testStarted']

##teamcity[testStarted name='Given  I have a user with email address "scottp@lexigenic.com"' captureStandardOutput='true' locationHint='file:///home/scottp/workspace/lexigenic/features/manage_authentications.feature:4' timestamp='2011-04-13T17:36:26.025-0600']

##teamcity[testFinished name='Given  I have a user with email address "scottp@lexigenic.com"' duration='932' timestamp='2011-04-13T17:36:26.027-0600']


It ends with...
##teamcity[testSuiteFinished name='Feature: Authentication' timestamp='2011-04-13T17:36:29.422-0600']

##teamcity[customProgressStatus testsCategory='' count='0']
4 scenarios (4 passed)
15 steps (15 passed)
0m6.097s


Process finished with exit code 0

Spork reports:
Running tests with args ["/home/scottp/workspace/lexigenic/features", "--verbose", "--color", "--format", "Teamcity::Cucumber::Formatter", "--no-profile"]...
Done.


Without DRB and with "--format Teamcity::Cucumber::Formatter"
There is no test GUI structure
it launches with the commandline: /home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --verbose --color --format Teamcity::Cucumber::Formatter

The script errors out.
  * /home/scottp/workspace/lexigenic/features/step_definitions/web_steps.rb
no such file to load -- teamcity/cucumber/formatter
Error creating formatter: Teamcity::Cucumber::Formatter (LoadError)

Nothing from spork of course.

With DRB and without "--format Teamcity::Cucumber::Formatter"
Gui reports to tests found (see screenshots)

it launchs with commandline: /home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --expand --drb --verbose --color

The script starts with output
Testing started at 5:50 PM ...
Disabling profiles...
Code:
  * /home/scottp/workspace/lexigenic/features/support/env.rb
  * /home/scottp/workspace/lexigenic/features/support/paths.rb
  * /home/scottp/workspace/lexigenic/features/support/selectors.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/authentication_steps.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/manage_user_registration_steps.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/web_steps.rb
Features:
  * /home/scottp/workspace/lexigenic/features/manage_authentications.feature
Parsing feature files took 0m0.066s
Feature: Authentication
  Scenario: Log in user                                           # /home/scottp/workspace/lexigenic/features/manage_authentications.feature:3


Continues testing with lots of
Scenario: Login user incorrectly                                # /home/scottp/workspace/lexigenic/features/manage_authentications.feature:15
    Given I have a user with email address "scottp@lexigenic.com" # features/step_definitions/authentication_steps.rb:16
    When I go to the sign in page                                 # features/step_definitions/web_steps.rb:48
    And I do not sign in correctly                                # features/step_definitions/authentication_steps.rb:26
    Then I should see "Invalid email or password."                # features/step_definitions/web_steps.rb:105


Ends with
4 scenarios (4 passed)
15 steps (15 passed)
0m5.998s
Empty test suite.

Process finished with exit code 0


Spork says:
Running tests with args ["/home/scottp/workspace/lexigenic/features", "--expand", "--verbose", "--color", "--no-profile"]...
Done.


Without DRB and without "--format Teamcity::Cucumber::Formatter"
Things work the way they should
Gui says All tests pass (see screenshot)
it launches with commandline of: /home/scottp/.rvm/rubies/ruby-1.9.2-head/bin/ruby -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /home/scottp/workspace/lexigenic/script/cucumber /home/scottp/workspace/lexigenic/features --format Teamcity::Cucumber::Formatter --expand --verbose --color

It entire output is:
Testing started at 5:57 PM ...
Code:
  * /home/scottp/workspace/lexigenic/features/support/env.rb
  * /home/scottp/workspace/lexigenic/features/support/paths.rb
  * /home/scottp/workspace/lexigenic/features/support/selectors.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/authentication_steps.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/manage_user_registration_steps.rb
  * /home/scottp/workspace/lexigenic/features/step_definitions/web_steps.rb
Features:
  * /home/scottp/workspace/lexigenic/features/manage_authentications.feature
Parsing feature files took 0m0.068s
4 scenarios (4 passed)
15 steps (15 passed)
0m3.131s

Process finished with exit code 0


I repeat, the only thing different in these runs are in the "Run->Edit Configurations" settings for All Features.  DRB checked or unchecked, and the "--format Teamcity::Cucumber::Formatter" in and out of the Runner Options

I hope this helps.



Attachment(s):
RunConfigurations.png
GuiPass.png
GUIFail.png
NoTestGUI.png
0
Comment actions Permalink

Hi,

At first there is no need to specify Teamcity::Cucumber::Formatter formatter explicitly. If no '--format' option is given RubyMine will use our formatter implicitly. More over if --format or --drb options are explicitly specified RubyMine wont attach GUI to tests results.

1) With DRB and with "--format Teamcity::Cucumber::Formatter"
2) Without DRB and with "--format Teamcity::Cucumber::Formatter"
As I explained before GUI won't be shown and there is no sense in such settings.

With DRB and with "--format pretty" corresponds to console behaviour but spork server is launched from RubyMine with Teamcity::Cucumber::Formatter in loadpath

Without DRB and with "--drb --format pretty" corresponds to console behaviour if spork server is launched in OS console.



3) With DRB and without "--format Teamcity::Cucumber::Formatter"
Is expected use case for using Cucumber+Spork in RubyMine. Spork server should be launched using RubyMine | Tools | Run Spork Drb... otherwise I'll get an error that formatter isn't loaded. "--format Teamcity::Cucumber::Formatter" isn't visible in cmdline because formatter is passed using CUCUMBER_FORMAT environment variable which was set for spork server process. Normally cucumber should prefer it to other ways of specifying --format option.

"No tests found" mean that no events were sent from formatter to RubyMine GUI. According to your screenshot most likely default cmdline formatter somewhere overrides CUCUMBER_FORMAT settings. (In RM 4.0 in such cases we will show better error msg that our test reporter wasn't attached to test framework).

4) Without DRB and without "--format Teamcity::Cucumber::Formatter"
Expected use case for Cucumber without Spork in RubyMine.

P.S: I've tested it on fresh Rails 3.0.5 project with cucumber 0.10.2, cucumber-rails 0.4.0 and spork 0.9.0.rc4 / 0.8.4. In past cucumber-rails gem generated buggy cucumber.yml where CUCUMBER_FORMAT not always was preferred. cucumber.yml was discussed at http://devnet.jetbrains.net/message/5300866#5300866. In mine recently created project cucumber.yml is

<%
rerun = File.file?('rerun.txt') ? IO.read('rerun.txt') : ""
rerun_opts = rerun.to_s.strip.empty? ? "--format #{ENV['CUCUMBER_FORMAT'] || 'progress'} features" : "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} #{rerun}"
std_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} --strict --tags ~@wip"
%>
default: <%= std_opts %> features
wip: --tags @wip:3 --wip features
rerun: <%= rerun_opts %> --format rerun --out rerun.txt --strict --tags ~@wip
0

Please sign in to leave a comment.