New JUnit ( or plugable test engine ) runner idea...

Along with unit testing, "Test Driven Design" is becoming a popular term
around the place. There's a TDD plugin available for IDEA, but the last
time I looked at it it lacked a spot of polish that just made it, well -
suck a little.

With little effort, I could see JetBrains providing a good one based on
reusing the existing code we already have.

I'm sitting here running some tests, the run window is along the bottom of
idea, but the majority of the screen is taken up by the output, and
statistics tab. And whilst the statistics tab is handy, I don't
particularly care about it, or the output unless I need to.

What I'm thinking of is a dedicated "Test Runner" toolwindow, docked to
the right of the screen, that a) automatically picks up any Test Suites
under configured test source directories for each module, showing a
hierarchical (sp?) display of suites, sub suites, and the tests within
them. This window doesn't have any output or statistics displayed. Just
the test tree, showing red/green tests ( we'd move the progress bar and
label beneath the tree thou ).

From here, I have quick access to all test suites, drilling down to
individual test methods ( and groups if support for TestNG is provided )
that I can run at on a whim. I don't have to worry about recreating run
profiles, etc.

There could be overall test runner settings for:

- Rebuild project in background before running any test. (one of
things that drives me insane in idea is that right-clicking inside
a testXXX() method and selecting run doesn't always compile the test
first. If the build failed for any reason, mark all tests black.
- Run ant targets in background before running any test
- Over-ridable Ant properties for test runner
- Run all tests in background every XXX minutes
This may need some form of funky configuration, maybe just tick off
a list at the module level for tests to run automatically. We want
all or nothing on the tests. As they're running in the background
we shouldn't be alerted/bothered with errors modally, or any form of
modal dialog.


Thoughts?

9 comments
Comment actions Permalink


I think there's definitely value in expanding beyond the "tests are something that you ask to have run and we show you the results" model currently to a more "your application has a bunch of tests which IDEA keeps track of and may be triggered to run in various ways" model. Continuous testing has been discussed here before a few times, although I don't believe anyone from JetBrains has commented on it. The structure you describe also fits in well with code coverage calculations, and with the ability to only rerun tests where the results might have changed (by tracking which tests hit which methods.) Overall, I very much like the idea, as it means that automation is keeping track of my testing status rather than me. When I wrote about "Wow!" ideas for Java coding that should be considered for the next release, something like this could certainly qualify.

--Dave Griffith

0
Comment actions Permalink

In article <pan.2005.03.04.19.56.06.89173@gmail.com>,
Mark Derricutt <talios@gmail.com> wrote:

There could be overall test runner settings for:

- Rebuild project in background before running any test. (one of
things that drives me insane in idea is that right-clicking inside
a testXXX() method and selecting run doesn't always compile the test
first. If the build failed for any reason, mark all tests black.
- Run ant targets in background before running any test
- Over-ridable Ant properties for test runner
- Run all tests in background every XXX minutes
This may need some form of funky configuration, maybe just tick off
a list at the module level for tests to run automatically. We want
all or nothing on the tests. As they're running in the background
we shouldn't be alerted/bothered with errors modally, or any form of
modal dialog.


Yeah do it manually. Save some time. Create those tests for you, and
you're done. AND you have to compile the project to run your tests.
Setup your compile to run in the background, and no one is forcing you
to look at the tests while they run. I still modify my javafiles while
my code's test is running.

TDD is something someone gets in the habit of doing, and what you
describe above doesn't help someone practice TDD, it just runs their
tests in a different pane in the background.

I would rather idea NOT do anything to enforce any way of writing code,
just like it does now. If I want to write a test then code, test some
more, then code, then I'll do it. I don't need someone holding my hand.
Do I want idea continuously compiling and testing? heck no, that's
annoying, I know when I'm ready to run my test... if the test
compiles it will run, but it doesn't mean it will pass, I don't want
idea telling me something I already know.

Sorry -1 on this one.

R

0
Comment actions Permalink

>I would rather idea NOT do anything to enforce any way of writing code, just like it does now. If I want to write a test then code, test some more, then code, then I'll do it. I don't need someone holding my hand.

This doesn't seem to conflict with the proposal. The idea here is that you wouldn't even notice that compilation and testing were occurring, and would only see the results by opening a tool window (although I would love to see editor highlighting for failed tests as well, while we're at it.)

>Do I want idea continuously compiling and testing? heck no, that's annoying, I know when I'm ready to run my test.

The same could be said for syntax checking, index creation and inspections. It's all a matter of just how unobtrusive that the functionality could be made. Right now, syntax checking and inspections are pretty darn unobtrusive, and index creation almost unnoticable. I see no reason to believe some sort of background testing and test state tracking couldn't be made just as unobtrusive.

--Dave Griffith

0
Comment actions Permalink

In article <27553809.1109972260549.JavaMail.itn@is.intellij.net>,
Dave Griffith <dave.griffith@cnn.com> wrote:

unobtrusive


keyword.

Codeguide and Eclipse are darn right annoying with their way they do it.

R

0
Comment actions Permalink

Thoughts?

Maybe to directly jump on TestNG would be a much better idea:
http://beust.com/testng/

It's supposed to be better than JUnit, so maybe one shouldn't invest too
much effort in something that's already dead :).

Ahmed.

0
Comment actions Permalink

On Fri, 04 Mar 2005 23:29:34 +0000, Ahmed Mohombe wrote:

It's supposed to be better than JUnit, so maybe one shouldn't invest too
much effort in something that's already dead :).


My initial thought was "we don't want to loose support for JUnit as people
already have LOTS of JUnit tests", but then, from memory - TestNG supports
running JUnit tests anyway...

0
Comment actions Permalink

On Fri, 04 Mar 2005 15:57:51 -0500, Robert Sfeir wrote:

TDD is something someone gets in the habit of doing, and what you describe
above doesn't help someone practice TDD, it just runs their tests in a
different pane in the background.


I suppose my idea isn't so much TDD, but as Dave says, Continuous Testing,
and as you say in another post, in an unobtrusive manner, such that it
doesn't get in the way of you coding your app any which way you want to.
You can ignore the results of the tests if you wish, or not, your choice.

just like it does now. If I want to write a test then code, test some
more, then code, then I'll do it. I don't need someone holding my hand.


I'm not suggesting removing the existing JUnit integration, or stopping
you from doing just this, but more adding additional magic productivity
gains.

Do I want idea continuously compiling and testing? heck no, that's
annoying,


What is it that makes it annoying? A modal progress dialog showing you
compilation progress? Messages window popping up to show you the output
of the build process? I'm suggesting an automated build/test that
provides pretty much NO UI shuffling or interruptions. The status bar may
flash a message up to say "4 Tests failed, 2 error" but other than that,
until you want to know details. You know nothing.

I know when I'm ready to run my test... if the test compiles
it will run, but it doesn't mean it will pass, I don't want idea telling
me something I already know.


The purpose is more to reveal things you didn't know, and keep you
informed of things you didn't know.

0
Comment actions Permalink

This is a great idea. Id still keep it junit based though.

If we can auto-run the tests in a dockable right hand window pane, and make the icons red/green according to their status, then how about we do the same for the project tree view, and highlight the icons of the source files that fail to compile. (ala Eclipse - this is a feature I miss)

Nick

0
Comment actions Permalink

+1 for testng integration. I like its test groups, and test dependencies, as well as its parallel test groups.

Has anyone tried the junit integration? Looks simple. Maybe intellij could auto generate the needed xml.

The latest release is also jdk 1.4 compatible (different jar, xdoclet style annotations instead of 1.5)

--pete

0

Please sign in to leave a comment.