Code Coverage feedback

Build: 5383

Having code coverage built-in into IDEA is a great addition and done right
it will be a sweet feature. However I think it might be missing a few
details to make it truely awesome. Some comments:

o I love the simplicity of the integration. Click a checkbox, rerun and
that's it, coverage data! Very nice. But...

o ...Getting to coverage is less than obvious. I believe such a cool
feature could deserve a spot on the main UI (a button maybe?). Right now
it's hard to know it's even there for someone who isn't aware of the
feature. Even when the results are turned on, it's hard to know where to
look. The projet tree is annotated but unless one looks carefully, they'll
never notice. Overall, a few UI hints here and there would help.

o The coverage results in the editor are great, but having red/green just
in the gutter (as opposed to whole lines being colored) makes it difficult
to quickly find uncovered code. I have to precisely scan for a line with a
red mark and then look at what's in the line. I think having the option of
highlighting the whole line in the editor would make this a lot easier, as I
could read code and have the line background tell me whether it's covered or
not. Clover does this and it works very nicely.

o I get red marks for blank lines and comment lines. Why? How am I
supposed to cover comments? :) I'd rather have blank/comment lines stay
uncolored, or if anything they should be green. When I have a file that's
90% covered and I see 50% of red in the gutter because of the comments, it
looks funny.

o I'd like IDEA to be more clever when coloring multi-line statements. For
something like this:
BufferedReader reader =
new BufferedReader(
new ReallyLongNameForSomeKindOfReader(foo, bar));
IDEA colors only the first line of the statement, making its coverage
information less obvious. All three lines should be colored.

o I see some weird coverage data being shown which doesn't inspire trust
in the system. For example, I have something like this:
public testFoo() {
myfield = "abcdef";

try {
doSomething(myfield);
fail();
} catch (SomeException e) {
// ok;
}
}
It shows me that "myfield =..." and the try part of the try-catch block were
never executed, but that the catch block was executed. The test passes. How
would that even be possible? It makes me wonder if coverage data got out of
sync with the line numbers, or what. In comparison, Clover gets it pretty
much right. (note: this was after a full rebuild of the project and one
coverage run, just to make sure.)

o Is there a way to sort coverage data (for example in the projet view) by
"least covered" metric? It'd be a great way to see what really needs to be
worked on. Clover does this nicely with offline reports.

o The green/red bars the Clover plugin shows in its result tree are very
useful to quickly decode overall coverage information. Have the project tree
annotated with (10% classes, 1% lines) helps but doesn't have a big visual
impact.

A cool feature that could be even better! :)

Vince.


8 comments
Comment actions Permalink

Any chance to get some JetBrains feedback on this? Do you want plenty of
Jira issues? :)

Vince.


0
Comment actions Permalink

Build: 5383

o ...Getting to coverage is less than obvious. I
believe such a cool
feature could deserve a spot on the main UI (a button
maybe?). Right now
it's hard to know it's even there for someone who
isn't aware of the
feature.



True. I had no idea IDEA had this. Thanks.




Message was edited by:
Michael Morett

0
Comment actions Permalink

Sorry for a long delay.

I rather doubt we will make a button in the toolbar for switching coverage scheme. The action opens popup and so is not very well suited for being triggered from toolbar. As for project view annotation information, to me it looks quite unintrusive. Do you think coverage information should be the first the user sees when he looks at project view? Still we definitely need to update productivity hints and describe the feature.

As for highlighting whole lines in the editor, I don't have my opinion on this. What do others think? Would it be better if we provided an action to go to previous/next uncovered line?

Finally all the issues with lines showed/not showed as covered are the result of compiler generating debug info this way. To verify I'm right you can try to set the breakpoint on that line.

0
Comment actions Permalink

I found it difficult to use the code-coverage with only the side highlighting. I'm now fairly used to it, and if the option for full line highlight was there I might not use it. I tried another editor with full line highlighting for code coverage and that was not very easy on the eyes. Of course that might have been due to the colors chosen for the highlighting. That said, I think there should be an option for either/or style of highlighting.

Also, we need a way to change the color for the highlights. The current colors don't stand out enough on my laptop screen. I agree with the suggestion I've seen that there be an option to highlight only the uncovered code. Uncovered code is usually the focus point of any code coverage analysis, and so just highlighting that might reduce the clutter.

0
Comment actions Permalink

The colors for coverage marks will be configurable in the near future. Just let us make the beta, and breathe some fresh air...

0
Comment actions Permalink

Thanks for your response Eugene!

I rather doubt we will make a button in the toolbar for switching
coverage scheme. The action opens popup and so is not very
well suited for being triggered from toolbar.


I agree. But you guys are the UI geniuses :) What I was saying is that it's
a cool feature and it's hard to find right now, and without something a
little more obvious somewhere a lot of people are going to miss it.

As for project view annotation information, to me it looks quite
unintrusive.
Do you think coverage information should be the first the user sees when
he looks at project view?


It it very unintrusive and I like that: I can have the project view with
useful coverage information while I'm working on some stuff.

However, when coverage is the focus of my attention, I want it to "jump to
my eyes". In that case, I wouldn't mind clicking somewhere or even switching
views to have a very clear, very obvious way to see the information. For
example with Clover I bring out the Clover toolwindow that has a very easy
to drill-through result window. I work with that for a little bit to improve
my coverage and then I return to my normal project view. I don't mind
switching because the Clover toolwindow makes it very easy to analyze
coverage results and I only need it for a short period of time.

Still we definitely need to update productivity hints and describe the
feature.


That'd be a good thing to do :) When I look at how people work with IDEA
around me though, I see that the productivity hints are not used that
much -- which is why I'm insisting on my first point :)

Finally all the issues with lines showed/not showed as covered are
the result of compiler generating debug info this way. To verify I'm
right you can try to set the breakpoint on that line.


Yeah, I understand the technical reasons as to why it is the way it is now..
But you guys are JetBrains after all and you've proven you can do amazing
things :) Once you know that the first line of a statement is covered (or
not covered), you could highlight all the way to the end of the statement.
You could ditch coloring altogether for comment lines. Etc :)

Are you going to open some of the coverage API?

Thanks,

Vince.


0
Comment actions Permalink

Obvious, perhaps, but you could use Quick Switch to switch between two
different color schemes. One normal and one coverage scheme where the
coverage scheme uses line highlighting.


0
Comment actions Permalink

I tried out the Code Coverage feature and it worked. Here are my comments:

1) Why not have a 'Run with Code Coverage' at the same level as 'Run' and '
Debug'. You select your Run Configuration, and then choose one of those three
actions. Currently, what I have to do is edit a Run Configuration, flip the
Enable Code Coverage switch, run it, and then I have to edit the Run
Configuration again, and turn it off.

Just like I don't want to have two copies of a configuration for Run and Debug,
I don't want to create another configuration for coverage.

You can leave the Code Coverage configuration in the Run Configuration, like
the list of classes/packages to collect coverage information on -- just make
the "Enable code coverage" flag a menu and toolbar action.

This will add visibility to the feature and also it is more consistent with how
Run/Debug works with the same configuration.


2) I would like to see better visualization of the code coverage.

I think I would prefer to highlight the entire line with the code coverage information.
I would use a very light green and red background color. It can be very subtle while still
conveying the information.

I find with the code coverage only in the left gutter, my eye is flitting back and forth
between the left gutter and the code. I can absorb it faster if it is overlayed with the
code itself.

Options something like this:
Covered: Left Gutter Editor
Uncovered: Left Gutter Editor

I would like to see the editor lines colored for the entire width, not just where there is
text. I think that might be more pleasant to the eye.

Someone might want to only highlight the Uncovered lines in the editor, and rely on
the gutter marks for the covered lines.

I would also be interested in seeing the code coverage in the right gutter. I find that
for large files, I am flying blind because there is no bird's eye view of the code coverage
information. Again, I think if a very subtle red and green colors are used, it might look ok.
Also, I wouldn't use 3D look and feel for the code coverage bars. (See attached picture where you
can see 3D borders for the error strips.)

You could also experiment by making the code coverage stripes narrower or wider than the error
stripes. I would try flat (non-3D) very subdued (unsaturated) green and red strips which fill
the entire horizontal width of the right gutter, so it appears to be a light background color.

3) If you "Run with Code Coverage", and View Code Coverage Info is not already enabled, then after
the new coverage information is collected, it should automatically select it and display it.
This would make it easier to use.

4) Some Statistics / Reports would be nice. Top Uncovered methods, classes, etc. This would also
help address the lack of a bird's eye view of the coverage information.



Attachment(s):
5561_right_gutter_example.png
0

Please sign in to leave a comment.