Code coverage

I understand it wasn't on the announcement list, but is code coverage supposed to be working yet? Whenever I attempt to run with coverage enabled I get this:

C:\JDKs\jdk1.5.0\bin\java "-javaagent:C:\Demetra\lib\emma-agent.jar=-f com/cnn/elections/loader/* -o \"C:\Documents and Settings\dgriffith\.IntelliJIdea60\system\coverage.es\"" -Didea.launcher.port=7532 -Didea.launcher.bin.path=C:\Demetra\bin -Dfile.encoding=windows-1252 -classpath C:\JDKs\jdk1.5.0\jre\lib\charsets.jar;C:\JDKs\jdk1.5.0\jre\lib\deploy.jar;C:\JDKs\jdk1.5.0\jre\lib\javaws.jar;C:\JDKs\jdk1.5.0\jre\lib\jce.jar;C:\JDKs\jdk1.5.0\jre\lib\jsse.jar;C:\JDKs\jdk1.5.0\jre\lib\plugin.jar;C:\JDKs\jdk1.5.0\jre\lib\rt.jar;C:\JDKs\jdk1.5.0\jre\lib\ext\dnsns.jar;C:\JDKs\jdk1.5.0\jre\lib\ext\localedata.jar;C:\JDKs\jdk1.5.0\jre\lib\ext\sunjce_provider.jar;C:\JDKs\jdk1.5.0\jre\lib\ext\sunpkcs11.jar;C:\Election\loader\classes;C:\Election\classes;C:\Election\lib\junit.jar;C:\Election\lib\log4j-1.2.8.jar;C:\Demetra\lib\idea_rt.jar com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit2.JUnitStarter -ideVersion5 com.cnn.elections.loader.CountyByStateLoaderTest
java.lang.ClassNotFoundException: C:\Demetra\lib\emma-agent.jar
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:290)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:279)
at java.lang.ClassLoader.loadClass(ClassLoader.java:236)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:165)
FATAL ERROR in native method: processing of -javaagent failed
Exception in thread "main"
Process finished with exit code 1

which strikes me as more likely to be a configuration bug on my machine than a half-completed feature.

(BTW, I am so looking forward to dumping Clover I can't tell you. Not that it is bad, it's just that I'm so sure that you guys are going to be better.)

--Dave Griffith

12 comments

Dave, this feature is indeed not ready yet. But the error you get puzzles me a lot.

0

BTW, as an active coverage user you could help me with this feature, describing what aspects of Clover reports you consider worth being implemented in IDEA.

0


Good things from Clover:
--Package rollups (but you should include module rollups)
--Editor-inline highlighting, including left-gutter coloring, toggleable

Not so good things about Clover:
--Separate "project-ish" view in the IDEA integration. You should have this project view annotated, instead.
--Icky and forgettable recompilation step necessary. Since you're going for the command-line agent instrumentation solution instead, this won't be a problem.
--Result merges are too painful be worth the effort. Automatic and transparent merging of results would be quite cool, and you should have enough information to do it.
--Editor highlighting only works as long as no editing is done, otherwise it gets out of sync. Solving this problem would be excellent.
--Rolling up line/method/branch coverage into one "unified" number is way pointless. Code coverage isn't about numbers, other than as indicators of issues to be fixed. Separate line/method/branch coverage is the way to go.
--Configuring include/exclude sets is fairly clunky. It looks like you're doing classname regexes, but that's probably not going to be enough. Support the full named scope functionality. Default to test classes not being covered (because they should by definition always be at 100% if they pass, and if they don't pass you shouldn't be worrying about coverage.) That said, it looks like you're choosing the right defaults, which is half the battle.

--Dave Griffith

0

I use and like Clover - I think it's a great product. But, if you can do better, that's great too.

I agree with Dave about the good and bad points. Inline highlighting is the most useful part of Clover to me, and why I bought it rather than just using Cobertura. I'm not too bothered about the recompilation step, but I'd be equally happy with instrumentation. I'd prefer it if it could be used independently of IDEA in an appserver, though.

I like being able to ignore certain lines - for example, I always turn off coverage of asserts. Clover is also supposed to support line-level exclusion regexes, so for example you can turn off coverage of log.isDebugEnabled blocks, but I've never got that working. If you could make a better interface to that, such as using structured search and replace to see what will be excluded, that would be useful.

0

Out of curiousity, is the code coverage integration implemented in a way that could be replicated by a plugin, if there were an OpenAPI way to

1)Add a configuration panel to one or more types of run profile

2)Patch the command-line based on the configuration

then there are a lot of dynamic analysis extensions that could be added to IDEA cleanly and quickly. Between the new java.lang.instrumentation stuff, annotations, and some of the new lightweight bytecode munging libraries, dynamic analysis is set to become big. The runtime @NotNull checks in the Demetra roadmap, for instance, would be a pretty trivial plugin, if you could do the above actions.

The last time I went down this path, the best ways were creating a new compiler stage, and creating a new sort of runner, both of which are too clunky for the use cases I'm thinking about. If there's now some better way, I'd love to know about it.

--Dave Griffith

0

Dave,

The coverage integration is not implemented using openapi now, but I would make it available through openapi if it proved to be valuable. However by coincidence I'm the author of runtime @NotNull checks also, so I could speak on the feasibility of runtime analysis via instrumentation.

Indeed my first try was not to make it a compiler stage, but an online instrumentation. Experimenting with instrumenting IDEA itself, however, showed that the overall startup time would be increased by approx. 3 minutes to instrument all classes (yes, I instrument with ASM, not BCEL).

On the other hand, when having this as a compiler stage, I have all the static information about instrumented application cached by IDEA at hand.

Using word occurences index, the compiler @NoNull emitter stage works in around 10 sec, which is quite applicable. Also having dynamic analyses emitted statically allows for distribution of already instrumented classes (Demetra builds are shipped with @NotNull checks inserted, which is esp. convenient for plugin writers).

Eugene.

0

--Editor highlighting only works as long as no editing is done, otherwise it gets out of sync. Solving this problem would be excellent.

We now construct coverage information based on diff of two versions of the file: the current one, and the one before coverage was gathered. There are however certain "sync" requirements: you have to launch make before running your unit tests with coverage to ensure class files and sources are in sync.

Eugene.

0

The build 5181 says it is ready. Is it really ready in 5181? I get an error like Dave's unless I add a class to the new tab. It runs without an error after adding a class, but where do I find the coverage output?

0

The error I saw looks to be JDK related. It looks like it doesn't work with early 1.5 JDKs. I got coverage working with the JDK that ships with IDEA (on Windows, at least).

--Dave Griffith

0

Eugene Vigdorchik wrote:

The coverage integration is not implemented using openapi now, but I
would make it available through openapi if it proved to be valuable.


I see the latest EAP shows code coverage support in the JUnit run
configurations (at least it says it on the changes, I'm still downloading).

How can I add that into my TestNG plugin? Any pointers anywhere?

0

I'd like to see code coverage support for TestNG. I'm using TestNG for a new project now and don't feel like returning to JUnit - not even because of the code coverage support in IDEA.

So: Please make it possible for Mark to use it in his TestNG-plugin.

0

Robert F. Beeger wrote:

So: Please make it possible for Mark to use it in his TestNG-plugin.


Hopefully it won't be too hard - first up I need to get the plugin
compiling under demetra. I'm either doing something really odd in my
setup, or theres something very fishy going on with this machine (but
then its windows...)

0

Please sign in to leave a comment.