code coverage measurement in demetra roadmap

the demetra roadmap included code coverage support. i was considering writing a plugin to use for showing the coverage information from some OS coverage tools myself, but this seems likely to do it so i dont need to do it myself. would it be possible to get some more details on what is planned for it? what levels of coverage (class, method, lines, ..) , what types of coverage (http://www.bullseye.com/coverage.html), how well is it integrated with idea, is the data available through openapi? this would help me decide if i will do some work on my own or not..

thanks,
-t

5 comments

We are currently working on integrating with EMMA coverage tool (so we are likely to have line coverage). We are planning to do online instrumentation using java 5.0 agents mechanism (this is the pnly way we could get the use of it ourselves, since class loader mechanism is inapplicabble for us). The results will be visualized in the editor, but any client could read the coverage data through emma API for their purposes.

Eugene.

0

Outstanding. I used to work with the guy behind Emma (Vlad Roubtsov), and he's created some fine code.

--Dave Griffith

0

Thanks for the information. EMMA is the tool I was mainly considering myself. I suppose it will be several months before this will be available in EAP though? Your reply made me look up what java 5.0 agents are, but I am still a bit unsure what you mean. I guess I would need to spend some more time reading on the agents. Do you mean to use agents to run emma instrumentation before running any code being measured? I understood agents are something that allows hooking into before code is run and replacing class bytecode before it is executed, or something like that. Did I understand wrong? In considering a plugin I was thinking of running EMMA instrumentation after compile, then after running the code dumping the coverage data to disk in EMMAs XML format. I would then have used this information to visualize the data, which I guess is quite similar to what you will do. However, what is the EMMA API you talk about? I have written some code to integrate with EMMA but never found any API for specifically integrating and accessing the coverage data other than dumping it to disk? Do you plan to offer an API for direct access to the coverage data? That would be sweet.

Thanks,
-Teemu

0

Yes, you got the concept of 1.5 agents right. What it does is to allow on the fly code instrumentation (Emma already supports online code instrumentation, but only through the use of custom class loaders, which is inapplicable if the application itself uses custom loaders). We decided to go instrumenting online, because this way we don't have to maintain two versions of class files (we don't want to have only instrumented classes that run tens percents slower that the original, right?). Also having online instrumentation would allow to limit the coverage path to subset of all classes: when running unit tests for certain package, it seems sufficient to instrument source classes from that package only.

As for dumping XML, it seems too expensive to dump report files for all instrumented classes. What we would like to accomplish is our own implementation of emma ReportGenerator interface, that constructs per-file coverage information on demand when a request to visualize in the editor is made.

Also we should of course support visualizing coverage information got from running unit tests on a separate server, but this seems rather straightforward.

Eugene.

P.S. It is rather hard to estimate when coverage support appears in EAP, but the work has already started:)

0

OK, thanks for the information. I agree with your points and your implementation. Maintaining 2 sets of source files or a single set of instrumented files is not efficient. I would probably have provided a button to separately instrument classes after having built them, if I had made a plugin. This way the user would have had a single set of classes that they could have chosen to have instrumented or not. But this is because I wouldn't have had the resources to implement something as efficient as custom online instrumentation. For the same reason I would have used the XML output as it is already provided by EMMA and I wouldn't have needed to write my own report generator. I did always find it unnecessarily expensive but never did get to writing my own ReportGenerator.

As it seems I still have plenty of work to do before I actually can use this functionality, I am not in such a hurry to get it from the EAP but it is always nice to know someone is doing the work instead of me needing to do it :). Sounds like online instrumentation should also give you nice flexibility with coverage measurement. I'll check it out when its available and post some RFE's etc :).

-Teemu

0

Please sign in to leave a comment.