Cobertura

Code coverage: running unit tests in such a way as to be able to tell how much of your source code is being exercised by those tests.

It’s a very useful thing to do and there are quite a few tools out there to help. Some time back Goul introduced me to what I think of as the market leader: Clover from Cenqua. There are now lots of open source alternatives, so I thought I’d try the best one out to see how well it works.

The best open source alternative to my eyes is Cobertura. Best because it works, is being actively developed and because it looks just like Clover. I decided to try it out (version 1.4) on one of our JDK 1.5 projects which uses WebWork, Hibernate, a bit of Apache Jakarta networking and some threading. Not a big project, but real-world rather than “Hello World”.

There’s no top-level direct comparison to be made between Cobertura and Clover because the two products measure different things. Cobertura gives you line coverage, branch coverage and complexity; Clover gives you statements, methods, and conditionals. However they both report how many times each line of code has been executed, and both highlight the zero lines in red.

What I found was very good agreement on a line-by-line basis between the reports produced by the products. But there are differences:

• Cobertura’s summary figures are optimistic, telling me I have 100% branch coverage in places where I don’t… but the detailed reports do show non-executed branches. Perhaps it’s just a rounding error.
• Clover shows lines of code that have been partly executed. For example: a test on a boolean that has only evaluated to true even though it has been called many times. Cobertura treats the line as exercised. Of course if you always have a matching else for every if, this won’t worry you because Cobertura will spot the non-executed condition.
• Clover shows details for inner classes (Cobertura bundles them together). I don’t think this is a big deal; I just happened to notice the difference.
• Clover took 90 seconds to set up (their “quickstart” is that quick), whereas I spent about 2 hours with Cobertura. Having said that, out of the box Cobertura didn’t report on the coverage of test cases themselves, which is how I like it to be (others don’t). I didn’t find a way to make Clover ignore test cases in reports, but I’m sure there’s a way.
• I had to wriggle with the classpath a little bit for Cobertura because of a clash in the asm.jar used by Cobertura and Hibernate. The solution was to add Cobertura’s jar to the end of my classpath. I had no such problems with Clover.
• Clover’s documentation is better, although Cobertura has a nice write up over at IBM’s developerWorks site.
• Clover can track coverage over time. This isn’t that interesting for me, but I can see how it could be important in some working environments.
• Clover has IDE plugins; Cobertura does not. I run coverage reports on the server, so this isn’t an issue for me.

Summary: The Cobertura developers have produced something stunningly useful, but Clover is the better product for now. But don’t be put off: Cobertura is good enough if you’re willing to put in a little bit of time.