EMMA Code Coverage Tool

Introduction

Code coverage refers to a software engineering technique whereby you track quality and comprehensiveness of your test suite by determining simple metrics like the percentage of {classes, methods, lines, etc} that got executed when the test suite ran.

Practice shows that coverage percentages below, say, 60-70% correspond to poorly tested software. You can expect undiscovered bugs in such software . Because of this, "good" software companies instill internal processes whereby a team cannot release a piece of software unless it passes release gates like "line coverage must be 80% or higher", etc. Incidentally, reaching for 100.0% coverage isn't profitable either. You just get a lot less quality improvement for considerably more effort to reach such perfection. Coverage close to 85-90% is "good enough" for all practical purposes.

EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer's work fast and iterative.

Below is a short intro on the features of EMMA taken from (http://emma.sourceforge.net )

How does EMMA compare to other coverage tools?

Instrumentation approach:

EMMA belongs to a class of pure Java (i.e. not JVMPI- or JVMDI-based) coverage tools based on Java bytecode instrumentation. This means you don't need special JVM switches for enabling coverage: rather, you run EMMA-processed .class files more or less as you normally would (after adding EMMA as a runtime dependency). It also means EMMA does not instrument your .java sources (as is done by Clover, for example).

As a somewhat unique feature, EMMA offers two different options for when to instrument: either as an offline step ("second" compiler) or on-the-fly, when the classes are loaded by EMMA's custom classloader. The former is a generic approach that works for pretty much all contexts (J2SE, J2EE, distributed client/server applications, etc) while the latter is a handy and lightweight shortcut for simple standalone programs.

To reiterate, bytecode instrumentation means EMMA takes in .class content and outputs an enhanced version of it, with added coverage tracking hooks. In the offline mode this happens

explicitly (you start with a .class file on disk and get an instrumented version of it also on disk) and in the on-the-fly mode it happens inside the JVM (the instrumented classes are never written out to disk).

Supported coverage metrics:

EMMA supports class, method, line, and basic block coverage, aggregated at "all", package, source file, class, and method levels. Line and block metrics are offered in "normal" and "weighted" varieties. Other entries in this FAQ provide details on these. EMMA does not do branch or path coverage. However, EMMA reports fractional line coverage to help you visualize untested branches in code.

Performance and scale:

EMMA trades off the advantages of using full syntactical data in .java sources for the speed with which you can process .class content. It was created as a fast tool for both personal developments as well as for massive enterprise software projects. During performance testing, EMMA's instrumentor had been run on more than 20000 .class files in a single run. By now, EMMA has been used on a real-world app with at least 16000 classes (NetBeans).

Java platform support:

EMMA pays attention to the needs of enterprise software developers who can't always run in the latest Java version or track down obscure 3rd-party library incompatibilities: EMMA runs in any Java 1.2+ JVM and has no library dependencies.

Code Coverage of WCS classes with EMMA

STEP 1:

Download EMMA binary zip file from http://emma.sourceforge.net/downloads.html

Extract the zip file to a directory

STEP 2:

Copy emma.jar to the WCS java runtime lib folder <RAD6 installation dir>\runtimes\base_v6\java\jre\lib\ext

Copy emma.jar to WCToolkitEE60\workspace\WC\lib folder

Add emma.jar to the corresponding project build path. For example add the jar to WebSphereCommerceServerExtensionsLogic project

STEP 3:

Instrument the custom classes developed by the below command

java -cp emma.jar emma instr -ip C:\WCToolkitEE60\workspace\WebSphereCommerceServerExtensionsLogic\bin\com\commerce\security\commands -m overwrite

To run the above command you must be in the extracted Emma folder or simply add emma.jar to system path

In the above example I have chosen to overwrite the original class files with the instrumented class files by choosing –m overwrite

After instrumentation EMMA generates a file named coverage.em in the current folder or the folder we mention. This file will hold all metadata about the instrumented class. We need this file to generate report finally

For more command line option go to http://emma.sourceforge.net/userguide/userguide.html

STEP 4:

Now start WCS server and test the commands using JUNIT. For this example I have tested logon command using JUNIT. While testing you will see system out line like

SystemOut    O EMMA: collecting runtime coverage data ...

If you see the above line then it means that the instrumented class has been accessed and EMMA has collected required information and the information is stored under a file coverage.ec and will be placed under WCToolkitEE60\wasprofile folder. The coverage file creation will happen only when we stop the server.

STEP 5:

Now we will generate a HTML report using the below command

java -cp emma.jar emma report -r html -in coverage.em,C:\WCToolkitEE60\wasprofile\coverage.ec -sp C:\WCToolkitEE60\workspace\WebSphereCommerceServerExtensionsLogic\src

coverage.em and coverage.ec are the 2 files generated in the above steps and we need to ensure both these files are present.

-sp option tells where the source JAVA files are present so that we will color coded HTML output of source files.

Below are the screen shots of the report HTML

NOTE: I have hidden sensitive class names, so you will see some whitespaces in the screen shots