Basic Use of Testwell CTC++

With a small example program to decide if a number is a prime, we describe the first iteration of coverage measurement with Testwell CTC++.

Assume we have five source files making up the complete program as follows:

  • prime.c
  • io.h
  • io.c
  • calc.h
  • calc.c

This example can be found also in the \Examples directory of the Testwell CTC++ delivery package.

This program can be compiled and linked in many ways, here Microsoft's Visual Studio compiler cl is used: 
cl –Feprime.exe prime.c io.c calc.c
which results in the prime.exe program. When the program is run, it repeatedly prompts for integer values and determines if the value is a prime or not. Finally, when input 0 is given, the program ends.

Now, we wish to measure code coverage for this program and the manually performed tests.

First we need to instrument the source files. Assume we wish to measure multicondition coverage. This can be done as follows: 
ctc -i m cl –Feprime.exe prime.c io.c calc.c

As a result we get the instrumented prime.exe program. Here ctc is the Testwell CTC++ preprocessor tool, which instruments the given C source files and compiles and links them to the new instrumented target. The -i m command-line option sets the instrument mode to multicondition coverage.

When a file is compiled, instrumentation means that ctc takes a temporary copy of the source file and adds additional code to it, often called probes, and then compiles the file. The original source file remains intact, the object file gets replaced with the instrumented version of the file. During the linking phase, ctc adds its run-time library to the linkage, because the instrumented object files need it. The linking result is the instrumented program replacing the original program in this example.

When ctc instruments source files, it maintains descriptions about the relevant content in a symbol file. If no other name is specified, it is called MON.sym, and is generated in current directory. If this file existed already, it is updated.

Now prime.exe is the instrumented version of the program. Let's do one test run on it as follows: 
prime

Enter a number (0 for stop program): 2
2 IS a prime.

Enter a number (0 for stop program): 5
5 IS a prime.
		
Enter a number (0 for stop program): 20
20 IS NOT a prime.

Enter a number (0 for stop program): 0

After this first test run, a data file, called MON.dat, is created, containing the coverage counters collected during execution. If this file exists already, it is updated.

Now we wish to see the coverage results of our test. We use the Testwell CTC++ postprocessor tool ctcpost as follows: 
ctcpost -p profile.txt

ctcpost takes the (default) symbol file MON.sym and the data file MON.dat as input and generates a text report profile.txt, called Execution Profile Listing, as an output.

To get a full, browsable report including the source code, we use the tool ctc2html to generate an HTML report: 
ctc2html -i profile.txt
From both these reports we can read quite a lot of information on how the code was executed during the test run.
  • For example, we can see that in the file calc.c there is the function is_prime on line 4. That function was called 3 times.
  • The conditional expression on line 8 in file calc.c has been executed in both ways (once true, twice false).
  • Because we instrumented the source for multicondition coverage, there is a closer analysis on how that non-trivial decision was evaluated:
    • We can see that the second combination F || T || _, where "_" means "not evaluated", has determined the overall decision once to true.
    • The fourth combination F || F || F has determined the overall decision to false in the two remaining cases.
    • The combinations T || _ || _ (first) and F || F || T (third) have not been tested.

So, from the perspective of thorough testing, the test input values 1 and 3 should be added 1 . There are also other locations in this function not fully tested yet.

Testwell CTC++ shows the test thoroughness as TER, Test Effectiveness Ratio. It is a ratio between the number of locations that did get a hit and the number of locations that should have gotten a hit. The hit locations correspond to the probes Testwell CTC++ inserted.

The TER is measured for structural coverage, in this example multicondition coverage, as well as for statement coverage. Look at the real source code of function is_prime() in file calc.c. Testwell CTC++ calculates that the there are 11 statements in that function (7 standard statements ending with ";" 3 if-statements and 1 for-statement). 9 of the 11 statements are reported as executed.

Now we could continue the testing and run the instrumented prime.exe program again, entering new integers, which we concluded to be missing from one of the reports. When the new test run ends, the MON.dat file is updated automatically. Then, generating the reports again, we should see some increase in the coverage.

1 By the way, even full coverage does not mean that the test results match the expectation. In other words: 1 is not a prime.