Timing (Execution Cost) Instrumentation

Beside structural coverage, Testwell CTC++ is also able to measure the time or similar cost during code execution.

When the timing instrumentation mode has been selected for ctc, an execution cost is measured per function. With the inclusive option (-i ti), the cost for a called function also counts for the calling function. With the exclusive option (-i te), these costs are excluded.

"Time" is the default execution cost measure, giving this option its name. However, the cost can be almost everything: wall clock time, CPU time, number of page faults, number of I/O operations. The user has to provide a cost function returning the current value of the quantity to be measured.

There are three values that are measured for each function: total execution cost, maximum execution cost and average execution cost.

With the TIMER configuration parameter, the function to be used is defined. By default, the ctc.ini file has the setting

TIMER = clock

using the standard C library function clock() found from time.h for execution cost measuring. Normally, the implementation for the clock() function is also automatically found from the standard C library when linking the instrumented executable. You can change the setting to your own function, which must return a value of type clock_t defined in time.h. And when linking the instrumented executable you need to introduce the object or library file containing the implementation for your own cost function.

Testwell CTC++ maintains the timing data as 32-bit signed int. Hence a timing measure can also be less than the previous one, and cumulative timing value can also be negative. You might use this feature e.g. if your timer function would be “free heap size”, i.e. you would find out in which functions heap is consumed and released.

By default, whether the default clock() or your own timer function is used, and whether the instrumented code is C or C++, the timer function is assumed to have "C-linkage". I.e., it needs to be compiled as C code. You can also use a timer function that has "C++-linkage" (i.e. compiled as C++ code), but at instrumentation time you need to inform ctc about it. For example, with a cost function mycpptimer having "C++-linkage", the command could be:

ctc –i mti -C TIMER=mycpptimer \
-C OPT_ADD_COMPILE+-DCTC_CPP_TIMER_FUNC \
-C LIBRARY+mycpptimer.obj \
cl –FeMyprog.exe *.cpp

Each function is associated with a timer. Immediately when the function has been entered, its timer is started by calling the cost function and storing the value. When the function returns (just before a return-statement or function ending-‘}’) the timer is stopped by calling the cost function again. The time spent in the function is computed and the result is added to the time spent in the function during previous calls. When maintaining the maximum execution time, the comparison if the latest measurement is a new maximum is done by signed integral value comparison.

If the code under test is C++ and if the function return takes place with an exception, the timer is stopped and function cumulative time collected also in that situation.

Generally you should not use timing instrumentation, if the code under test is multithreaded. The measured times may be meaningless.