Most of the changes are print statement to print function.
The iterkeys and iteritems methods were changed to keys and items. This has
the potential to affect performance because these methods return a list
rather than an iterator.
Some additional changes from the previous commit for merging gcov files:
With the --long-file-names (-l) and --preserve-paths (-p) options to gcov,
some of the file names would exceed the filesystem limits (255 characters).
One solution for shortening the names is to use the -source-prefix (-s) option
which removes some of the path elements in the name. The shortened path is
used in the file name and in the Source: tag in the file.
This causes problems with gcovr, so the path prefix is restored to the Source:
tag when merging gcov files.
The processing pipeline for gcov files has an extra step - the long file names
are written to the tgcov_*_raw directories, and then merged to the corresponding
tgcov_* directory.
Many files have some executed code related to static initializers. It clutters
the output to have those included in the output, even though they only have one
line in them and it's considered covered. Add files of this type to the files
to be removed in 'remove_unwanted_file'.
The gcov file merge and write does not yet handle branch information, which also
includes function headers. These function headers are necessary for filtering the
static initializers. Add an explicit step to filter the gcov files in the *_raw
directories.
Fixes because include files do not get coverage handled correctly for unit tests.
There are two types of files produced for coverage:
.gcno - produced at compile-time, contains notes for translating basic blocks
to source lines
.gcda - produced at run-time, contains the counts for basic blocks and arcs
between then (for branch coverage)
The .gcda files can be merged, but only for multiple runs of the same executable.
This works fine for the system tests, since they all run 'qmcpack'.
For the unit tests, they cannot be merged because they come from different executables.
(The .gcda files get automatically merged at run time, and newer versions of gcc
have a 'gcov-tool' that can merge .gcda files)
CMake puts object files for different executables in different directories (even
for the same source file), and the .gcno and .gcda files follow, so nothing gets
overwritten during collection.
However, while running 'gcov', these files get processed into files with the '.gcov'
suffix, and these can get overwritten. It's particularly bad for include files,
because they don't even have a separate .gcno/.gcda file. (The collected data for one
object file can generate multiple .gcov files, usually for included files)
Gcov has a -l (--long-file-names) switch to help with this. The .gcov filename has
two parts, separated by '##'. The first part contains the cpp.gcda file the gcov file
originated from, and the second part contains the usual source file name.
The -p (--preserve-paths) option keeps all the path components, separated by '#'.
Combining the -l and -p options with the CMake directory layout, the (long!) gcov file name is unique.
The next step is to combine the data at the .gcov file level. This is the purpose of
merge_gcov.py. Lines are combined based on the priority: covered > uncovered > no code.
The collection script is modified to first put the .gcov files in a 'raw' directory
where files have the two-part name. Then those files are merged and put into the
previously used directory (with a one-part name), and the processing proceeds as
before.
This commit only fixes the standalone coverage script. The coverage processing for the
nightly runs will be a future commit.
For lines where the unit test has coverage, but the base tests do not,
the resulting diff was marked as 'covered'. For consistency, every line
not covered in the base tests should be marked as 'No code' in the resulting
diff.
For files that are completely untouched in the unit tests, but have some execution
in the the base tests, the behavior was to mark all the executed lines as uncovered.
This would incorrectly handle lines that are not executed (uncovered) in the base.
These lines should be marked as 'no-code' (not counted in coverage) in the output
directory.
This was most obvious in the case of formic/utils/lmyengine/engine_timing.cpp.
Even though the new optimizer is not called by the base test, there are some static
initializers that get called. All the executable lines of the file were marked
as uncovered in the output, which was incorrect. After the fix, only the lines
called by the static initializers are marked as uncovered.
With this fix the percentage coverage rises to 67% locally (probably will vary
some from that on the official test system)
The code coverage model measures unit test coverage relative to a baseline
run (or runs). There are three relevant runs:
base - the baseline run (a typical run activating features to test against)
unit - the unit test run
diff - the results from the scripts comparing the coverage of unit relative to base
A test using a shortened diamondC_1x1x1_pp case is used for the base.
The added test has the label 'coverage'. It should be possible to add more tests
with the 'coverage' label and automatically expand the baseline. (The compiler
coverage code automatically accumulates data from multiple runs)
Running with more than one MPI process will have race conditions when writing the coverage
data, and will likely corrupt it. There might be ways to address this later.
Scripts in the tests/coverage directory:
- read_gcov.py - read and write GCOV files
- This can test that reading and writing is idempotent (produces the same file)
'python read_gcov.py --action test <gcov file>' will run that test.
- demangle.py - call c++filt to demangle function names
- compare_gcov.py - compare gcov files from two runs and produce the relative output
- It can also produce text output and summary of uncovered lines and
uncovered functions. The output from this needs to be cleaned up more.
- sample.cpp - a sample file for testing coverage scripts. Directions in the code comments.
Still to come are scripts to integrate this with CTest.
Mark Dewing