The three test directories jbmc, jbmc-strings and strings-smoke-tests all have a
second test profile that uses --symex-driven-lazy-loading, but this meant after the
test completed the logfiles from the first profile would have been overwritten, and
if very unlucky the two profiles could run concurrently and fight over the same logfile.
Therefore we now allow test.pl to be given a log-file suffix, both for its main log
and test output logs, use that facility for these tests.
This is a change to the multi-line match option to enable traditional regex multi-line matching. The m flag causes ^ and $ to actually match correctly on new line/carriage returns, where before the match was actually on the entire file so ^ was just beginning of file $ was the end. This also fixes a few tests' failure condition where this subtle difference wasn't noticed and '^warning: ignoring' is used.
Test cases can now have secondary tags after their main CORE/KNOWNBUG/FUTURE/THOROUGH tag,
which can be filtered using -I (require tag) or -X (exclude tag) options. Both options can
be repeated; -I x -I y means a case needs either x or y tags, and -X x -X y means it cannot
have either if it is to run.
Added a new option "-p" to test.pl to request printing failure logs instead of
doing so via an external script. This simplifies various Makefiles and enables
use in cmake builds.
As we no longer use sequences of commands in the Makefile, AppVeyor's make does
not actually invoke a shell to call test.pl. Hence invoking "echo" from within
test.pl fails.
From the `make` documentation: "When it is time to execute commands to update a
target, they are executed by making a new subshell for each line. (In practice,
make may take shortcuts that do not affect the results.)" It seems these
"shortcuts" are being taken by AppVeyor's make.
In the specific case where a test died to SIGINT, we probably ought to stop.
Contrast SIGTERM, SIGKILL and so on, where we will carry on.
This also makes ctest exit cleanly when SIGINT'd
Fixes a bug where test.pl would ignore directories that contain
valid tests (*.desc and *.java combo) and only test those directories
that contained a file with exact name "test.desc".
Changes test.pl to name output *.out files after *.desc instead
of *.class files. Also updates failed-test-printer.pl to work with
those new *.out files and multiple *.desc files.
This allows us to have tests that are intended to abort.
Given that tests can (and mostly do) check the exit code this should
not reduce the utility of the test system.
- strip <CR> from end of lines in test.pl
- tests for test-script do not depend on gcc
- Makefile in regression/goto-analyser has the same structure as other test Makefiles
- remove tests for empty lines
Includes tests for excluded lines and multi lines.
Includes tests that the test runner should fail, these are set as KNOWNBUG
so we can run the regressions folder.
We also don't print out when grep options activated as no chance of doing
by accident as very clearly different to any other syntax. The printing
messes up the formatting of the output so better to not have it.
Added test script to the Makefile
Also ordered the tests into alphabetical orde.r
If the fourth line starts with a - we pass that line as flags to the
grep command when checking the lines in this test. This allows for tests
to pass flags such as: `-Pzo` to check multiple consecutive lines.
Michael Tautschnig