mirror of https://gitlab.com/QEF/q-e.git
131 lines
5.1 KiB
ReStructuredText
131 lines
5.1 KiB
ReStructuredText
.. _testcode.py:
|
|
|
|
testcode.py
|
|
===========
|
|
|
|
.. only:: html
|
|
|
|
testcode.py - a command-line interface to testcode.
|
|
|
|
Synopsis
|
|
--------
|
|
|
|
testcode.py [options] [action1 [action2...]]
|
|
|
|
Description
|
|
-----------
|
|
|
|
Run a set of actions on a set of tests.
|
|
|
|
Requires two configuration files, :ref:`jobconfig` and :ref:`userconfig`. See
|
|
testcode documentation for further details.
|
|
|
|
testcode.py provides a command-line interface to testcode, a simple framework
|
|
for comparing output from (principally numeric) programs to previous output to
|
|
reveal regression errors or miscompilation.
|
|
|
|
Actions
|
|
-------
|
|
|
|
''run'' is th default action.
|
|
|
|
compare
|
|
compare set of test outputs from a previous testcode run against the
|
|
benchmark outputs.
|
|
diff
|
|
diff set of test outputs from a previous testcode run against the benchmark
|
|
outputs.
|
|
make-benchmarks
|
|
create a new set of benchmarks and update the :ref:`userconfig` file with
|
|
the new benchmark id. Also runs the 'run' action unless the 'compare'
|
|
action is also given.
|
|
recheck
|
|
compare set of test outputs from a previous testcode run against
|
|
benchmark outputs and rerun any failed tests.
|
|
run
|
|
run a set of tests and compare against the benchmark outputs.
|
|
tidy
|
|
Remove files from previous testcode runs from the test directories.
|
|
|
|
Options
|
|
-------
|
|
|
|
-h, --help
|
|
show this help message and exit
|
|
-b BENCHMARK, --benchmark=BENCHMARK
|
|
Set the file ID of the benchmark files. If BENCHMARK is in the format
|
|
t:ID, then the test files with the corresponding ID are used. This
|
|
allows two sets of tests to be compared. Default: specified in the [user]
|
|
section of the :ref:`userconfig` file.
|
|
-c CATEGORY, --category=CATEGORY
|
|
Select the category/group of tests. Can be specified multiple times.
|
|
Wildcards or parent directories can be used to select multiple directories
|
|
by their path. Default: use the `_default_` category if run is an action
|
|
unless make-benchmarks is an action. All other cases use the `_all_`
|
|
category by default. The `_default_` category contains all tests unless
|
|
otherwise set in the :ref:`jobconfig` file.
|
|
-e EXECUTABLE, --executable=EXECUTABLE
|
|
Set the executable(s) to be used to run the tests. Can be a path or name
|
|
of an option in the :ref:`userconfig` file, in which case all test programs are
|
|
set to use that value, or in the format program_name=value, which affects
|
|
only the specified program. Only relevant to the run action. Default: exe
|
|
variable set for each program listed in the :ref:`userconfig` file.
|
|
-i, --insert
|
|
Insert the new benchmark into the existing list of benchmarks in userconfig
|
|
rather than overwriting it. Only relevant to the make-benchmarks action.
|
|
Default: False.
|
|
--jobconfig=JOBCONFIG
|
|
Set path to the job configuration file. Default: jobconfig.
|
|
--job-option=JOB_OPTION
|
|
Override/add setting to :ref:`jobconfig`. Takes three arguments. Format:
|
|
section_name option_name value. Default: none.
|
|
--older-than=OLDER_THAN
|
|
Set the age (in days) of files to remove. Only relevant to the tidy
|
|
action. Default: 14 days.
|
|
-p NPROCS, --processors=NPROCS
|
|
Set the number of processors to run each test on. Only relevant to the run
|
|
action. Default: run tests as serial jobs.
|
|
-q, --quiet
|
|
Print only minimal output. Default: False.
|
|
-s QUEUE_SYSTEM, --submit=QUEUE_SYSTEM
|
|
Submit tests to a queueing system of the specified type. Only PBS system
|
|
is currently implemented. Only relevant to the run action. Default: none.
|
|
-t TEST_ID, --test-id=TEST_ID
|
|
Set the file ID of the test outputs. If TEST_ID is in the format b:ID, then
|
|
the benchmark files with the corresponding ID are used. This allows two
|
|
sets of benchmarks to be compared. Default: unique filename based upon
|
|
date if running tests and most recent test_id if comparing tests.
|
|
--total-processors=TOT_NPROCS
|
|
Set the total number of processors to use to run as many tests as possible
|
|
at the same time. Relevant only to the run option. Default: run all tests
|
|
concurrently run if --submit is used; run tests sequentially otherwise.
|
|
--userconfig=USERCONFIG
|
|
Set path to the user configuration file. Default: userconfig.
|
|
--user-option=USER_OPTION
|
|
Override/add setting to :ref:`userconfig`. Takes three arguments. Format:
|
|
section_name option_name value. Default: none.
|
|
-v, --verbose
|
|
Increase verbosity of output. Can be specified up to two times.
|
|
The default behaviour is to print out the test and its status. (See the
|
|
--quiet option to suppress even this.) Specify -v or --verbose once to
|
|
show (if relevant) which data values caused warnings or failures.
|
|
Specify -v or --verbose twice to see all (external) commands run and all
|
|
data extracted from running the tests. Using the maximum verbosity level
|
|
is highly recommended for debugging.
|
|
|
|
Exit status
|
|
-----------
|
|
|
|
1 if one or more tests fail (run and compare actions only) and 0 otherwise.
|
|
|
|
License
|
|
-------
|
|
|
|
Modified BSD License. See LICENSE in the source code for more details.
|
|
|
|
Bugs
|
|
----
|
|
|
|
Contact James Spencer (j.spencer@imperial.ac.uk) regarding bug reports,
|
|
suggestions for improvements or code contributions.
|