docs: Sphinxify TestSuiteMakefileGuide
Some small related fixups to TestingGuide too. llvm-svn: 167996
This commit is contained in:
Sean Silva
parent
27d126f97b
commit
e0db519ae9
|
|
@ -1,351 +0,0 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
|
||||
"http://www.w3.org/TR/html4/strict.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
||||
<title>LLVM test-suite Makefile Guide</title>
|
||||
<link rel="stylesheet" href="_static/llvm.css" type="text/css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<h1>
|
||||
LLVM test-suite Makefile Guide
|
||||
</h1>
|
||||
|
||||
<ol>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#testsuitestructure">Test suite structure</a></li>
|
||||
<li><a href="#testsuiterun">Running the test suite</a>
|
||||
<ul>
|
||||
<li><a href="#testsuiteexternal">Configuring External Tests</a></li>
|
||||
<li><a href="#testsuitetests">Running different tests</a></li>
|
||||
<li><a href="#testsuiteoutput">Generating test output</a></li>
|
||||
<li><a href="#testsuitecustom">Writing custom tests for test-suite</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<div class="doc_author">
|
||||
<p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p>
|
||||
</div>
|
||||
|
||||
<!--=========================================================================-->
|
||||
<h2><a name="overview">Overview</a></h2>
|
||||
<!--=========================================================================-->
|
||||
|
||||
<div>
|
||||
|
||||
<p>This document describes the features of the Makefile-based LLVM
|
||||
test-suite. This way of interacting with the test-suite is deprecated in favor
|
||||
of running the test-suite using LNT, but may continue to prove useful for some
|
||||
users. See the Testing
|
||||
Guide's <a href="TestingGuide.html#testsuitequickstart">test-suite
|
||||
Quickstart</a> section for more information.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!--=========================================================================-->
|
||||
<h2><a name="testsuitestructure">Test suite Structure</a></h2>
|
||||
<!--=========================================================================-->
|
||||
|
||||
<div>
|
||||
|
||||
<p>The <tt>test-suite</tt> module contains a number of programs that can be compiled
|
||||
with LLVM and executed. These programs are compiled using the native compiler
|
||||
and various LLVM backends. The output from the program compiled with the
|
||||
native compiler is assumed correct; the results from the other programs are
|
||||
compared to the native program output and pass if they match.</p>
|
||||
|
||||
<p>When executing tests, it is usually a good idea to start out with a subset of
|
||||
the available tests or programs. This makes test run times smaller at first and
|
||||
later on this is useful to investigate individual test failures. To run some
|
||||
test only on a subset of programs, simply change directory to the programs you
|
||||
want tested and run <tt>gmake</tt> there. Alternatively, you can run a different
|
||||
test using the <tt>TEST</tt> variable to change what tests or run on the
|
||||
selected programs (see below for more info).</p>
|
||||
|
||||
<p>In addition for testing correctness, the <tt>test-suite</tt> directory also
|
||||
performs timing tests of various LLVM optimizations. It also records
|
||||
compilation times for the compilers and the JIT. This information can be
|
||||
used to compare the effectiveness of LLVM's optimizations and code
|
||||
generation.</p>
|
||||
|
||||
<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource,
|
||||
SingleSource, and External.</p>
|
||||
|
||||
<ul>
|
||||
<li><tt>test-suite/SingleSource</tt>
|
||||
<p>The SingleSource directory contains test programs that are only a single
|
||||
source file in size. These are usually small benchmark programs or small
|
||||
programs that calculate a particular value. Several such programs are grouped
|
||||
together in each directory.</p></li>
|
||||
|
||||
<li><tt>test-suite/MultiSource</tt>
|
||||
<p>The MultiSource directory contains subdirectories which contain entire
|
||||
programs with multiple source files. Large benchmarks and whole applications
|
||||
go here.</p></li>
|
||||
|
||||
<li><tt>test-suite/External</tt>
|
||||
<p>The External directory contains Makefiles for building code that is external
|
||||
to (i.e., not distributed with) LLVM. The most prominent members of this
|
||||
directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt>
|
||||
directory does not contain these actual tests, but only the Makefiles that know
|
||||
how to properly compile these programs from somewhere else. The presence and
|
||||
location of these external programs is configured by the test-suite
|
||||
<tt>configure</tt> script.</p></li>
|
||||
</ul>
|
||||
|
||||
<p>Each tree is then subdivided into several categories, including applications,
|
||||
benchmarks, regression tests, code that is strange grammatically, etc. These
|
||||
organizations should be relatively self explanatory.</p>
|
||||
|
||||
<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
|
||||
others are features that we haven't added yet (or may never add). In the
|
||||
regression tests, the result for such tests will be XFAIL (eXpected FAILure).
|
||||
In this way, you can tell the difference between an expected and unexpected
|
||||
failure.</p>
|
||||
|
||||
<p>The tests in the test suite have no such feature at this time. If the
|
||||
test passes, only warnings and other miscellaneous output will be generated. If
|
||||
a test fails, a large <program> FAILED message will be displayed. This
|
||||
will help you separate benign warnings from actual test failures.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!--=========================================================================-->
|
||||
<h2><a name="testsuiterun">Running the test suite</a></h2>
|
||||
<!--=========================================================================-->
|
||||
|
||||
<div>
|
||||
|
||||
<p>First, all tests are executed within the LLVM object directory tree. They
|
||||
<i>are not</i> executed inside of the LLVM source tree. This is because the
|
||||
test suite creates temporary files during execution.</p>
|
||||
|
||||
<p>To run the test suite, you need to use the following steps:</p>
|
||||
|
||||
<ol>
|
||||
<li><tt>cd</tt> into the <tt>llvm/projects</tt> directory in your source tree.
|
||||
</li>
|
||||
|
||||
<li><p>Check out the <tt>test-suite</tt> module with:</p>
|
||||
|
||||
<div class="doc_code">
|
||||
<pre>
|
||||
% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
|
||||
</pre>
|
||||
</div>
|
||||
<p>This will get the test suite into <tt>llvm/projects/test-suite</tt>.</p>
|
||||
</li>
|
||||
<li><p>Configure and build <tt>llvm</tt>.</p></li>
|
||||
<li><p>Configure and build <tt>llvm-gcc</tt>.</p></li>
|
||||
<li><p>Install <tt>llvm-gcc</tt> somewhere.</p></li>
|
||||
<li><p><em>Re-configure</em> <tt>llvm</tt> from the top level of
|
||||
each build tree (LLVM object directory tree) in which you want
|
||||
to run the test suite, just as you do before building LLVM.</p>
|
||||
<p>During the <em>re-configuration</em>, you must either: (1)
|
||||
have <tt>llvm-gcc</tt> you just built in your path, or (2)
|
||||
specify the directory where your just-built <tt>llvm-gcc</tt> is
|
||||
installed using <tt>--with-llvmgccdir=$LLVM_GCC_DIR</tt>.</p>
|
||||
<p>You must also tell the configure machinery that the test suite
|
||||
is available so it can be configured for your build tree:</p>
|
||||
<div class="doc_code">
|
||||
<pre>
|
||||
% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
|
||||
</pre>
|
||||
</div>
|
||||
<p>[Remember that <tt>$LLVM_GCC_DIR</tt> is the directory where you
|
||||
<em>installed</em> llvm-gcc, not its src or obj directory.]</p>
|
||||
</li>
|
||||
|
||||
<li><p>You can now run the test suite from your build tree as follows:</p>
|
||||
<div class="doc_code">
|
||||
<pre>
|
||||
% cd $LLVM_OBJ_ROOT/projects/test-suite
|
||||
% make
|
||||
</pre>
|
||||
</div>
|
||||
</li>
|
||||
</ol>
|
||||
<p>Note that the second and third steps only need to be done once. After you
|
||||
have the suite checked out and configured, you don't need to do it again (unless
|
||||
the test code or configure script changes).</p>
|
||||
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<h3>
|
||||
<a name="testsuiteexternal">Configuring External Tests</a>
|
||||
</h3>
|
||||
<!-- _______________________________________________________________________ -->
|
||||
|
||||
<div>
|
||||
<p>In order to run the External tests in the <tt>test-suite</tt>
|
||||
module, you must specify <i>--with-externals</i>. This
|
||||
must be done during the <em>re-configuration</em> step (see above),
|
||||
and the <tt>llvm</tt> re-configuration must recognize the
|
||||
previously-built <tt>llvm-gcc</tt>. If any of these is missing or
|
||||
neglected, the External tests won't work.</p>
|
||||
<dl>
|
||||
<dt><i>--with-externals</i></dt>
|
||||
<dt><i>--with-externals=<<tt>directory</tt>></i></dt>
|
||||
</dl>
|
||||
This tells LLVM where to find any external tests. They are expected to be
|
||||
in specifically named subdirectories of <<tt>directory</tt>>.
|
||||
If <tt>directory</tt> is left unspecified,
|
||||
<tt>configure</tt> uses the default value
|
||||
<tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>.
|
||||
Subdirectory names known to LLVM include:
|
||||
<dl>
|
||||
<dt>spec95</dt>
|
||||
<dt>speccpu2000</dt>
|
||||
<dt>speccpu2006</dt>
|
||||
<dt>povray31</dt>
|
||||
</dl>
|
||||
Others are added from time to time, and can be determined from
|
||||
<tt>configure</tt>.
|
||||
</div>
|
||||
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<h3>
|
||||
<a name="testsuitetests">Running different tests</a>
|
||||
</h3>
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<div>
|
||||
<p>In addition to the regular "whole program" tests, the <tt>test-suite</tt>
|
||||
module also provides a mechanism for compiling the programs in different ways.
|
||||
If the variable TEST is defined on the <tt>gmake</tt> command line, the test system will
|
||||
include a Makefile named <tt>TEST.<value of TEST variable>.Makefile</tt>.
|
||||
This Makefile can modify build rules to yield different results.</p>
|
||||
|
||||
<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
|
||||
create the nightly test reports. To run the nightly tests, run <tt>gmake
|
||||
TEST=nightly</tt>.</p>
|
||||
|
||||
<p>There are several TEST Makefiles available in the tree. Some of them are
|
||||
designed for internal LLVM research and will not work outside of the LLVM
|
||||
research group. They may still be valuable, however, as a guide to writing your
|
||||
own TEST Makefile for any optimization or analysis passes that you develop with
|
||||
LLVM.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<h3>
|
||||
<a name="testsuiteoutput">Generating test output</a>
|
||||
</h3>
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<div>
|
||||
<p>There are a number of ways to run the tests and generate output. The most
|
||||
simple one is simply running <tt>gmake</tt> with no arguments. This will
|
||||
compile and run all programs in the tree using a number of different methods
|
||||
and compare results. Any failures are reported in the output, but are likely
|
||||
drowned in the other output. Passes are not reported explicitly.</p>
|
||||
|
||||
<p>Somewhat better is running <tt>gmake TEST=sometest test</tt>, which runs
|
||||
the specified test and usually adds per-program summaries to the output
|
||||
(depending on which sometest you use). For example, the <tt>nightly</tt> test
|
||||
explicitly outputs TEST-PASS or TEST-FAIL for every test after each program.
|
||||
Though these lines are still drowned in the output, it's easy to grep the
|
||||
output logs in the Output directories.</p>
|
||||
|
||||
<p>Even better are the <tt>report</tt> and <tt>report.format</tt> targets
|
||||
(where <tt>format</tt> is one of <tt>html</tt>, <tt>csv</tt>, <tt>text</tt> or
|
||||
<tt>graphs</tt>). The exact contents of the report are dependent on which
|
||||
<tt>TEST</tt> you are running, but the text results are always shown at the
|
||||
end of the run and the results are always stored in the
|
||||
<tt>report.<type>.format</tt> file (when running with
|
||||
<tt>TEST=<type></tt>).
|
||||
|
||||
The <tt>report</tt> also generate a file called
|
||||
<tt>report.<type>.raw.out</tt> containing the output of the entire test
|
||||
run.
|
||||
</div>
|
||||
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<h3>
|
||||
<a name="testsuitecustom">Writing custom tests for the test suite</a>
|
||||
</h3>
|
||||
<!-- _______________________________________________________________________ -->
|
||||
|
||||
<div>
|
||||
|
||||
<p>Assuming you can run the test suite, (e.g. "<tt>gmake TEST=nightly report</tt>"
|
||||
should work), it is really easy to run optimizations or code generator
|
||||
components against every program in the tree, collecting statistics or running
|
||||
custom checks for correctness. At base, this is how the nightly tester works,
|
||||
it's just one example of a general framework.</p>
|
||||
|
||||
<p>Lets say that you have an LLVM optimization pass, and you want to see how
|
||||
many times it triggers. First thing you should do is add an LLVM
|
||||
<a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which
|
||||
will tally counts of things you care about.</p>
|
||||
|
||||
<p>Following this, you can set up a test and a report that collects these and
|
||||
formats them for easy viewing. This consists of two files, a
|
||||
"<tt>test-suite/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
|
||||
test) and a "<tt>test-suite/TEST.XXX.report</tt>" file that indicates how to
|
||||
format the output into a table. There are many example reports of various
|
||||
levels of sophistication included with the test suite, and the framework is very
|
||||
general.</p>
|
||||
|
||||
<p>If you are interested in testing an optimization pass, check out the
|
||||
"libcalls" test as an example. It can be run like this:<p>
|
||||
|
||||
<div class="doc_code">
|
||||
<pre>
|
||||
% cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level
|
||||
% make TEST=libcalls report
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<p>This will do a bunch of stuff, then eventually print a table like this:</p>
|
||||
|
||||
<div class="doc_code">
|
||||
<pre>
|
||||
Name | total | #exit |
|
||||
...
|
||||
FreeBench/analyzer/analyzer | 51 | 6 |
|
||||
FreeBench/fourinarow/fourinarow | 1 | 1 |
|
||||
FreeBench/neural/neural | 19 | 9 |
|
||||
FreeBench/pifft/pifft | 5 | 3 |
|
||||
MallocBench/cfrac/cfrac | 1 | * |
|
||||
MallocBench/espresso/espresso | 52 | 12 |
|
||||
MallocBench/gs/gs | 4 | * |
|
||||
Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
|
||||
Prolangs-C/agrep/agrep | 33 | 12 |
|
||||
Prolangs-C/allroots/allroots | * | * |
|
||||
Prolangs-C/assembler/assembler | 47 | * |
|
||||
Prolangs-C/bison/mybison | 74 | * |
|
||||
...
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<p>This basically is grepping the -stats output and displaying it in a table.
|
||||
You can also use the "TEST=libcalls report.html" target to get the table in HTML
|
||||
form, similarly for report.csv and report.tex.</p>
|
||||
|
||||
<p>The source for this is in test-suite/TEST.libcalls.*. The format is pretty
|
||||
simple: the Makefile indicates how to run the test (in this case,
|
||||
"<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for
|
||||
each column of the output. The first value is the header for the column and the
|
||||
second is the regex to grep the output of the command for. There are lots of
|
||||
example reports that can do fancy stuff.</p>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- *********************************************************************** -->
|
||||
|
||||
<hr>
|
||||
<address>
|
||||
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img
|
||||
src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a>
|
||||
<a href="http://validator.w3.org/check/referer"><img
|
||||
src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a>
|
||||
|
||||
John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br>
|
||||
<a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br>
|
||||
Last modified: $Date$
|
||||
</address>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,279 @@
|
|||
==============================
|
||||
LLVM test-suite Makefile Guide
|
||||
==============================
|
||||
|
||||
Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
|
||||
Lattner
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
This document describes the features of the Makefile-based LLVM
|
||||
test-suite. This way of interacting with the test-suite is deprecated in
|
||||
favor of running the test-suite using LNT, but may continue to prove
|
||||
useful for some users. See the Testing Guide's :ref:`test-suite Quickstart
|
||||
<test-suite-quickstart>` section for more information.
|
||||
|
||||
Test suite Structure
|
||||
====================
|
||||
|
||||
The ``test-suite`` module contains a number of programs that can be
|
||||
compiled with LLVM and executed. These programs are compiled using the
|
||||
native compiler and various LLVM backends. The output from the program
|
||||
compiled with the native compiler is assumed correct; the results from
|
||||
the other programs are compared to the native program output and pass if
|
||||
they match.
|
||||
|
||||
When executing tests, it is usually a good idea to start out with a
|
||||
subset of the available tests or programs. This makes test run times
|
||||
smaller at first and later on this is useful to investigate individual
|
||||
test failures. To run some test only on a subset of programs, simply
|
||||
change directory to the programs you want tested and run ``gmake``
|
||||
there. Alternatively, you can run a different test using the ``TEST``
|
||||
variable to change what tests or run on the selected programs (see below
|
||||
for more info).
|
||||
|
||||
In addition for testing correctness, the ``test-suite`` directory also
|
||||
performs timing tests of various LLVM optimizations. It also records
|
||||
compilation times for the compilers and the JIT. This information can be
|
||||
used to compare the effectiveness of LLVM's optimizations and code
|
||||
generation.
|
||||
|
||||
``test-suite`` tests are divided into three types of tests: MultiSource,
|
||||
SingleSource, and External.
|
||||
|
||||
- ``test-suite/SingleSource``
|
||||
|
||||
The SingleSource directory contains test programs that are only a
|
||||
single source file in size. These are usually small benchmark
|
||||
programs or small programs that calculate a particular value. Several
|
||||
such programs are grouped together in each directory.
|
||||
|
||||
- ``test-suite/MultiSource``
|
||||
|
||||
The MultiSource directory contains subdirectories which contain
|
||||
entire programs with multiple source files. Large benchmarks and
|
||||
whole applications go here.
|
||||
|
||||
- ``test-suite/External``
|
||||
|
||||
The External directory contains Makefiles for building code that is
|
||||
external to (i.e., not distributed with) LLVM. The most prominent
|
||||
members of this directory are the SPEC 95 and SPEC 2000 benchmark
|
||||
suites. The ``External`` directory does not contain these actual
|
||||
tests, but only the Makefiles that know how to properly compile these
|
||||
programs from somewhere else. The presence and location of these
|
||||
external programs is configured by the test-suite ``configure``
|
||||
script.
|
||||
|
||||
Each tree is then subdivided into several categories, including
|
||||
applications, benchmarks, regression tests, code that is strange
|
||||
grammatically, etc. These organizations should be relatively self
|
||||
explanatory.
|
||||
|
||||
Some tests are known to fail. Some are bugs that we have not fixed yet;
|
||||
others are features that we haven't added yet (or may never add). In the
|
||||
regression tests, the result for such tests will be XFAIL (eXpected
|
||||
FAILure). In this way, you can tell the difference between an expected
|
||||
and unexpected failure.
|
||||
|
||||
The tests in the test suite have no such feature at this time. If the
|
||||
test passes, only warnings and other miscellaneous output will be
|
||||
generated. If a test fails, a large <program> FAILED message will be
|
||||
displayed. This will help you separate benign warnings from actual test
|
||||
failures.
|
||||
|
||||
Running the test suite
|
||||
======================
|
||||
|
||||
First, all tests are executed within the LLVM object directory tree.
|
||||
They *are not* executed inside of the LLVM source tree. This is because
|
||||
the test suite creates temporary files during execution.
|
||||
|
||||
To run the test suite, you need to use the following steps:
|
||||
|
||||
#. ``cd`` into the ``llvm/projects`` directory in your source tree.
|
||||
#. Check out the ``test-suite`` module with:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
|
||||
|
||||
This will get the test suite into ``llvm/projects/test-suite``.
|
||||
|
||||
#. Configure and build ``llvm``.
|
||||
|
||||
#. Configure and build ``llvm-gcc``.
|
||||
|
||||
#. Install ``llvm-gcc`` somewhere.
|
||||
|
||||
#. *Re-configure* ``llvm`` from the top level of each build tree (LLVM
|
||||
object directory tree) in which you want to run the test suite, just
|
||||
as you do before building LLVM.
|
||||
|
||||
During the *re-configuration*, you must either: (1) have ``llvm-gcc``
|
||||
you just built in your path, or (2) specify the directory where your
|
||||
just-built ``llvm-gcc`` is installed using
|
||||
``--with-llvmgccdir=$LLVM_GCC_DIR``.
|
||||
|
||||
You must also tell the configure machinery that the test suite is
|
||||
available so it can be configured for your build tree:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
|
||||
|
||||
[Remember that ``$LLVM_GCC_DIR`` is the directory where you
|
||||
*installed* llvm-gcc, not its src or obj directory.]
|
||||
|
||||
#. You can now run the test suite from your build tree as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
% cd $LLVM_OBJ_ROOT/projects/test-suite
|
||||
% make
|
||||
|
||||
Note that the second and third steps only need to be done once. After
|
||||
you have the suite checked out and configured, you don't need to do it
|
||||
again (unless the test code or configure script changes).
|
||||
|
||||
Configuring External Tests
|
||||
--------------------------
|
||||
|
||||
In order to run the External tests in the ``test-suite`` module, you
|
||||
must specify *--with-externals*. This must be done during the
|
||||
*re-configuration* step (see above), and the ``llvm`` re-configuration
|
||||
must recognize the previously-built ``llvm-gcc``. If any of these is
|
||||
missing or neglected, the External tests won't work.
|
||||
|
||||
* *--with-externals*
|
||||
|
||||
* *--with-externals=<directory>*
|
||||
|
||||
This tells LLVM where to find any external tests. They are expected to
|
||||
be in specifically named subdirectories of <``directory``>. If
|
||||
``directory`` is left unspecified, ``configure`` uses the default value
|
||||
``/home/vadve/shared/benchmarks/speccpu2000/benchspec``. Subdirectory
|
||||
names known to LLVM include:
|
||||
|
||||
* spec95
|
||||
|
||||
* speccpu2000
|
||||
|
||||
* speccpu2006
|
||||
|
||||
* povray31
|
||||
|
||||
Others are added from time to time, and can be determined from
|
||||
``configure``.
|
||||
|
||||
Running different tests
|
||||
-----------------------
|
||||
|
||||
In addition to the regular "whole program" tests, the ``test-suite``
|
||||
module also provides a mechanism for compiling the programs in different
|
||||
ways. If the variable TEST is defined on the ``gmake`` command line, the
|
||||
test system will include a Makefile named
|
||||
``TEST.<value of TEST variable>.Makefile``. This Makefile can modify
|
||||
build rules to yield different results.
|
||||
|
||||
For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to
|
||||
create the nightly test reports. To run the nightly tests, run
|
||||
``gmake TEST=nightly``.
|
||||
|
||||
There are several TEST Makefiles available in the tree. Some of them are
|
||||
designed for internal LLVM research and will not work outside of the
|
||||
LLVM research group. They may still be valuable, however, as a guide to
|
||||
writing your own TEST Makefile for any optimization or analysis passes
|
||||
that you develop with LLVM.
|
||||
|
||||
Generating test output
|
||||
----------------------
|
||||
|
||||
There are a number of ways to run the tests and generate output. The
|
||||
most simple one is simply running ``gmake`` with no arguments. This will
|
||||
compile and run all programs in the tree using a number of different
|
||||
methods and compare results. Any failures are reported in the output,
|
||||
but are likely drowned in the other output. Passes are not reported
|
||||
explicitly.
|
||||
|
||||
Somewhat better is running ``gmake TEST=sometest test``, which runs the
|
||||
specified test and usually adds per-program summaries to the output
|
||||
(depending on which sometest you use). For example, the ``nightly`` test
|
||||
explicitly outputs TEST-PASS or TEST-FAIL for every test after each
|
||||
program. Though these lines are still drowned in the output, it's easy
|
||||
to grep the output logs in the Output directories.
|
||||
|
||||
Even better are the ``report`` and ``report.format`` targets (where
|
||||
``format`` is one of ``html``, ``csv``, ``text`` or ``graphs``). The
|
||||
exact contents of the report are dependent on which ``TEST`` you are
|
||||
running, but the text results are always shown at the end of the run and
|
||||
the results are always stored in the ``report.<type>.format`` file (when
|
||||
running with ``TEST=<type>``). The ``report`` also generate a file
|
||||
called ``report.<type>.raw.out`` containing the output of the entire
|
||||
test run.
|
||||
|
||||
Writing custom tests for the test suite
|
||||
---------------------------------------
|
||||
|
||||
Assuming you can run the test suite, (e.g.
|
||||
"``gmake TEST=nightly report``" should work), it is really easy to run
|
||||
optimizations or code generator components against every program in the
|
||||
tree, collecting statistics or running custom checks for correctness. At
|
||||
base, this is how the nightly tester works, it's just one example of a
|
||||
general framework.
|
||||
|
||||
Lets say that you have an LLVM optimization pass, and you want to see
|
||||
how many times it triggers. First thing you should do is add an LLVM
|
||||
`statistic <ProgrammersManual.html#Statistic>`_ to your pass, which will
|
||||
tally counts of things you care about.
|
||||
|
||||
Following this, you can set up a test and a report that collects these
|
||||
and formats them for easy viewing. This consists of two files, a
|
||||
"``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of
|
||||
your test) and a "``test-suite/TEST.XXX.report``" file that indicates
|
||||
how to format the output into a table. There are many example reports of
|
||||
various levels of sophistication included with the test suite, and the
|
||||
framework is very general.
|
||||
|
||||
If you are interested in testing an optimization pass, check out the
|
||||
"libcalls" test as an example. It can be run like this:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
% cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level
|
||||
% make TEST=libcalls report
|
||||
|
||||
This will do a bunch of stuff, then eventually print a table like this:
|
||||
|
||||
::
|
||||
|
||||
Name | total | #exit |
|
||||
...
|
||||
FreeBench/analyzer/analyzer | 51 | 6 |
|
||||
FreeBench/fourinarow/fourinarow | 1 | 1 |
|
||||
FreeBench/neural/neural | 19 | 9 |
|
||||
FreeBench/pifft/pifft | 5 | 3 |
|
||||
MallocBench/cfrac/cfrac | 1 | * |
|
||||
MallocBench/espresso/espresso | 52 | 12 |
|
||||
MallocBench/gs/gs | 4 | * |
|
||||
Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
|
||||
Prolangs-C/agrep/agrep | 33 | 12 |
|
||||
Prolangs-C/allroots/allroots | * | * |
|
||||
Prolangs-C/assembler/assembler | 47 | * |
|
||||
Prolangs-C/bison/mybison | 74 | * |
|
||||
...
|
||||
|
||||
This basically is grepping the -stats output and displaying it in a
|
||||
table. You can also use the "TEST=libcalls report.html" target to get
|
||||
the table in HTML form, similarly for report.csv and report.tex.
|
||||
|
||||
The source for this is in ``test-suite/TEST.libcalls.*``. The format is
|
||||
pretty simple: the Makefile indicates how to run the test (in this case,
|
||||
"``opt -simplify-libcalls -stats``"), and the report contains one line
|
||||
for each column of the output. The first value is the header for the
|
||||
column and the second is the regex to grep the output of the command
|
||||
for. There are lots of example reports that can do fancy stuff.
|
||||
|
|
@ -8,6 +8,11 @@ Lattner
|
|||
.. contents::
|
||||
:local:
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
TestSuiteMakefileGuide
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
|
|
@ -98,10 +103,9 @@ regressions tests are in the main "llvm" module under the directory
|
|||
``llvm/test`` (so you get these tests for free with the main llvm tree).
|
||||
Use "make check-all" to run the regression tests after building LLVM.
|
||||
|
||||
The more comprehensive test suite that includes whole programs in C and
|
||||
C++ is in the ``test-suite`` module. See ```test-suite``
|
||||
Quickstart <#testsuitequickstart>`_ for more information on running
|
||||
these tests.
|
||||
The more comprehensive test suite that includes whole programs in C and C++
|
||||
is in the ``test-suite`` module. See :ref:`test-suite Quickstart
|
||||
<test-suite-quickstart>` for more information on running these tests.
|
||||
|
||||
Regression tests
|
||||
----------------
|
||||
|
|
@ -688,6 +692,8 @@ SingleSource, and External.
|
|||
programs from somewhere else. When using ``LNT``, use the
|
||||
``--test-externals`` option to include these tests in the results.
|
||||
|
||||
.. _test-suite-quickstart:
|
||||
|
||||
``test-suite`` Quickstart
|
||||
-------------------------
|
||||
|
||||
|
|
@ -710,4 +716,4 @@ under the covers and so developers who are interested in how LNT works
|
|||
under the hood may want to understand the Makefile based setup.
|
||||
|
||||
For more information on the ``test-suite`` Makefile setup, please see
|
||||
the `Test Suite Makefile Guide. <TestSuiteMakefileGuide.html>`_
|
||||
the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.
|
||||
|
|
|
|||
Loading…
Reference in New Issue