This adds a new parameter, --skip-category, that can be used to list
categories that should be skipped. For example, to run all tests except for
Objective-C ones, one can now write:
./dotest.py --skip-category objc [...]
llvm-svn: 187590
- pass through to base-class implementation when raised exception is not from an LLDBTest
- should make the test suite errors a little easier to root-cause
llvm-svn: 187450
Set your env variable LLDB_TEST_ARGUMENTS to one or more options to be passed to the lldb test suite and those will be picked automatically
No more fighting about whether the progress bar is good or bad :-)
llvm-svn: 184615
This ensures that we won't try to do cleanups of test cases that we are skipping
e.g. this brings down the time required to run the cmdline category on my machine from ~70s to ~30s
llvm-svn: 184363
When -T is specified, the test suite will call svn info and dump the output on screen (this used to be the default behavior)
When -T is not specified, this step won't be performed (the new default)
llvm-svn: 179342
- fix TestCPPBool (was failing with some versions of GCC due to num_breakpoint_locations)
- speed up tests by skipping git/svn revision parsing by the test harness when running in lit-compatible (parsable) mode
This should resolve the failures reported by http://lab.llvm.org:8011/builders/lldb-x86_64-linux
llvm-svn: 176972
- change string "ERROR" to "FAIL" to match clang lit test results
Also, make LLDB tests work on machines that do not have svn and/or git installed
llvm-svn: 176633
Categories were conceptually meant to be placeable on test methods as well as test classes and test directories
However, that was broken. This checkin fixes that.
The incantation required to put categories on individual test case methods is not exactly elegant, unfortunately:
def test_case(self):
"""Test me."""
self.do_it()
def _test_case_get_categories(self):
return ["demo"]
test_case.getCategories = _test_case_get_categories
del _test_case_get_categories
llvm-svn: 176158
Remove the getCategory from TestDataFormatterObjC.py, since it was superceded by the .categories file,
and didn't work anyway (getCategories currently has to be a method on the test class, not on the test.)
Add a "basic_process" category, and start to find some tests for simple process running sniff tests.
llvm-svn: 176061
The decorators @expectedFailure (plain and special-case like i386, clang, ...) are modified to optionally take a bugnumber argument
If such an argument is specified, the failure report (or unexpected success report) will include the information passed in as part of the message
This is mostly useful for associating failures to issue IDs in issue management systems (e.g. the LLVM bugzilla)
llvm-svn: 175942
-Change the build_dir variable name to lib_dir
-Set lib_dir to the correct location on Linux
-Set LD_EXTRAS to the actual lldb library
llvm-svn: 175664
- Add a "parsable" mode to dotest.py that outputs test results in exactly the same format as clang's lit tests
- Improve dosep script to output list of failing tests (output should look like clang test failure summaries)
- Cleanup lldb/test/Makefile to remove needless parameters and environment variables
- Switch makefile tests to use parsable-mode output; should make the buildbot results parsable
- Switch makefile tests to use dosep to log catch crashing tests (instead of halting the test suite)
llvm-svn: 175309
Added a new line of information that reports the count of tests that pass, fail or have other things happen to them.
Again no flag to have the dots back. If you care, let us know!
llvm-svn: 174784
The LLDB test suite now shows a progress bar instead of dots when not in verbose mode
If you crave the dots, make your Terminal window smaller than 10 columns :-)
(or ask for a flag to have the dots come back on demand)
llvm-svn: 174777
- now prints the correct PYTHONPATH
- update dotest.py to use lldb -P result correctly
- resolves TestPublicAPIHeaders test failure (on Linux)
llvm-svn: 171558
This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.
Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).
The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)
All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.
All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.
In the end, failures will be reported on a per-category basis, as well as in the usual format.
This is the very first stage of this feature. Feel free to chime in with ideas for improvements!
llvm-svn: 164403
Changed the '-A' option to also have a long option of '--arch'. This is now specified multiple times to get multiple architectures.
Old: -A i386^x86_64
New: -A i386 -A x86_64
--arch i386 --arch x86_64
Changed the '-C' option to also have a long option of '--compiler'. This is now specified multiple times to get multiple compiler.
Old: -C clang^gcc
New: -C clang -C gcc
--compiler clang --compiler gcc
llvm-svn: 163141
before running the test suite. A usage example looks like this:
test $ ./dotest.py -A x86_64 -R /tmp/x86_64 &
test $ ./dotest.py -A i386 -R /tmp/i386 &
where we would want to run the x86_64 and i386 archs concurrently but relocate the test suite to different directory
hierarchies in order not to stump on each other's intermediate files.
llvm-svn: 155491
rdar://problem/11283401
Example:
Collected 1 test
1: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
Test calling std::String member function. ... FAIL
======================================================================
FAIL: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
Test calling std::String member function.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 427, in wrapper
return func(self, *args, **kwargs)
File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 34, in test_with_dwarf
self.call_function()
File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 48, in call_function
substrs = ['Hello world'])
File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 1235, in expect
msg if msg else EXP_MSG(str, exe))
AssertionError: False is not True : 'Hello world' returns expected result
Config=i386-clang
----------------------------------------------------------------------
Ran 1 test in 1.148s
FAILED (failures=1)
llvm-svn: 155157
the pre-flight code gets executed during setUp() after the debugger instance is available
and the post-flight code gets executed during tearDown() after the debugger instance has
done killing the inferior and deleting all the target programs.
Example:
[11:32:48] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
config: {'pre_flight': <function pre_flight at 0x1098541b8>, 'post_flight': <function post_flight at 0x109854230>}
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-139
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154753
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154730
Last Changed Date: 2012-04-13 18:42:46 -0700 (Fri, 13 Apr 2012)
lldb.pre_flight: def pre_flight(test):
__import__("lldb")
__import__("lldbtest")
print "\nRunning pre-flight function:"
print "for test case:", test
lldb.post_flight: def post_flight(test):
__import__("lldb")
__import__("lldbtest")
print "\nRunning post-flight function:"
print "for test case:", test
Session logs for test failures/errors/unexpected successes will go into directory '2012-04-16-11_34_08'
Command invoked: python ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
compilers=['clang']
Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests
1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
Test a simple sequence of watchpoint creation and watchpoint hit. ...
Running pre-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
Running post-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
Test a simple sequence of watchpoint creation and watchpoint hit. ...
Running pre-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
Running post-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
----------------------------------------------------------------------
Ran 2 tests in 1.584s
OK
llvm-svn: 154847
Daniel Malea