Commit Graph

207 Commits

Author SHA1 Message Date
Daniel Malea 249287afde Minor test runner improvemenst
- rework the way SBDebugger.SetAsync() is used to avoid side effects (reset original value at TearDownHook)
- refactor expectedFailureClang (and add expectedFailureGcc decorator)
- mark TestChangeValueAPI.py as expectedFailureGcc due to PR-15039

llvm-svn: 175523
2013-02-19 16:08:57 +00:00
Daniel Malea 2dd69bb5f2 Fix misuse of python subprocess module (caused "leaking" processes and garbling the terminal)
- fixed cleanup of Popen objects by pushing spawn logic into test Base and out of test cases
- connect subprocess stdin to PIPE (rather than the parent's STDIN) to fix silent terminal issue

Tested on Linux and Mac OS X

llvm-svn: 175301
2013-02-15 21:21:52 +00:00
Daniel Malea 318cbcef91 Disable confirmation prompts for testing
- set auto-confirm to false when running TestExprs (avoid hang when using API)
- set prompt-on-quit to false in test helper (avoid timeout when using lldb CLI)

llvm-svn: 173485
2013-01-25 20:38:49 +00:00
Daniel Malea be230793fb Mark test cases affected by PR 15036 (GCC-generated DWARF causes parser crash)
- Add new decorator "@skipIfGcc" to lldbtest.py

llvm-svn: 173394
2013-01-24 23:52:09 +00:00
Jim Ingham 9732e08a4d The skipOnLinux decorator wasn't calling the test method correctly (no need to pass in the "self") resulting
in errors on MacOS X for the tests so decorated.

llvm-svn: 168662
2012-11-27 01:21:28 +00:00
Daniel Malea 179ff29811 Fix for TestSharedLib.py (on Linux)
- use lldb 'settings' command to help testcase find shared library
- pull up dyldPath variable from TestLoadUnload.py to fixture base class (applicable in multiple cases)

llvm-svn: 168612
2012-11-26 21:21:11 +00:00
Daniel Malea 93aec0f9a9 Update test status on Linux
- add decorators @expectedFailLinux and @skipOnLinux
- skip/mark xfail cases due to open bugzillas # 14323, 14416, 14423, 14424, 14425, 14426

Patch by Ashok Thirumurthi!

llvm-svn: 168529
2012-11-23 21:59:29 +00:00
Enrico Granata df26a90dc2 And one more logging message goes away
llvm-svn: 166628
2012-10-24 21:54:04 +00:00
Enrico Granata 1918627d93 Reverting unwanted changes to the test suite
llvm-svn: 166627
2012-10-24 21:44:48 +00:00
Enrico Granata 5020f958d8 Reverting the changes to Scalar since this class needs to follow C rules for type promotion
llvm-svn: 166626
2012-10-24 21:42:49 +00:00
Greg Clayton 5db6b79955 Patch from Ashok Thirumurthi that enabled FPU registers for POSIX x86_64.
llvm-svn: 166604
2012-10-24 18:24:14 +00:00
Enrico Granata 7e137e3d8f <rdar://problem/12481949> Fixing SBValue.GetValueAsSigned() to do the right thing when dealing with a 32-bit negative value
llvm-svn: 166603
2012-10-24 18:14:21 +00:00
Enrico Granata 44818163ed <rdar://problem/12523238> Commit 3 of 3
Changed all relevant test cases to verify that MightHaveChildren() works correctly for objects of interest
Added a bunch of convenience methods for test cases to use: target(), process(), thread() and frame() which mimic the lldb.X convenience variables
As a bonus, edited the documentation on the website to describe the new method available for synthetic children providers writers to implement!

That's all folks!

llvm-svn: 166535
2012-10-24 01:23:57 +00:00
Enrico Granata bc08ab450a <rdar://problem/12500212> Test case for the new plugin feature
llvm-svn: 166453
2012-10-23 00:09:02 +00:00
Sean Callanan 794baf6fb7 If a full path to a compiler is provided on the
command line to dotest.py, replace / with _ in
the logfile names that mention that compiler so
that we don't try to put log files in weird
places.

llvm-svn: 166038
2012-10-16 18:22:04 +00:00
Jim Ingham 63dfc725a7 Fix all the test case breakages caused by folks writing tests all over the place that depended explicitly
on the output of "break set".  Please don't do this sort of thing!!!!!

llvm-svn: 164433
2012-09-22 00:05:11 +00:00
Enrico Granata 165f8af8c5 Initial commit of a new testsuite feature: test categories.
This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.

Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).

The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)

All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.

All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.

In the end, failures will be reported on a per-category basis, as well as in the usual format.

This is the very first stage of this feature. Feel free to chime in with ideas for improvements!

llvm-svn: 164403
2012-09-21 19:10:53 +00:00
Johnny Chen 552d6719f2 Silence the "Command ... failed!" message when not running in trace mode.
You run with trace mode by passing '-t' to the test driver.

llvm-svn: 161130
2012-08-01 19:56:04 +00:00
Filipe Cabecinhas 0eec15acf2 Continue the cleanup started on r158737
Adds a utility class method to TestBase that checks and removes a temp
file.
Removed every use of system() to execute rm -f.

llvm-svn: 158809
2012-06-20 10:13:40 +00:00
Johnny Chen c98892ec1f Update the comments for lldbtest module. The test driver is the only way to run the test suite.
llvm-svn: 156943
2012-05-16 20:41:28 +00:00
Johnny Chen 92f162a798 The session file name should be tagged with (architecture, compiler) in addition
to the already existing (test result, test id) to avoid collision and to
facilitate postmortem analysis.

llvm-svn: 155148
2012-04-19 19:39:11 +00:00
Johnny Chen 44d2497138 Add the capability of supplying the pre/post-flight functions to the test suite such that
the pre-flight code gets executed during setUp() after the debugger instance is available
and the post-flight code gets executed during tearDown() after the debugger instance has
done killing the inferior and deleting all the target programs.

Example:

[11:32:48] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight  functionalities/watchpoint/hello_watchpoint
config: {'pre_flight': <function pre_flight at 0x1098541b8>, 'post_flight': <function post_flight at 0x109854230>}
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-139
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154753
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154730
Last Changed Date: 2012-04-13 18:42:46 -0700 (Fri, 13 Apr 2012)


lldb.pre_flight: def pre_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning pre-flight function:"
    print "for test case:", test

lldb.post_flight: def post_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning post-flight function:"
    print "for test case:", test


Session logs for test failures/errors/unexpected successes will go into directory '2012-04-16-11_34_08'
Command invoked: python ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok

----------------------------------------------------------------------
Ran 2 tests in 1.584s

OK

llvm-svn: 154847
2012-04-16 18:55:15 +00:00
Johnny Chen f1548d4f74 Add a new option to the test driver, -N dsym or -N dwarf, in order to exclude tests decorated with
either @dsym_test or @dwarf_test to be executed during the testsuite run.  There are still lots of
Test*.py files which have not been decorated with the new decorator.

An example:

# From TestMyFirstWatchpoint.py ->
class HelloWatchpointTestCase(TestBase):

    mydir = os.path.join("functionalities", "watchpoint", "hello_watchpoint")

    @dsym_test
    def test_hello_watchpoint_with_dsym_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDsym(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()

    @dwarf_test
    def test_hello_watchpoint_with_dwarf_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDwarf(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()


# Invocation ->
[17:50:14] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-137
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154133
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154109
Last Changed Date: 2012-04-05 10:43:02 -0700 (Thu, 05 Apr 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-04-05-17_50_49'
Command invoked: python ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... skipped 'dsym tests'
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.138s

OK (skipped=1)
Session logs for test failures/errors/unexpected successes can be found in directory '2012-04-05-17_50_49'
[17:50:50] johnny:/Volumes/data/lldb/svn/ToT/test $ 

llvm-svn: 154154
2012-04-06 00:56:05 +00:00
Johnny Chen 531c085815 Fix a bug in getRunOptions() where there was no space between the arch spec and the compiler spec.
llvm-svn: 152941
2012-03-16 20:44:00 +00:00
Johnny Chen 3bc8ae4014 Add a test case to go with <rdar://problem/11052829> and svn check-in r152809.
llvm-svn: 152825
2012-03-15 19:10:00 +00:00
Johnny Chen ac25713f60 During the test case tearDown(), give it one final blow to make sure the child process spawned by pexpect is terminated.
Fix the issue of many '(lldb)' zombie processes observed by Jim.

llvm-svn: 151583
2012-02-27 23:07:40 +00:00
Johnny Chen 57816730f0 Add safe guard for when the 'expect' program cannot be located and skip the test.
llvm-svn: 150133
2012-02-09 02:01:59 +00:00
Johnny Chen 15f247ad8c Add test cases for APIs to get template arguments from an SBType.
llvm-svn: 149707
2012-02-03 20:43:00 +00:00
Johnny Chen fdc80a5cf7 lldb should warn when dSYM does not match the binary.
o Symbols.cpp:

  Emit a warning message when dSYM does not match the binary.

o warnings/uuid:

  Added regression test case.

o lldbtest.py:

  Modified to allow test case writer to demand that the build command does not begin
  with a clean first; required to make TestUUIDMismatchWanring.py work.

rdar://problem/10515708

llvm-svn: 149465
2012-02-01 01:49:50 +00:00
Johnny Chen 98aceb08f8 o CommandObjectSettingsSet.cpp:
Fix a bug where "settings set -r th" wouldn't complete.

o UserSettingsController.cpp:

  Fix a bug where "settings set target.process." wouldn't complete.

o test/functionalities/completion:

  Add various completion test cases related to 'settings set' command.

llvm-svn: 148596
2012-01-20 23:02:51 +00:00
Johnny Chen a33843f5b4 Decorate the two test cases in TestReturnValue.py as i386 only expectedFailure, aka @expectedFailurei386.
llvm-svn: 147177
2011-12-22 21:14:31 +00:00
Johnny Chen 5daa6de433 Let's also record the compiler version used for compiling the inferior into the session info
llvm-svn: 145732
2011-12-03 00:16:59 +00:00
Johnny Chen 0fddfb2ceb Add an option '-S' to skip the build and cleanup while running the test.
Use this option with care as you would need to build the inferior(s) by hand
and build the executable(s) with the correct name(s).  This option can be used
with '-# n' to stress test certain test cases for n number of times.

An example:

[11:55:11] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ ls
Makefile		TestValueAPI.pyc	linked_list
TestValueAPI.py		change_values		main.c
[11:55:14] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ make EXE=test_with_dsym
clang -gdwarf-2 -O0  -arch x86_64   -c -o main.o main.c
clang -gdwarf-2 -O0  -arch x86_64   main.o -o "test_with_dsym"
/usr/bin/dsymutil  -o "test_with_dsym.dSYM" "test_with_dsym"
[11:55:20] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ cd ../..
[11:55:24] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
LLDB build dir: /Volumes/data/lldb/svn/trunk/build/Debug
LLDB-89
Path: /Volumes/data/lldb/svn/trunk
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 144914
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 144911
Last Changed Date: 2011-11-17 09:22:31 -0800 (Thu, 17 Nov 2011)



Session logs for test failures/errors/unexpected successes will go into directory '2011-11-17-11_55_29'
Command invoked: python ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
----------------------------------------------------------------------
Collected 1 test

1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.163s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.200s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.198s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.239s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.215s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.105s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.098s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.195s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.197s

OK
[11:55:34] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 144919
2011-11-17 19:57:27 +00:00
Johnny Chen 8eb14a9500 Docstring clarification.
llvm-svn: 144848
2011-11-16 22:44:28 +00:00
Johnny Chen 0bd8c311cb Also dump the pid of the process launching through the lldbtest.system(*popenargs, **kwargs) API.
This helps track down possible zombie processes.

llvm-svn: 144846
2011-11-16 22:41:53 +00:00
Johnny Chen 2a80858903 Modify lldbtest.Base.runHooks() to now take the following keyword arguments:
child=None, child_prompt=None, use_cmd_api=False

By default, expect a pexpect spawned child and child prompt to be
supplied (use_cmd_api=False).  If use_cmd_api is true, ignore the child
and child prompt and use self.runCmd() to run the hooks one by one.

Modify existing client to reflect the change.

llvm-svn: 142532
2011-10-19 16:48:07 +00:00
Johnny Chen a737ba55af Extract the run hooks functionality into the base class lldbtest.Base.
llvm-svn: 142469
2011-10-19 01:06:21 +00:00
Johnny Chen ebe51726b8 If we spawn an lldb process for test (via pexpect), do not load the init file unless told otherwise.
Set up self.lldbOption to be "--no-lldbibit" unless env variable NO_LLDBIBIT is defined and equals "NO".
Also add "-nx" to gdb spawned.

llvm-svn: 141384
2011-10-07 19:21:09 +00:00
Johnny Chen 86268e4459 o lldbtest.py:
Add a keyword argument 'endstr' to TestBase.expect() method to assert that the output
will end with 'endstr'.

Add TestBase.switch_to_thread_with_stop_reason(stop_reason) to select the thread with
the stop reason = 'stop_reason' as the current thread.

o TestWatchLocation.py:

Modified to switch to the stopped thread with stop reason = watchpoint and to evaluate
an expression with expected output for stronger assertion.

llvm-svn: 140890
2011-09-30 21:48:35 +00:00
Johnny Chen f68cc12453 Add a simple watchpoint test to exercise watchpoint creation followed by watchpoint hit events.
llvm-svn: 139847
2011-09-15 21:09:59 +00:00
Johnny Chen d890bfc962 Add a new attribute self.lldbHere, representing the fullpath to the 'lldb' executable
built locally from the source tree.  This is distinguished from self.lldbExec, which
can be used by test/benchmarks to measure the performances against other debuggers.

You can use environment variable LLDB_EXEC to specify self.lldbExec to the dotest.py
test driver, otherwise it is going to be populated with self.lldbHere.

Modify the regular tests under test dir, i.e., not test/benchmarks, to use self.lldbHere.
Also modify the benchmarks tests to use self.lldbHere when it needs an 'lldb' executable
with debug info to do the performance measurements.

llvm-svn: 138608
2011-08-26 00:00:01 +00:00
Johnny Chen b7bdd1001e Pretty print the run options for dumpSessionInfo(self) client.
llvm-svn: 138466
2011-08-24 19:48:51 +00:00
Johnny Chen b5825b82ad Minor change for the @expectedFailureClang logic. For the non-clang test failure, re-raise
the exception to get more stack trace information.

llvm-svn: 138021
2011-08-19 01:17:09 +00:00
Johnny Chen 31963cea0a Add a decorator for marking clang only expectedFailure. Use it for the test_step_over_3_times_with_dsym/dwarf()
test cases in TestThreadAPI.py by decorating it with @expectedFailureClang.

Example:

    @expectedFailureClang
    @python_api_test
    def test_step_over_3_times_with_dwarf(self):
        """Test Python SBThread.StepOver() API."""
        # We build a different executable than the default buildDwarf() does.
        d = {'CXX_SOURCES': 'main2.cpp', 'EXE': self.exe_name}
        self.buildDwarf(dictionary=d)
        self.setTearDownCleanup(dictionary=d)
        self.step_over_3_times(self.exe_name)

llvm-svn: 138019
2011-08-19 00:54:27 +00:00
Johnny Chen 0241f1441f Add comment about recent change.
llvm-svn: 137776
2011-08-16 22:06:17 +00:00
Johnny Chen bfcf37f18b Fix the test suite error of:
...

  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 243, in __init__
    self.session = test.session if test else None
AttributeError: 'StdCXXDisassembleTestCase' object has no attribute 'session'

llvm-svn: 137724
2011-08-16 17:06:45 +00:00
Johnny Chen f79b076783 Forgot to initialize the self.__skipped__ attribute lldbtest.Base.
llvm-svn: 137696
2011-08-16 00:48:58 +00:00
Johnny Chen c5cc625c79 Test driver should also report skipped tests because there were cases when tests were skipped
due to incorrect skip-logic.

llvm-svn: 137678
2011-08-15 23:09:08 +00:00
Johnny Chen 9f4f5d94cd Add TestInferiorChanged.py to test that lldb reloads the inferior after it was changed during the debug session.
First, main.c causes a crash, the inferior then gets re-built with main2.c which is not crashing.

Add Base.cleanup(self, dictionary=None) for platform specfic way to do cleanup after build.
This plugin method is used by the above test case to cleanup the main.c build before rebuild for main2.c.

llvm-svn: 137500
2011-08-12 20:19:22 +00:00
Johnny Chen 8082a00dcb When a benchmarks test fails, the re-run command should include the '+b' option instead of the '-t' option.
llvm-svn: 137285
2011-08-11 00:16:28 +00:00