The main dotest.py should exit with a system return code of 1 on any
issue. This change fixes a place where I omitted counting the
exceptional exit value to determine if we should return 1 when using the
new summary results.
This change also puts a banner around the Issue Details section that comes
before the Test Result Summary.
llvm-svn: 255138
The results formatter system is now fed timeouts and exceptional process
exits (i.e. inferior dotest.py process that exited by signal on POSIX
systems).
If a timeout or exceptional exit happens while a test method is running
on the worker queue, the timeout or exceptional exit is charged and
reported against that test method. Otherwise, if no test method was
running at the time of the timeout or exceptional exit, only the test
filename will be reported as the TIMEOUT or ERROR.
Implements:
https://llvm.org/bugs/show_bug.cgi?id=24830https://llvm.org/bugs/show_bug.cgi?id=25703
In support of:
https://llvm.org/bugs/show_bug.cgi?id=25450
llvm-svn: 255097
There is already a class called LLDBTestResults which I would like
to move into a separate file, but the most appropriate filename
was taken.
llvm-svn: 254946
Test summary counts now show at the end, with issue details
above.
Issue details now print "ISSUE_TYPE: test method (relative path)".
Relative paths are relative to the
packages/Python/lldbsuite/test directory.
Sample output:
test/dotest.py --executable `pwd`/build/Debug/lldb --results-formatter lldbsuite.test.basic_results_formatter.BasicResultsFormatter --threads 12
Testing: 415 test suites, 12 threads
415 out of 415 test suites processed - TestLldbGdbServer.py
Test Results
Total Test Methods Run (excluding reruns): 2470
Test Method rerun count: 0
Details:
UNEXPECTED SUCCESS: test_symbol_name_dsym (functionalities/completion/TestCompletion.py)
UNEXPECTED SUCCESS: test_symbol_name_dwarf (functionalities/completion/TestCompletion.py)
===================
Test Result Summary
===================
Success: 1329
Expected Failure: 79
Failure: 0
Error: 0
Unexpected Success: 2
Skip: 1060
llvm-svn: 254890
Also cleans up some usages of strings where symbolic names
were safer and made more sense.
Try a test run with something like this to check out the new
basic results formatter (not used by default):
time test/dotest.py --executable `pwd`/build/Debug/lldb --results-formatter lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file stdout
This will yield something like:
Testing: 1 test suites, 8 threads
1 out of 1 test suites processed - TestHelp.py
Test Results
Total Test Methods Run (excluding reruns): 13
Test Method rerun count: 0
===================
Test Result Summary
===================
Success: 13
Expected Failure: 0
Failure: 0
Error: 0
Unexpected Success: 0
Skip: 0
Whereas something with a bit of error will look more like this:
42 out of 42 test suites processed - TestSymbolTable.py
Test Results
Total Test Methods Run (excluding reruns): 166
Test Method rerun count: 0
===================
Test Result Summary
===================
Success: 93
Expected Failure: 10
Failure: 2
Error: 2
Unexpected Success: 0
Skip: 59
Details:
FAIL:
TestModulesInlineFunctions.ModulesInlineFunctionsTestCase.test_expr_dsym
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/modules-inline-functions/TestModulesInlineFunctions.py)
FAIL:
TestModulesInlineFunctions.ModulesInlineFunctionsTestCase.test_expr_dwarf
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/modules-inline-functions/TestModulesInlineFunctions.py)
ERROR: TestObjCCheckers.ObjCCheckerTestCase.test_objc_checker_dsym
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/objc-checker/TestObjCCheckers.py)
ERROR: TestObjCCheckers.ObjCCheckerTestCase.test_objc_checker_dwarf
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/objc-checker/TestObjCCheckers.py)
The Details header only prints if there are any issues to report. The
Details section has tags that should get picked up using the normal
issue text scrapers (e.g. buildbot).
Test numbers reported are strictly test method runs.
The rerun bit at the top is in support of the multi-pass test
runner code (to run the low-load, single worker test pass for
tests that failed the first run), which I'll be able to put up
for review after this.
ResultsFormatters now have the ability to indicate they replace
the legacy summary, as this one does.
Once we come to agreement on the exact format, I will switch
us over to using this by default.
llvm-svn: 254530
