Commit Graph

153 Commits

Author SHA1 Message Date
Dan Liew
7efde3c440 [lit] Remove a timing senstive part of shtest-timeout.py
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.

The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.

We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.

rdar://problem/38774530

llvm-svn: 328702
2018-03-28 13:55:08 +00:00
Mircea Trofin
56ba71b2a7 Revert "Revert "[lit] Generalized /dev/null support on Windows.""
Summary:
This reverts commit r328596.

Checking if the arguments are strings before testing if they contain "/dev/null".

Reviewers: rnk

Reviewed By: rnk

Subscribers: delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D44914

llvm-svn: 328603
2018-03-27 01:39:17 +00:00
Jan Korous
1e0e0b077d [lit] Temporarily disable shtest-timeout.py on darwin
Disabled until fixed in order to avoid random failures on green dragon.

rdar://problem/38774530

llvm-svn: 328598
2018-03-27 00:16:28 +00:00
Mircea Trofin
373c445c24 Revert "[lit] Generalized /dev/null support on Windows."
This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92.

llvm-svn: 328596
2018-03-26 23:59:39 +00:00
Mircea Trofin
88911686c8 [lit] Generalized /dev/null support on Windows.
Generalized /dev/null remapping on Windows, and added test.

Reviewers: rnk

Reviewed By: rnk

Subscribers: amccarth, zturner, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D44771

llvm-svn: 328589
2018-03-26 22:41:06 +00:00
Reid Kleckner
8252892951 [lit] Implement 'cat' command for internal shell
Fixes PR36449

Patch by Chamal de Silva

Differential Revision: https://reviews.llvm.org/D43501

llvm-svn: 328563
2018-03-26 18:05:12 +00:00
Brian Homerding
d5c558ff21 [lit] - Allow 1 test to report multiple micro-test results to provide support for microbenchmarks.
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.

This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.

This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.

Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).

These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.


Reviewers: MatzeB, hfinkel, rengolin, delcypher

Differential Revision: https://reviews.llvm.org/D43314

llvm-svn: 327422
2018-03-13 16:37:59 +00:00
Lei Huang
9db7bd5cab Losen time contraint to accommodate system loads
llvm-svn: 326359
2018-02-28 20:05:24 +00:00
Max Moroz
975eacada5 [lit] Implement "-r" option for builtin "diff" command + a test using that.
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.

Reviewers: zturner, morehouse, vsk

Reviewed By: zturner

Subscribers: kcc, llvm-commits

Differential Revision: https://reviews.llvm.org/D41776

llvm-svn: 322102
2018-01-09 18:23:34 +00:00
Ying Yi
cfb08e9e6b [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands.
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’
commands as the internal shell builtins.

Reviewed by: Zachary Turner, Reid Kleckner

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39567

llvm-svn: 319528
2017-12-01 09:54:27 +00:00
Greg Bedwell
34a83f0faf Make check-lit tests respect LLVM_LIT_TOOLS_DIR
Differential Revision: https://reviews.llvm.org/D40520

llvm-svn: 319329
2017-11-29 18:05:26 +00:00
Ying Yi
a0903c6e5d Reverted rL318911 since it broke the sanitizer-windows.
llvm-svn: 318914
2017-11-23 13:23:21 +00:00
Ying Yi
989c9e75a6 [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. 
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ 
commands as the internal shell builtins.

Reviewers: Zachary Turner, Reid Kleckner

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39567

llvm-svn: 318911
2017-11-23 12:48:41 +00:00
Jordan Rose
e55d9a1403 Revert "lit.py: Add new %{shared_output(LABEL)} substitution"
This reverts r315697 and my ill-fated attempts to fix it on Windows.
I'll try again when I get access to a Windows machine.

llvm-svn: 315793
2017-10-14 04:01:27 +00:00
Jordan Rose
a33a17605d lit.py: Give up and disable the new shared-output.py test on Windows
"No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp"

And yet other forward-slashes don't seem to be causing the same
problem. I'll see if I can get ahold of a Windows machine to poke at
this directly later.

llvm-svn: 315792
2017-10-14 03:33:46 +00:00
Jordan Rose
fddfb5ad4c lit.py: Previous test fix was a red herring; backslashes are escapes
I don't have access to a Windows machine at the moment, so if this
doesn't fix it I'll just revert for now.

llvm-svn: 315782
2017-10-14 00:49:19 +00:00
Jordan Rose
aafbc1c4db lit.py: Fix new test harder for systems that don't use / as os.path.sep
I didn't think about '%{inputs}' having the same problem. This one
should be a fully Windows path name.

llvm-svn: 315779
2017-10-14 00:26:48 +00:00
Jordan Rose
f4fd5938e7 lit.py: Fix new test for systems that don't use / as os.path.sep
llvm-svn: 315773
2017-10-13 22:46:40 +00:00
Jordan Rose
53be3224b2 lit.py: Add new %{shared_output(LABEL)} substitution
This refers to a temporary path that can be shared across all tests,
identified by a particular label. This can be used for things like
caches.

At the moment, the character set for the LABEL is limited to C
identifier characters, plus '-', '+', '=', and '.'. This is the same
set of characters currently allowed in REQUIRES clause identifiers.

llvm-svn: 315697
2017-10-13 16:12:23 +00:00
Michal Gorny
d6a4c79b14 [lit] Fix running lit tests in unconfigured source dir
Fix llvm_tools_dir attribute access not to fail when the variable is not
present. This directory is not really necessary to run lit tests,
and the code already accounts for it being None.

The reference was added in r313407, and it breaks the stand-alone lit
package in Gentoo.

Differential Revision: https://reviews.llvm.org/D38442

llvm-svn: 314620
2017-10-01 07:13:25 +00:00
Zachary Turner
1ca789bdba [lit] Actually do normalize the case of files in the config map.
This has gone back and forth, but it seems this is necessary
after all.  realpath is not sufficient because if you have a
file named 'C:\foo.txt', then both realpath('c:\foo.txt') and
realpath(C:\foo.txt') return the string that was passed to them
exactly as is, meaning the case of the drive-letter won't match.

The problem before was not that we were normalizing the case of
items going into the config map, but rather that we were
normalizing the case of something we needed to print.  The value
that is used to key on the config map should never be printed.

llvm-svn: 313918
2017-09-21 21:27:11 +00:00
Zachary Turner
43bcf226c1 [lit] Don't norm case when inserting into the config map.
This makes all paths lowercase on Windows, which seemed like a
good idea at the time, but it means that tests can't properly
use FileCheck to match expected path names.

llvm-svn: 313889
2017-09-21 17:02:08 +00:00
Zachary Turner
71deeee593 [lit] Add a test for the builtin config map.
Config map is not exposed through the command line, so testing this
is somewhat tricky.  But basically we need a test that if a custom
driver builds a config map and passes it to main, it gets respected.

A config map allows config files in the source tree to be mapped
to alternate config files in the build tree.  This particular test
works by having two config files in separate directories, and
setting up a config map to have that redirects A/lit.site.cfg
to B/altconfig.  Then, we print a message in A/lit.site.cfg
and B/altconfig and check that we do see the output from B
but don't see the output from A.  Additionally we test that
the test suite specified by A's config map is properly discovered.

Differential Revision: https://reviews.llvm.org/D38105

llvm-svn: 313887
2017-09-21 16:18:28 +00:00
Zachary Turner
957d611575 [lit] Make lit support config files with .py extension.
Many editors and Python-related diagnostics tools such as
debuggers break or fail in mysterious ways when python files
don't end in .py.  This is especially true on Windows, but
still exists on other platforms.  I don't want to be too heavy
handed in changing everything across the board, but I do want
to at least *allow* lit configs to have .py extensions.  This
patch makes the discovery process first look for a config file
with a .py extension, and if one is not found, then looks for
a config file using the old method.  So for existing users, there
should be no functional change.

Differential Revision: https://reviews.llvm.org/D37838

llvm-svn: 313849
2017-09-21 00:24:52 +00:00
Zachary Turner
f734daf36d [lit] Fix the lit unit tests.
A few tests were manually constructing a LitConfig object, since
I added a new argument to it this was triggering some failures
I didn't detect.  `ninja check-lit` passes now.

llvm-svn: 313461
2017-09-16 15:48:30 +00:00
Zachary Turner
ce92db13ea Resubmit "[lit] Force site configs to run before source-tree configs"
This is a resubmission of r313270.  It broke standalone builds of
compiler-rt because we were not correctly generating the llvm-lit
script in the standalone build directory.

The fixes incorporated here attempt to find llvm/utils/llvm-lit
from the source tree returned by llvm-config.  If present, it
will generate llvm-lit into the output directory.  Regardless,
the user can specify -DLLVM_EXTERNAL_LIT to point to a specific
lit.py on their file system.  This supports the use case of
someone installing lit via a package manager.  If it cannot find
a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or
invalid, then we print a warning that tests will not be able
to run.

Differential Revision: https://reviews.llvm.org/D37756

llvm-svn: 313407
2017-09-15 22:10:46 +00:00
Zachary Turner
83dcb68468 Revert "[lit] Force site configs to run before source-tree configs"
This patch is still breaking several multi-stage compiler-rt bots.
I already know what the fix is, but I want to get the bots green
for now and then try re-applying in the morning.

llvm-svn: 313335
2017-09-15 02:56:40 +00:00
Zachary Turner
a0e55b6403 [lit] Force site configs to be run before source-tree configs
This patch simplifies LLVM's lit infrastructure by enforcing an ordering
that a site config is always run before a source-tree config.

A significant amount of the complexity from lit config files arises from
the fact that inside of a source-tree config file, we don't yet know if
the site config has been run.  However it is *always* required to run
a site config first, because it passes various variables down through
CMake that the main config depends on.  As a result, every config
file has to do a bunch of magic to try to reverse-engineer the location
of the site config file if they detect (heuristically) that the site
config file has not yet been run.

This patch solves the problem by emitting a mapping from source tree
config file to binary tree site config file in llvm-lit.py. Then, during
discovery when we find a config file, we check to see if we have a
target mapping for it, and if so we use that instead.

This mechanism is generic enough that it does not affect external users
of lit. They will just not have a config mapping defined, and everything
will work as normal.

On the other hand, for us it allows us to make many simplifications:

* We are guaranteed that a site config will be executed first
* Inside of a main config, we no longer have to assume that attributes
  might not be present and use getattr everywhere.
* We no longer have to pass parameters such as --param llvm_site_config=<path>
  on the command line.
* It is future-proof, meaning you don't have to edit llvm-lit.in to add
  support for new projects.
* All of the duplicated logic of trying various fallback mechanisms of
  finding a site config from the main config are now gone.

One potentially noteworthy thing that was required to implement this
change is that whereas the ninja check targets previously used the first
method to spawn lit, they now use the second. In particular, you can no
longer run lit.py against the source tree while specifying the various
`foo_site_config=<path>` parameters.  Instead, you need to run
llvm-lit.py.

Differential Revision: https://reviews.llvm.org/D37756

llvm-svn: 313270
2017-09-14 16:47:58 +00:00
Reid Kleckner
9da05740d3 [lit] Fix some issues with short paths in test discovery introduced in r312254
llvm-svn: 312288
2017-08-31 21:56:14 +00:00
Victor Leschuk
9f11c0bddf Make lit :: shtest-format.py supported on Windows again
It was marked as unsupported on Windows in r311230 because on some Win10 
machines it failed or caused hang. The problem was that on these machines
system bash (C:\Windows\System32\bash.exe) was used which requires paths to be
passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'.

TODO: we should make lit detect if system bash is used instead of msys and set
appropriate path format.

llvm-svn: 311558
2017-08-23 14:59:09 +00:00
Victor Leschuk
59dc64f3af Temporary mark lit :: shtest-format as unsupported on windows
When run manually it fails, but when run under buildbot it causes hang.

llvm-svn: 311230
2017-08-19 07:58:07 +00:00
Ben Dunbobbin
7b76de2dcb [lit] support unsetting env variables (again!)
This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton.

The patch was accepted but not landed.

This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour.

Differential Revision: https://reviews.llvm.org/D36403

llvm-svn: 311180
2017-08-18 17:32:57 +00:00
Greg Bedwell
fd2c379568 Fix check-lit compatibility with multi-config CMake generators
Multi-configuration CMake generators such as those for Visual Studio or Xcode do not
specify a build config at configure time, but let the user choose at build
time.  In these cases binaries go into build/${Configuration}/bin rather than
build/bin.  Prior to this commit, check-lit would fail when using multi-configuration
generators as it did not know how to resolve ${Configuration} in order
to find tools such as FileCheck.  This commit teaches it to resolve
llvm_tools_dir within lit using the value specified with --param
build_mode.

Differential Revision: https://reviews.llvm.org/D36263

llvm-svn: 309967
2017-08-03 17:55:54 +00:00
Reid Kleckner
7a2274aaed Revert "[lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs"
This reverts r309602, check-lit still leaves Output directories in the
source directory.

llvm-svn: 309833
2017-08-02 17:16:25 +00:00
Reid Kleckner
1b4e9ae384 [lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs
Summary:
This is an alternative solution to running the lit test suite on bots
without polluting the source directory. Each input test suite gets an
auto-generated site config in the build directory that points back to
the test input source directory.

This adds some cmake comlexity, but now we don't need to remove and
re-copy the test input directory before every test.

Reviewers: delcypher, modocache

Subscribers: mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D36026

llvm-svn: 309602
2017-07-31 18:45:44 +00:00
NAKAMURA Takumi
15b5c8b206 lit::shtest-format.py: Make write-bad-encoding.py py3-aware.
Traceback (most recent call last):
    File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module>
      sys.stdout.write(b"a line with bad encoding: \xc2.")

sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts.

llvm-svn: 309473
2017-07-29 02:52:56 +00:00
Reid Kleckner
67de34897c [lit] Use a %{python} substitution to avoid relying on python being on PATH
llvm-svn: 309434
2017-07-28 21:13:47 +00:00
Reid Kleckner
125c74bc56 [lit] Dump some FileCheck inputs to try to debug some failing tests
llvm-svn: 309400
2017-07-28 16:24:18 +00:00
Reid Kleckner
432914bba0 [lit] Fix shtest-format external_shell failures
When using win32 cmd.exe, turn off command echoing at the beginning of
the script (@echo off).

Replace a bash shell script with a python script for the
fail_with_bad_encoding test.

llvm-svn: 309399
2017-07-28 16:13:02 +00:00
Reid Kleckner
4ca8d21ef3 [lit] Port googletest lit tests to Windows
Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a valid win32 executable.

Instead, rename all the python scripts masquerading as gtest executables
to have .py extensions, so we can easily detect then and call the python
executable for them. Do this on Linux as well as Windows for
consistency.

The test suite directory names also come out in lower-case on Windows.
We can consider removing that in a later patch. This change just updates
the FileCheck lines to match on Windows.

Fixes PR33933

Reviewers: modocache, mgorny

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35909

llvm-svn: 309347
2017-07-28 01:05:55 +00:00
Brian Gesiak
f0850e3ddb Un-revert "Teach the CMake build system to run lit's test suite. These can be run"
Summary:
Depends on https://reviews.llvm.org/D35879.

This reverts rL257268, which in turn was a revert of rL257221.
https://reviews.llvm.org/D35879 marks the tests in the lit test suite
that fail on Windows as XFAIL, which should allow these tests to pass
on Windows-based buildbots.

Reviewers: delcypher, beanz, mgorny, jroelofs, rnk

Reviewed By: mgorny

Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits

Differential Revision: https://reviews.llvm.org/D35880

llvm-svn: 309310
2017-07-27 19:18:35 +00:00
Brian Gesiak
d256538b3a [lit] Fix order of checks in shtest-shell.py test
Summary:
An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt`
expects that first a string printed to stdout is seen, and then a
string printed to stderr. Add `flush()` calls to ensure that stdout is
printed before stderr, as expected.

Reviewers: rnk, mgorny, jroelofs

Reviewed By: rnk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35947

llvm-svn: 309292
2017-07-27 16:50:40 +00:00
Reid Kleckner
c9e700042b [lit] Fix race between shtest-shell and max-failures tests
Previously these tests would use the same Output directory leading to
flaky non-deterministic failures.

llvm-svn: 309227
2017-07-26 22:57:32 +00:00
Reid Kleckner
6dac9134f9 [lit] Fix shtest-shell and max-failures lit tests on Windows
Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell
scripts as python scripts and call python on them.

Fixes PR33940

llvm-svn: 309200
2017-07-26 22:21:25 +00:00
Reid Kleckner
fa83d5a398 [lit] Fix shtest-output-printing.py on Windows by matching either / or \\
Fixes PR33938

llvm-svn: 309198
2017-07-26 22:11:30 +00:00
Reid Kleckner
f63d4d121b [lit] Fix discovery.py on Windows by matching backslashes when necessary
Fixes PR33932

llvm-svn: 309194
2017-07-26 22:00:38 +00:00
Reid Kleckner
cb98c14f89 [lit] Un-XFAIL selecting.py test on Windows
This passes locally for me, which fails the overall lit test suite. I
can't debug a passing test, but I will try to help debug the test when
we get some failing logs.

llvm-svn: 309190
2017-07-26 21:48:41 +00:00
Reid Kleckner
0ff0eacf32 Un-XFAIL some internal lit tests on Windows, they pass for me locally
llvm-svn: 309144
2017-07-26 18:04:18 +00:00
George Karpenkov
2fae2465c1 Fix LIT test breakage
Differential Revision: https://reviews.llvm.org/D35867

llvm-svn: 309140
2017-07-26 17:19:36 +00:00
Brian Gesiak
0787253cb6 [lit] Mark several of lit's tests XFAIL on Windows
Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.

In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.

Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:

```
utils/lit/lit.py \
    --path /path/to/your/llvm/build/bin \
    utils/lit/tests
```

Verify that the test suite is run and a successful exit code is
returned.

Reviewers: mgorny, rnk, delcypher, beanz

Reviewed By: rnk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D35879

llvm-svn: 309123
2017-07-26 15:10:50 +00:00
George Karpenkov
22a402fb66 [lit] Modify LIT to accept environment variable LIT_FILTER to select tests.
This is especially useful when lit is invoked indirectly by the build
system, and additional arguments can not be easily specified.

Differential Revision: https://reviews.llvm.org/D35091

llvm-svn: 307339
2017-07-07 00:22:11 +00:00
Reid Kleckner
578c36d952 [lit] Implement timeouts and max_time for process pool testing
This is necessary to pass the lit test suite at llvm/utils/lit/tests.

There are some pre-existing failures here, but now switching to pools
doesn't regress any tests.

I had to change test-data/lit.cfg to import DummyConfig from a module to
fix pickling problems, but I think it'll be OK if we require test
formats to be written in real .py modules outside lit.cfg files.

I also discovered that in some circumstances AsyncResult.wait() will not
raise KeyboardInterrupt in a timely manner, but you can pass a non-zero
timeout to work around this. This makes threading.Condition.wait use a
polling loop that runs through the interpreter, so it's capable of
asynchronously raising KeyboardInterrupt.

llvm-svn: 299605
2017-04-06 00:38:28 +00:00
Brian Gesiak
600f04a435 lit: remove python2-isms
Summary:
`assert.assertItemEqual` went away in Python 3. Seeing how lists
are ordered, comparing a list against each other should work just
as well.

Patch by @jbergstroem (Johan Bergström).

Reviewers: modocache, gparker42

Reviewed By: modocache

Differential Revision: https://reviews.llvm.org/D31229

llvm-svn: 298479
2017-03-22 04:23:01 +00:00
Greg Parker
17db7704cd Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL
and UNSUPPORTED"

This reverts the revert in r292942.

llvm-svn: 293007
2017-01-25 02:26:03 +00:00
Alex Lorenz
9111cc217d Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL
and UNSUPPORTED"

After r292904 llvm-lit fails to emit the test results in the XML format for
Apple's internal buildbots.

rdar://30164800

llvm-svn: 292942
2017-01-24 16:17:04 +00:00
Greg Parker
ed0a95cbec [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED
A `lit` condition line is now a comma-separated list of boolean expressions. 
Comma-separated expressions act as if each expression were on its own 
condition line:
For REQUIRES, if every expression is true then the test will run. 
For UNSUPPORTED, if every expression is false then the test will run. 
For XFAIL, if every expression is false then the test is expected to succeed. 
As a special case "XFAIL: *" expects the test to fail.

Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux 
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows

Syntax: 
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier. 
* Substrings of the target triple are true identifiers for UNSUPPORTED 
 and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+

Differential Revision: https://reviews.llvm.org/D18185

llvm-svn: 292904
2017-01-24 09:58:02 +00:00
Greg Parker
d972882f06 Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"
This change needs to be better-coordinated with libc++.

llvm-svn: 292900
2017-01-24 08:58:20 +00:00
Greg Parker
2ab45201e7 [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED
A `lit` condition line is now a comma-separated list of boolean expressions. 
Comma-separated expressions act as if each expression were on its own 
condition line:
For REQUIRES, if every expression is true then the test will run. 
For UNSUPPORTED, if every expression is false then the test will run. 
For XFAIL, if every expression is false then the test is expected to succeed. 
As a special case "XFAIL: *" expects the test to fail.

Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux 
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows

Syntax: 
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier. 
* Substrings of the target triple are true identifiers for UNSUPPORTED 
  and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+

Differential Revision: https://reviews.llvm.org/D18185

llvm-svn: 292896
2017-01-24 08:45:50 +00:00
Graydon Hoare
ae5d7bb4f5 [lit] Support sharding testsuites, for parallel execution.
Summary:
This change equips lit.py with two new options, --num-shards=M and
--run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD).

The options must be used together, and N must be in 1..M.

Together these options effect only test selection: they partition the testsuite
into M equal-sized "shards", then select only the Nth shard. They can be used
in a cluster of test machines to achieve a very crude (static) form of
parallelism, with minimal configuration work.

Reviewers: modocache, ddunbar

Reviewed By: ddunbar

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D28789

llvm-svn: 292417
2017-01-18 18:12:20 +00:00
Eric Fiselier
df87d070c9 [lit] Support custom parsers in parseIntegratedTestScript
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.

This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.

As an example after this change libc++ can implement the `MODULES-DEFINES` simply using: 
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```


Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs

Subscribers: mgrang, llvm-commits, cfe-commits

Differential Revision: https://reviews.llvm.org/D27005

llvm-svn: 288694
2016-12-05 20:21:21 +00:00
Michal Gorny
1d62f4ab15 [lit] Fix test shtest-timeout.py for modern output
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjusting the tests.

Differential Revision: https://reviews.llvm.org/D25236

llvm-svn: 284057
2016-10-12 21:40:08 +00:00
Brian Gesiak
8d89c3c6f7 [lit] Run unit tests as part of lit test suite
Summary:
The Python file `utils/lit/lit/ShUtil.py` contains:

1. Logic used by lit itself
2. A set of unit tests for that logic, which can be run by invoking
  `python utils/lit/lit/ShUtil.py`

Move these unit tests to a `tests/unit` subdirectory of lit, and run
the tests as part of lit's test suite. This ensures that, should the
lit test suite be included in LLVM's own regression test suite, these
unit tests will also be run.

(Instructions on how to run lit's test suite can be found in
`utils/lit/README.txt`.)

Reviewers: ddunbar, echristo, delcypher, beanz

Subscribers: mehdi_amini, llvm-commits

Differential Revision: https://reviews.llvm.org/D25411

llvm-svn: 283968
2016-10-12 03:35:04 +00:00
Chris Bieneman
1f49329e81 [lit] Use argparse instead of optparse
Summary:
optparse is deprecated in Python 2.7, which is the minimum version of
Python required to run the LLVM test suite. Replace its usage in lit
with argparse, optparse's 2.7 replacement module.

argparse has several benefits over optparse, but this commit does not
make use of those benefits yet. Instead, it simply uses the new API,
and attempts to keep the number of changes to a minimum.

Confirmed that lit's test suite, as well as LLVM's regression test suite,
still pass with these changes.

Patch By Brian Gesiak!

Reviewers: ddunbar, echristo, beanz, delcypher

Subscribers: llvm-commits, mehdi_amini

Differential Revision: https://reviews.llvm.org/D25173

llvm-svn: 283152
2016-10-03 22:12:42 +00:00
Chris Bieneman
dec815c8ec [lit] Remove unused imports (NFC)
Reviewers: ddunbar, echristo, beanz

Patch by Brian Gesiak!

Subscribers: llvm-commits, mehdi_amini

Differential Revision: https://reviews.llvm.org/D25169

llvm-svn: 283089
2016-10-03 04:48:18 +00:00
Daniel Dunbar
40b6500445 [lit] Add a --max-failures option.
- This is primarily useful as a "fail fast" mode for lit, where it will stop
   running tests after the first failure.

 - Patch by Max Moiseev.

llvm-svn: 282452
2016-09-26 23:38:23 +00:00
Daniel Dunbar
1934108270 [lit] Improve logging with file redirection.
- This will cause lit to automatically include the first 1K of data in
   redirected output files when a command fails (previously if the command
   failed, but the main point of the test was, say, a `FileCheck` later on, then
   the log wasn't helpful in showing why the command failed).

llvm-svn: 272021
2016-06-07 16:13:40 +00:00
Daniel Dunbar
591838d1d3 [lit] Improve readability of failing scripts.
- This only applies to scripts executed by the _internal_ shell script
   interpreter.

 - This patch reworks the log to look more like a shell transcript, and be less
   verbose (but in the interest of calling attention to the important parts).

Here is an example of the new format, for commands with/without failures and
with/without output:
```
$ true
$ echo hi
hi

$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1

```

llvm-svn: 271610
2016-06-02 23:49:42 +00:00
Eric Fiselier
cd24b0d0d8 Add "REQUIRES-ANY" feature test
Summary:
This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available.

Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03`  when testing of behavior that is specific to older dialects but has since changed.


Reviewers: rnk, ddunbar

Subscribers: ddunbar, probinson, llvm-commits, cfe-commits

Differential Revision: http://reviews.llvm.org/D20757

llvm-svn: 271468
2016-06-02 01:59:57 +00:00
Daniel Dunbar
42881eac30 [lit][googletest] Handle upstream gtest output
Summary:
Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior
to running tests. LLVM removed that print statement in r61540. If a user were
to use lit to run tests that use upstream googletest, however, lit
reports "Running main()" as an invalid test name.

To avoid such a failure, add an extra conditional to `formats/googletest.py`.
Also add tests to demonstrate the modified behavior.

Reviewers: abdulras, ddunbar

Subscribers: ddunbar, llvm-commits, kastiglione

Differential Revision: http://reviews.llvm.org/D18606

llvm-svn: 265034
2016-03-31 18:22:55 +00:00
Dan Liew
c754ace22f Revert "Teach the CMake build system to run lit's test suite. These can be run"
This reverts r257221.

This caused several build bot failures

* It looks like some of the tests don't work correctly under Windows
* It looks like the lit per test timeout tests fail

So I'm reverting for now. Once the above failures are fixed running
lit's tests can be enabled again.

llvm-svn: 257268
2016-01-09 21:42:57 +00:00
Dan Liew
9d49638f44 Teach the CMake build system to run lit's test suite. These can be run
directy with ``make check-lit`` and are run as part of
``make check-all``.

In principle we should run lit's testsuite before testing LLVM using lit
so that any problems with lit get discovered before testing LLVM so we
can bail out early.  However this implementation (``check-all`` runs all
tests together) seemed simpler and will still report failing lit tests.

Note that the tests and the configured ``lit.site.cfg`` have to be
copied into the build directory to avoid polluting the source tree.

llvm-svn: 257221
2016-01-08 22:36:22 +00:00
Dan Liew
7574241053 [lit] Implement support of per test timeout in lit.
This should work with ShTest (executed externally or internally) and GTest
test formats.

To set the timeout a new option ``--timeout=`` has
been added which specifies the maximum run time of an individual test
in seconds. By default this 0 which causes no timeout to be enforced.

The timeout can also be set from a lit configuration file by modifying
the ``lit_config.maxIndividualTestTime`` property.

To implement a timeout we now require the psutil Python module if a
 timeout is requested. This dependency is confined to the newly added
 ``lit.util.killProcessAndChildren()``. A note has been added into the
 TODO document describing how we can remove the dependency on the
 ``pustil`` module in the future. It would be nice to remove this
 immediately but that is a lot more work and Daniel Dunbar believes it is
better that we get a working implementation first and then improve it.

To avoid breaking the existing behaviour the psutil module will not be
imported if no timeout is requested.

The included testcases are derived from test cases provided by
 Jonathan Roelofs which were in an previous attempt to add a per test
 timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan!

Reviewers: ddunbar, jroelofs, cmatthews, MatzeB

Subscribers: cmatthews, llvm-commits

Differential Revision: http://reviews.llvm.org/D14706

llvm-svn: 256471
2015-12-27 14:03:49 +00:00
Eric Fiselier
61459cef40 [LIT] Fix discovery.py test once again
llvm-svn: 242758
2015-07-21 04:25:13 +00:00
Eric Fiselier
8fcf50515b [LIT] Fix failing LIT tests
Summary:
I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine.

I made the following changes for the following reasons.

1. google-test.py: The Google test format now checks for "[  PASSED  ] 1 test." to check if a test passes.
2. discovery.py: The output appears in a different order on my machine than it did in the test.
3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test.
4. The classname is now formed differently in `getJUnitXML(...)`.

I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts?

Reviewers: ddunbar, danalbert, jroelofs

Reviewed By: jroelofs

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9864

llvm-svn: 239663
2015-06-13 06:55:44 +00:00
Rafael Espindola
21a400857f Require python 2.7.
We were already requiring 2.5, which meant that people on old linux distros
had to upgrade anyway.

Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X
exception syntax.

According to the discussion on llvmdev, there is not much value is requiring
just 2.6, we may as well just require 2.7.

llvm-svn: 224129
2014-12-12 15:29:31 +00:00
Chris Matthews
be5f168981 Add a test-case for lit xunit output
llvm-svn: 223307
2014-12-03 23:01:35 +00:00
Daniel Dunbar
cd625f4e54 [lit] Add an --output option, for writing results in a machine readable form.
llvm-svn: 190738
2013-09-14 01:19:17 +00:00
Daniel Dunbar
9aeba4963a [lit] Add support for attach arbitrary metrics to test results.
- This is a work-in-progress and all details are subject to change, but I am
   trying to build up support for allowing lit to be used as a driver for
   performance tests (or other tests which might want to record information
   beyond simple PASS/FAIL).

llvm-svn: 190535
2013-09-11 17:45:11 +00:00
Daniel Dunbar
dac381a323 [lit] Allow config files to pass arbitrary values to child configs.
- This aligns with how existing test suites end up wanting to use the local
   config files, conceptually it makes sense to consider them to be inherited.

llvm-svn: 189885
2013-09-03 23:32:55 +00:00
Daniel Dunbar
9d8271b351 [lit] [tests] Add missing test input file.
llvm-svn: 189561
2013-08-29 03:02:34 +00:00
Daniel Dunbar
b8da61fd1b [lit] Fix internal shell's argv[0] handling.
- At least on OS X, it is important for correct behavior of /bin/[ that argv[0]
   is passed as written, and not as the full executable path.

llvm-svn: 189559
2013-08-29 02:52:10 +00:00
Daniel Dunbar
898d0d34a9 [lit] Lift XFAIL handling to core infrastructure.
llvm-svn: 188949
2013-08-21 22:26:42 +00:00
Daniel Dunbar
301d387929 [lit] Add test coverage of gtest format.
llvm-svn: 188417
2013-08-14 22:21:11 +00:00
Daniel Dunbar
54d613b97f [lit] Ensure test output is converted to strings where possible.
- This cleans up the text output of failing tests when run under PY3.

llvm-svn: 188416
2013-08-14 22:21:01 +00:00
Daniel Dunbar
e469b981f3 [lit] Support parsing scripts with inconsistent or invalid encodings.
- For whatever reason, we have a lot of test files with bogus unicode
   characters. This patch allows those scripts to still be parsed on Python3 by
   changing the parsing logic to work on binary files, and only require the
   actual script commands to be convertible to ascii.

 - This patch has been tweaked to now ensure that the command strings are not of
   unicode type on Python 2.6-7.

llvm-svn: 188398
2013-08-14 18:22:41 +00:00
Daniel Dunbar
16b934d624 Revert r188376, "[lit] Support parsing scripts with inconsistent or invalid encodings.", this doesn't work yet for bots using the internal shell.
llvm-svn: 188379
2013-08-14 15:55:25 +00:00
Daniel Dunbar
cf0702601a [lit] Support parsing scripts with inconsistent or invalid encodings.
- For whatever reason, we have a lot of test files with bogus unicode
   characters. This patch allows those scripts to still be parsed on Python3 by
   changing the parsing logic to work on binary files, and only require the
   actual script commands to be convertible to ascii.

llvm-svn: 188376
2013-08-14 15:24:58 +00:00
Daniel Dunbar
6d0ed4cd16 [lit] Fix tests to execute lit with same python as invoked with.
llvm-svn: 188358
2013-08-14 05:07:09 +00:00
Daniel Dunbar
72868c74f3 [lit] Update lit's own tests to use lit_config and lit package, as appropriate.
llvm-svn: 188107
2013-08-09 21:39:17 +00:00
Daniel Dunbar
c4477dfd92 [lit] Change --show-{tests,suites} to exit after printing.
- This is a more sensible behavior than printing and also running tests.

llvm-svn: 188009
2013-08-08 20:59:25 +00:00
Daniel Dunbar
668c61f687 [lit] tests: Change discovery test to use --show-tests.
llvm-svn: 187851
2013-08-07 03:12:51 +00:00
Daniel Dunbar
175aed579d lit: Fix infinite recursion when an out-of-tree test root is located inside the source test root.
llvm-svn: 179402
2013-04-12 19:09:09 +00:00
Daniel Dunbar
b5e9727ee5 lit: Add a test for discovery w/ test_exec_root (out-of-tree test root).
llvm-svn: 179401
2013-04-12 19:08:57 +00:00
Daniel Dunbar
970faff8c1 lit: Add a test for discovery when exact test names are given.
llvm-svn: 179247
2013-04-11 00:31:22 +00:00
Daniel Dunbar
99a67ed76e lit: Add a trivial test of the basic progress bar.
llvm-svn: 179243
2013-04-11 00:05:37 +00:00
Daniel Dunbar
38d3798b94 [lit] Add a test for internal shell execution behaviors.
llvm-svn: 174102
2013-01-31 22:15:20 +00:00
Daniel Dunbar
14b8265f5f [lit] Add a test for the various ShTest format features.
llvm-svn: 174072
2013-01-31 18:05:01 +00:00
Daniel Dunbar
ae1f040d76 [lit] Move unittest adaptor code into discovery module.
- Also, add a test for it.

llvm-svn: 174019
2013-01-31 01:23:39 +00:00
Daniel Dunbar
25baa132f3 [lit] Run the ShUtil tests as part of the test suite.
llvm-svn: 174018
2013-01-31 01:23:26 +00:00
Daniel Dunbar
33d340e23b [lit] Set parallel mode for coverage tests.
llvm-svn: 174017
2013-01-31 01:23:18 +00:00