0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
1 =================================
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
2 LLVM Testing Infrastructure Guide
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
3 =================================
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
4
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
5 .. contents::
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
6 :local:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
7
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
8 .. toctree::
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
9 :hidden:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
10
|
|
147
|
11 TestSuiteGuide
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
12 TestSuiteMakefileGuide
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
13
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
14 Overview
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
15 ========
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
16
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
17 This document is the reference manual for the LLVM testing
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
18 infrastructure. It documents the structure of the LLVM testing
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
19 infrastructure, the tools needed to use it, and how to add and run
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
20 tests.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
21
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
22 Requirements
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
23 ============
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
24
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
25 In order to use the LLVM testing infrastructure, you will need all of the
|
|
83
|
26 software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
27 later.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
28
|
|
147
|
29 LLVM Testing Infrastructure Organization
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
30 ========================================
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
31
|
|
147
|
32 The LLVM testing infrastructure contains three major categories of tests:
|
|
|
33 unit tests, regression tests and whole programs. The unit tests and regression
|
|
|
34 tests are contained inside the LLVM repository itself under ``llvm/unittests``
|
|
|
35 and ``llvm/test`` respectively and are expected to always pass -- they should be
|
|
|
36 run before every commit.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
37
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
38 The whole programs tests are referred to as the "LLVM test suite" (or
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
39 "test-suite") and are in the ``test-suite`` module in subversion. For
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
40 historical reasons, these tests are also referred to as the "nightly
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
41 tests" in places, which is less ambiguous than "test-suite" and remains
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
42 in use although we run them much more often than nightly.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
43
|
|
147
|
44 Unit tests
|
|
|
45 ----------
|
|
|
46
|
|
|
47 Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/googletest/docs/primer.md>`_
|
|
|
48 and `Google Mock <https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md>`_
|
|
|
49 and are located in the ``llvm/unittests`` directory.
|
|
|
50
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
51 Regression tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
52 ----------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
53
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
54 The regression tests are small pieces of code that test a specific
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
55 feature of LLVM or trigger a specific bug in LLVM. The language they are
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
56 written in depends on the part of LLVM being tested. These tests are driven by
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
57 the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
58 are located in the ``llvm/test`` directory.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
59
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
60 Typically when a bug is found in LLVM, a regression test containing just
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
61 enough code to reproduce the problem should be written and placed
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
62 somewhere underneath this directory. For example, it can be a small
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
63 piece of LLVM IR distilled from an actual application or benchmark.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
64
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
65 ``test-suite``
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
66 --------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
67
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
68 The test suite contains whole programs, which are pieces of code which
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
69 can be compiled and linked into a stand-alone program that can be
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
70 executed. These programs are generally written in high level languages
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
71 such as C or C++.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
72
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
73 These programs are compiled using a user specified compiler and set of
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
74 flags, and then executed to capture the program output and timing
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
75 information. The output of these programs is compared to a reference
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
76 output to ensure that the program is being compiled correctly.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
77
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
78 In addition to compiling and executing programs, whole program tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
79 serve as a way of benchmarking LLVM performance, both in terms of the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
80 efficiency of the programs generated as well as the speed with which
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
81 LLVM compiles, optimizes, and generates code.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
82
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
83 The test-suite is located in the ``test-suite`` Subversion module.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
84
|
|
147
|
85 See the :doc:`TestSuiteGuide` for details.
|
|
|
86
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
87 Debugging Information tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
88 ---------------------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
89
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
90 The test suite contains tests to check quality of debugging information.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
91 The test are written in C based languages or in LLVM assembly language.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
92
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
93 These tests are compiled and run under a debugger. The debugger output
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
94 is checked to validate of debugging information. See README.txt in the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
95 test suite for more information . This test suite is located in the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
96 ``debuginfo-tests`` Subversion module.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
97
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
98 Quick start
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
99 ===========
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
100
|
|
147
|
101 The tests are located in two separate Subversion modules. The unit and
|
|
|
102 regression tests are in the main "llvm" module under the directories
|
|
|
103 ``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
|
|
|
104 main LLVM tree). Use ``make check-all`` to run the unit and regression tests
|
|
|
105 after building LLVM.
|
|
|
106
|
|
|
107 The ``test-suite`` module contains more comprehensive tests including whole C
|
|
|
108 and C++ programs. See the :doc:`TestSuiteGuide` for details.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
109
|
|
147
|
110 Unit and Regression tests
|
|
|
111 -------------------------
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
112
|
|
147
|
113 To run all of the LLVM unit tests use the check-llvm-unit target:
|
|
|
114
|
|
|
115 .. code-block:: bash
|
|
|
116
|
|
|
117 % make check-llvm-unit
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
118
|
|
120
|
119 To run all of the LLVM regression tests use the check-llvm target:
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
120
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
121 .. code-block:: bash
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
122
|
|
120
|
123 % make check-llvm
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
124
|
|
147
|
125 In order to get reasonable testing performance, build LLVM and subprojects
|
|
|
126 in release mode, i.e.
|
|
|
127
|
|
|
128 .. code-block:: bash
|
|
|
129
|
|
|
130 % cmake -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On
|
|
|
131
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
132 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
133 can run the LLVM and Clang tests simultaneously using:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
134
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
135 .. code-block:: bash
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
136
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
137 % make check-all
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
138
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
139 To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
140 variable to pass the required options to lit. For example, you can use:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
141
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
142 .. code-block:: bash
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
143
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
144 % make check LIT_ARGS="-v --vg --vg-leak"
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
145
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
146 to enable testing with valgrind and with leak checking enabled.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
147
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
148 To run individual tests or subsets of tests, you can use the ``llvm-lit``
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
149 script which is built as part of LLVM. For example, to run the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
150 ``Integer/BitPacked.ll`` test by itself you can run:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
151
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
152 .. code-block:: bash
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
153
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
154 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
155
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
156 or to run all of the ARM CodeGen tests:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
157
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
158 .. code-block:: bash
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
159
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
160 % llvm-lit ~/llvm/test/CodeGen/ARM
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
161
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
162 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
163 or the :doc:`lit man page <CommandGuide/lit>`.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
164
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
165 Debugging Information tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
166 ---------------------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
167
|
|
147
|
168 To run debugging information tests simply add the ``debuginfo-tests``
|
|
|
169 project to your ``LLVM_ENABLE_PROJECTS`` define on the cmake
|
|
|
170 command-line.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
171
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
172 Regression test structure
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
173 =========================
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
174
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
175 The LLVM regression tests are driven by :program:`lit` and are located in the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
176 ``llvm/test`` directory.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
177
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
178 This directory contains a large array of small tests that exercise
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
179 various features of LLVM and to ensure that regressions do not occur.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
180 The directory is broken into several sub-directories, each focused on a
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
181 particular area of LLVM.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
182
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
183 Writing new regression tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
184 ----------------------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
185
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
186 The regression test structure is very simple, but does require some
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
187 information to be set. This information is gathered via ``configure``
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
188 and is written to a file, ``test/lit.site.cfg`` in the build directory.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
189 The ``llvm/test`` Makefile does this work for you.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
190
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
191 In order for the regression tests to work, each directory of tests must
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
192 have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
193 how to run the tests. This file is just Python code and thus is very
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
194 flexible, but we've standardized it for the LLVM regression tests. If
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
195 you're adding a directory of tests, just copy ``lit.local.cfg`` from
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
196 another directory to get running. The standard ``lit.local.cfg`` simply
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
197 specifies which files to look in for tests. Any directory that contains
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
198 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
199 documentation <CommandGuide/lit>` for more information.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
200
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
201 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
202 how to run it. If there are no RUN lines, :program:`lit` will issue an error
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
203 while running a test.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
204
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
205 RUN lines are specified in the comments of the test program using the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
206 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
207 to execute. Together, these lines form the "script" that :program:`lit`
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
208 executes to run the test case. The syntax of the RUN lines is similar to a
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
209 shell's syntax for pipelines including I/O redirection and variable
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
210 substitution. However, even though these lines may *look* like a shell
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
211 script, they are not. RUN lines are interpreted by :program:`lit`.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
212 Consequently, the syntax differs from shell in a few ways. You can specify
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
213 as many RUN lines as needed.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
214
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
215 :program:`lit` performs substitution on each RUN line to replace LLVM tool names
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
216 with the full paths to the executable built for each tool (in
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
217 ``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
218 not invoke any stray LLVM tools in the user's path during testing.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
219
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
220 Each RUN line is executed on its own, distinct from other lines unless
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
221 its last character is ``\``. This continuation character causes the RUN
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
222 line to be concatenated with the next one. In this way you can build up
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
223 long pipelines of commands without making huge line lengths. The lines
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
224 ending in ``\`` are concatenated until a RUN line that doesn't end in
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
225 ``\`` is found. This concatenated set of RUN lines then constitutes one
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
226 execution. :program:`lit` will substitute variables and arrange for the pipeline
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
227 to be executed. If any process in the pipeline fails, the entire line (and
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
228 test case) fails too.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
229
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
230 Below is an example of legal RUN lines in a ``.ll`` file:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
231
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
232 .. code-block:: llvm
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
233
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
234 ; RUN: llvm-as < %s | llvm-dis > %t1
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
235 ; RUN: llvm-dis < %s.bc-13 > %t2
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
236 ; RUN: diff %t1 %t2
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
237
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
238 As with a Unix shell, the RUN lines permit pipelines and I/O
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
239 redirection to be used.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
240
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
241 There are some quoting rules that you must pay attention to when writing
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
242 your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
243 strip off any quote characters so they will get passed to the invoked program.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
244 To avoid this use curly braces to tell :program:`lit` that it should treat
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
245 everything enclosed as one value.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
246
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
247 In general, you should strive to keep your RUN lines as simple as possible,
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
248 using them only to run tools that generate textual output you can then examine.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
249 The recommended way to examine output to figure out if the test passes is using
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
250 the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
251 lines is deprecated - please do not send or commit patches that use it.]*
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
252
|
|
100
|
253 Put related tests into a single file rather than having a separate file per
|
|
|
254 test. Check if there are files already covering your feature and consider
|
|
|
255 adding your code there instead of creating a new file.
|
|
|
256
|
|
83
|
257 Extra files
|
|
|
258 -----------
|
|
|
259
|
|
|
260 If your test requires extra files besides the file containing the ``RUN:``
|
|
|
261 lines, the idiomatic place to put them is in a subdirectory ``Inputs``.
|
|
|
262 You can then refer to the extra files as ``%S/Inputs/foo.bar``.
|
|
|
263
|
|
|
264 For example, consider ``test/Linker/ident.ll``. The directory structure is
|
|
|
265 as follows::
|
|
|
266
|
|
|
267 test/
|
|
|
268 Linker/
|
|
|
269 ident.ll
|
|
|
270 Inputs/
|
|
|
271 ident.a.ll
|
|
|
272 ident.b.ll
|
|
|
273
|
|
|
274 For convenience, these are the contents:
|
|
|
275
|
|
|
276 .. code-block:: llvm
|
|
|
277
|
|
|
278 ;;;;; ident.ll:
|
|
|
279
|
|
|
280 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
|
|
|
281
|
|
|
282 ; Verify that multiple input llvm.ident metadata are linked together.
|
|
|
283
|
|
|
284 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
|
|
|
285 ; CHECK-DAG: "Compiler V1"
|
|
|
286 ; CHECK-DAG: "Compiler V2"
|
|
|
287 ; CHECK-DAG: "Compiler V3"
|
|
|
288
|
|
|
289 ;;;;; Inputs/ident.a.ll:
|
|
|
290
|
|
|
291 !llvm.ident = !{!0, !1}
|
|
|
292 !0 = metadata !{metadata !"Compiler V1"}
|
|
|
293 !1 = metadata !{metadata !"Compiler V2"}
|
|
|
294
|
|
|
295 ;;;;; Inputs/ident.b.ll:
|
|
|
296
|
|
|
297 !llvm.ident = !{!0}
|
|
|
298 !0 = metadata !{metadata !"Compiler V3"}
|
|
|
299
|
|
|
300 For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't
|
|
|
301 actually participate in the test besides holding the ``RUN:`` lines.
|
|
|
302
|
|
|
303 .. note::
|
|
|
304
|
|
|
305 Some existing tests use ``RUN: true`` in extra files instead of just
|
|
|
306 putting the extra files in an ``Inputs/`` directory. This pattern is
|
|
|
307 deprecated.
|
|
|
308
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
309 Fragile tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
310 -------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
311
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
312 It is easy to write a fragile test that would fail spuriously if the tool being
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
313 tested outputs a full path to the input file. For example, :program:`opt` by
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
314 default outputs a ``ModuleID``:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
315
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
316 .. code-block:: console
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
317
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
318 $ cat example.ll
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
319 define i32 @main() nounwind {
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
320 ret i32 0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
321 }
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
322
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
323 $ opt -S /path/to/example.ll
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
324 ; ModuleID = '/path/to/example.ll'
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
325
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
326 define i32 @main() nounwind {
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
327 ret i32 0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
328 }
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
329
|
|
121
|
330 ``ModuleID`` can unexpectedly match against ``CHECK`` lines. For example:
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
331
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
332 .. code-block:: llvm
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
333
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
334 ; RUN: opt -S %s | FileCheck
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
335
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
336 define i32 @main() nounwind {
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
337 ; CHECK-NOT: load
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
338 ret i32 0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
339 }
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
340
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
341 This test will fail if placed into a ``download`` directory.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
342
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
343 To make your tests robust, always use ``opt ... < %s`` in the RUN line.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
344 :program:`opt` does not output a ``ModuleID`` when input comes from stdin.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
345
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
346 Platform-Specific Tests
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
347 -----------------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
348
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
349 Whenever adding tests that require the knowledge of a specific platform,
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
350 either related to code generated, specific output or back-end features,
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
351 you must make sure to isolate the features, so that buildbots that
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
352 run on different architectures (and don't even compile all back-ends),
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
353 don't fail.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
354
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
355 The first problem is to check for target-specific output, for example sizes
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
356 of structures, paths and architecture names, for example:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
357
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
358 * Tests containing Windows paths will fail on Linux and vice-versa.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
359 * Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
360 * Tests where the debug information calculates the size of types and structures.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
361
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
362 Also, if the test rely on any behaviour that is coded in any back-end, it must
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
363 go in its own directory. So, for instance, code generator tests for ARM go
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
364 into ``test/CodeGen/ARM`` and so on. Those directories contain a special
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
365 ``lit`` configuration file that ensure all tests in that directory will
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
366 only run if a specific back-end is compiled and available.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
367
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
368 For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
369
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
370 .. code-block:: python
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
371
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
372 config.suffixes = ['.ll', '.c', '.cpp', '.test']
|
|
77
|
373 if not 'ARM' in config.root.targets:
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
374 config.unsupported = True
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
375
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
376 Other platform-specific tests are those that depend on a specific feature
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
377 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
378
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
379 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
380 variants:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
381
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
382 .. code-block:: llvm
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
383
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
384 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
385 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
386 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
387
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
388 And the checks are different:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
389
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
390 .. code-block:: llvm
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
391
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
392 ; SSE2: @test1
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
393 ; SSE2: psubusw LCPI0_0(%rip), %xmm0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
394 ; AVX1: @test1
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
395 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
396 ; AVX2: @test1
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
397 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
398
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
399 So, if you're testing for a behaviour that you know is platform-specific or
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
400 depends on special features of sub-architectures, you must add the specific
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
401 triple, test with the specific FileCheck and put it into the specific
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
402 directory that will filter out all other architectures.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
403
|
|
121
|
404
|
|
|
405 Constraining test execution
|
|
|
406 ---------------------------
|
|
120
|
407
|
|
121
|
408 Some tests can be run only in specific configurations, such as
|
|
|
409 with debug builds or on particular platforms. Use ``REQUIRES``
|
|
|
410 and ``UNSUPPORTED`` to control when the test is enabled.
|
|
|
411
|
|
|
412 Some tests are expected to fail. For example, there may be a known bug
|
|
|
413 that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
|
|
|
414 An ``XFAIL`` test will be successful if its execution fails, and
|
|
|
415 will be a failure if its execution succeeds.
|
|
120
|
416
|
|
|
417 .. code-block:: llvm
|
|
|
418
|
|
121
|
419 ; This test will be only enabled in the build with asserts.
|
|
120
|
420 ; REQUIRES: asserts
|
|
121
|
421 ; This test is disabled on Linux.
|
|
|
422 ; UNSUPPORTED: -linux-
|
|
|
423 ; This test is expected to fail on PowerPC.
|
|
|
424 ; XFAIL: powerpc
|
|
|
425
|
|
|
426 ``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
|
|
|
427 list of boolean expressions. The values in each expression may be:
|
|
|
428
|
|
|
429 - Features added to ``config.available_features`` by
|
|
|
430 configuration files such as ``lit.cfg``.
|
|
|
431 - Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
|
|
120
|
432
|
|
121
|
433 | ``REQUIRES`` enables the test if all expressions are true.
|
|
|
434 | ``UNSUPPORTED`` disables the test if any expression is true.
|
|
|
435 | ``XFAIL`` expects the test to fail if any expression is true.
|
|
|
436
|
|
|
437 As a special case, ``XFAIL: *`` is expected to fail everywhere.
|
|
120
|
438
|
|
121
|
439 .. code-block:: llvm
|
|
|
440
|
|
|
441 ; This test is disabled on Windows,
|
|
|
442 ; and is disabled on Linux, except for Android Linux.
|
|
|
443 ; UNSUPPORTED: windows, linux && !android
|
|
|
444 ; This test is expected to fail on both PowerPC and ARM.
|
|
|
445 ; XFAIL: powerpc || arm
|
|
|
446
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
447
|
|
77
|
448 Substitutions
|
|
|
449 -------------
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
450
|
|
77
|
451 Besides replacing LLVM tool names the following substitutions are performed in
|
|
|
452 RUN lines:
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
453
|
|
77
|
454 ``%%``
|
|
|
455 Replaced by a single ``%``. This allows escaping other substitutions.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
456
|
|
77
|
457 ``%s``
|
|
|
458 File path to the test case's source. This is suitable for passing on the
|
|
|
459 command line as the input to an LLVM tool.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
460
|
|
77
|
461 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
462
|
|
77
|
463 ``%S``
|
|
|
464 Directory path to the test case's source.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
465
|
|
77
|
466 Example: ``/home/user/llvm/test/MC/ELF``
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
467
|
|
77
|
468 ``%t``
|
|
|
469 File path to a temporary file name that could be used for this test case.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
470 The file name won't conflict with other test cases. You can append to it
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
471 if you need multiple temporaries. This is useful as the destination of
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
472 some redirected output.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
473
|
|
77
|
474 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
|
|
|
475
|
|
|
476 ``%T``
|
|
147
|
477 Directory of ``%t``. Deprecated. Shouldn't be used, because it can be easily
|
|
|
478 misused and cause race conditions between tests.
|
|
|
479
|
|
|
480 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.
|
|
77
|
481
|
|
|
482 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
|
|
|
483
|
|
|
484 ``%{pathsep}``
|
|
|
485
|
|
|
486 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows).
|
|
|
487
|
|
121
|
488 ``%/s, %/S, %/t, %/T:``
|
|
|
489
|
|
|
490 Act like the corresponding substitution above but replace any ``\``
|
|
|
491 character with a ``/``. This is useful to normalize path separators.
|
|
|
492
|
|
|
493 Example: ``%s: C:\Desktop Files/foo_test.s.tmp``
|
|
|
494
|
|
|
495 Example: ``%/s: C:/Desktop Files/foo_test.s.tmp``
|
|
|
496
|
|
|
497 ``%:s, %:S, %:t, %:T:``
|
|
|
498
|
|
|
499 Act like the corresponding substitution above but remove colons at
|
|
|
500 the beginning of Windows paths. This is useful to allow concatenation
|
|
|
501 of absolute paths on Windows to produce a legal path.
|
|
|
502
|
|
|
503 Example: ``%s: C:\Desktop Files\foo_test.s.tmp``
|
|
|
504
|
|
|
505 Example: ``%:s: C\Desktop Files\foo_test.s.tmp``
|
|
|
506
|
|
77
|
507
|
|
|
508 **LLVM-specific substitutions:**
|
|
|
509
|
|
|
510 ``%shlibext``
|
|
|
511 The suffix for the host platforms shared library files. This includes the
|
|
|
512 period as the first character.
|
|
|
513
|
|
147
|
514 Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)
|
|
77
|
515
|
|
|
516 ``%exeext``
|
|
|
517 The suffix for the host platforms executable files. This includes the
|
|
|
518 period as the first character.
|
|
|
519
|
|
|
520 Example: ``.exe`` (Windows), empty on Linux.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
521
|
|
77
|
522 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
|
|
|
523 The number of the line where this substitution is used, with an optional
|
|
|
524 integer offset. This can be used in tests with multiple RUN lines, which
|
|
|
525 reference test file's line numbers.
|
|
|
526
|
|
|
527
|
|
|
528 **Clang-specific substitutions:**
|
|
|
529
|
|
|
530 ``%clang``
|
|
|
531 Invokes the Clang driver.
|
|
|
532
|
|
|
533 ``%clang_cpp``
|
|
|
534 Invokes the Clang driver for C++.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
535
|
|
77
|
536 ``%clang_cl``
|
|
|
537 Invokes the CL-compatible Clang driver.
|
|
|
538
|
|
|
539 ``%clangxx``
|
|
|
540 Invokes the G++-compatible Clang driver.
|
|
|
541
|
|
|
542 ``%clang_cc1``
|
|
|
543 Invokes the Clang frontend.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
544
|
|
77
|
545 ``%itanium_abi_triple``, ``%ms_abi_triple``
|
|
|
546 These substitutions can be used to get the current target triple adjusted to
|
|
|
547 the desired ABI. For example, if the test suite is running with the
|
|
|
548 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
|
|
|
549 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
|
|
|
550 constraining it to a specific triple.
|
|
|
551
|
|
|
552 To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``.
|
|
|
553
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
554
|
|
95
|
555 Options
|
|
|
556 -------
|
|
|
557
|
|
|
558 The llvm lit configuration allows to customize some things with user options:
|
|
|
559
|
|
|
560 ``llc``, ``opt``, ...
|
|
|
561 Substitute the respective llvm tool name with a custom command line. This
|
|
|
562 allows to specify custom paths and default arguments for these tools.
|
|
|
563 Example:
|
|
|
564
|
|
|
565 % llvm-lit "-Dllc=llc -verify-machineinstrs"
|
|
|
566
|
|
|
567 ``run_long_tests``
|
|
|
568 Enable the execution of long running tests.
|
|
|
569
|
|
|
570 ``llvm_site_config``
|
|
|
571 Load the specified lit configuration instead of the default one.
|
|
|
572
|
|
|
573
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
574 Other Features
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
575 --------------
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
576
|
|
77
|
577 To make RUN line writing easier, there are several helper programs. These
|
|
|
578 helpers are in the PATH when running tests, so you can just call them using
|
|
|
579 their name. For example:
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
580
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
581 ``not``
|
|
77
|
582 This program runs its arguments and then inverts the result code from it.
|
0
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
583 Zero result codes become 1. Non-zero result codes become 0.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
584
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
585 To make the output more useful, :program:`lit` will scan
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
586 the lines of the test case for ones that contain a pattern that matches
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
587 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
588 that is related to the test case. The number after "PR" specifies the
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
589 LLVM bugzilla number. When a PR number is specified, it will be used in
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
590 the pass/fail reporting. This is useful to quickly get some context when
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
591 a test fails.
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
592
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
593 Finally, any line that contains "END." will cause the special
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
594 interpretation of lines to terminate. This is generally done right after
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
595 the last RUN: line. This has two side effects:
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
596
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
597 (a) it prevents special interpretation of lines that are part of the test
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
598 program, not the instructions to the test case, and
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
599
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
600 (b) it speeds things up for really big test cases by avoiding
|
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
601 interpretation of the remainder of the file.
|
