Mercurial > hg > CbC > CbC_llvm

comparison docs/TestSuiteMakefileGuide.rst @ 148:63bd29f05246

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
merged
author Shinji KONO <kono@ie.u-ryukyu.ac.jp>
date Wed, 14 Aug 2019 19:46:37 +0900
parents c2174574ed3a
children
comparison
equal deleted inserted replaced
146:3fc4d5c3e21e 148:63bd29f05246
1 ===================== 1 ======================================
2 LLVM test-suite Guide 2 test-suite Makefile Guide (deprecated)
3 ===================== 3 ======================================
4 4
5 .. contents:: 5 .. contents::
6 :local: 6 :local:
7 7
8 Overview 8 Overview
9 ======== 9 ========
10
11 This document describes the features of the Makefile-based LLVM
12 test-suite as well as the cmake based replacement. This way of interacting
13 with the test-suite is deprecated in favor of running the test-suite using LNT,
14 but may continue to prove useful for some users. See the Testing
15 Guide's :ref:`test-suite Quickstart <test-suite-quickstart>` section for more
16 information.
17
18 Test suite Structure
19 ====================
20
21 The ``test-suite`` module contains a number of programs that can be
22 compiled with LLVM and executed. These programs are compiled using the
23 native compiler and various LLVM backends. The output from the program
24 compiled with the native compiler is assumed correct; the results from
25 the other programs are compared to the native program output and pass if
26 they match.
27
28 When executing tests, it is usually a good idea to start out with a
29 subset of the available tests or programs. This makes test run times
30 smaller at first and later on this is useful to investigate individual
31 test failures. To run some test only on a subset of programs, simply
32 change directory to the programs you want tested and run ``gmake``
33 there. Alternatively, you can run a different test using the ``TEST``
34 variable to change what tests or run on the selected programs (see below
35 for more info).
36
37 In addition for testing correctness, the ``test-suite`` directory also
38 performs timing tests of various LLVM optimizations. It also records
39 compilation times for the compilers and the JIT. This information can be
40 used to compare the effectiveness of LLVM's optimizations and code
41 generation.
42
43 ``test-suite`` tests are divided into three types of tests: MultiSource,
44 SingleSource, and External.
45
46 - ``test-suite/SingleSource``
47
48 The SingleSource directory contains test programs that are only a
49 single source file in size. These are usually small benchmark
50 programs or small programs that calculate a particular value. Several
51 such programs are grouped together in each directory.
52
53 - ``test-suite/MultiSource``
54
55 The MultiSource directory contains subdirectories which contain
56 entire programs with multiple source files. Large benchmarks and
57 whole applications go here.
58
59 - ``test-suite/External``
60
61 The External directory contains Makefiles for building code that is
62 external to (i.e., not distributed with) LLVM. The most prominent
63 members of this directory are the SPEC 95 and SPEC 2000 benchmark
64 suites. The ``External`` directory does not contain these actual
65 tests, but only the Makefiles that know how to properly compile these
66 programs from somewhere else. The presence and location of these
67 external programs is configured by the test-suite ``configure``
68 script.
69
70 Each tree is then subdivided into several categories, including
71 applications, benchmarks, regression tests, code that is strange
72 grammatically, etc. These organizations should be relatively self
73 explanatory.
74
75 Some tests are known to fail. Some are bugs that we have not fixed yet;
76 others are features that we haven't added yet (or may never add). In the
77 regression tests, the result for such tests will be XFAIL (eXpected
78 FAILure). In this way, you can tell the difference between an expected
79 and unexpected failure.
80
81 The tests in the test suite have no such feature at this time. If the
82 test passes, only warnings and other miscellaneous output will be
83 generated. If a test fails, a large <program> FAILED message will be
84 displayed. This will help you separate benign warnings from actual test
85 failures.
86
87 Running the test suite via CMake
88 ================================
89
90 To run the test suite, you need to use the following steps:
91
92 #. The test suite uses the lit test runner to run the test-suite,
93 you need to have lit installed first. Check out LLVM and install lit:
94
95 .. code-block:: bash
96
97 % svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
98 % cd llvm/utils/lit
99 % sudo python setup.py install # Or without sudo, install in virtual-env.
100 running install
101 running bdist_egg
102 running egg_info
103 writing lit.egg-info/PKG-INFO
104 ...
105 % lit --version
106 lit 0.5.0dev
107
108 #. Check out the ``test-suite`` module with:
109
110 .. code-block:: bash
111
112 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
113
114 #. Use CMake to configure the test suite in a new directory. You cannot build
115 the test suite in the source tree.
116
117 .. code-block:: bash
118
119 % mkdir test-suite-build
120 % cd test-suite-build
121 % cmake ../test-suite
122
123 #. Build the benchmarks, using the makefiles CMake generated.
124
125 .. code-block:: bash
126
127 % make
128 Scanning dependencies of target timeit-target
129 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
130 [ 0%] Linking C executable timeit-target
131 [ 0%] Built target timeit-target
132 Scanning dependencies of target fpcmp-host
133 [ 0%] [TEST_SUITE_HOST_CC] Building host executable fpcmp
134 [ 0%] Built target fpcmp-host
135 Scanning dependencies of target timeit-host
136 [ 0%] [TEST_SUITE_HOST_CC] Building host executable timeit
137 [ 0%] Built target timeit-host
138
139
140 #. Run the tests with lit:
141
142 .. code-block:: bash
143
144 % lit -v -j 1 . -o results.json
145 -- Testing: 474 tests, 1 threads --
146 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
147 ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
148 compile_time: 0.2192
149 exec_time: 0.0462
150 hash: "59620e187c6ac38b36382685ccd2b63b"
151 size: 83348
152 **********
153 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
154
155
156 Running the test suite via Makefiles (deprecated)
157 =================================================
158 10
159 First, all tests are executed within the LLVM object directory tree. 11 First, all tests are executed within the LLVM object directory tree.
160 They *are not* executed inside of the LLVM source tree. This is because 12 They *are not* executed inside of the LLVM source tree. This is because
161 the test suite creates temporary files during execution. 13 the test suite creates temporary files during execution.
162 14
163 To run the test suite, you need to use the following steps: 15 To run the test suite, you need to use the following steps:
164 16
165 #. ``cd`` into the ``llvm/projects`` directory in your source tree.
166 #. Check out the ``test-suite`` module with: 17 #. Check out the ``test-suite`` module with:
167 18
168 .. code-block:: bash 19 .. code-block:: bash
169 20
170 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite 21 % git clone https://github.com/llvm/llvm-test-suite.git test-suite
171 22
172 This will get the test suite into ``llvm/projects/test-suite``. 23 #. FIXME: these directions are outdated and won't work. Figure out
24 what the correct thing to do is, and write it down here.
173 25
174 #. Configure and build ``llvm``. 26 #. Configure and build ``llvm``.
175 27
176 #. Configure and build ``llvm-gcc``. 28 #. Configure and build ``llvm-gcc``.
177 29
206 Note that the second and third steps only need to be done once. After 58 Note that the second and third steps only need to be done once. After
207 you have the suite checked out and configured, you don't need to do it 59 you have the suite checked out and configured, you don't need to do it
208 again (unless the test code or configure script changes). 60 again (unless the test code or configure script changes).
209 61
210 Configuring External Tests 62 Configuring External Tests
211 -------------------------- 63 ==========================
212 64
213 In order to run the External tests in the ``test-suite`` module, you 65 In order to run the External tests in the ``test-suite`` module, you
214 must specify *--with-externals*. This must be done during the 66 must specify *--with-externals*. This must be done during the
215 *re-configuration* step (see above), and the ``llvm`` re-configuration 67 *re-configuration* step (see above), and the ``llvm`` re-configuration
216 must recognize the previously-built ``llvm-gcc``. If any of these is 68 must recognize the previously-built ``llvm-gcc``. If any of these is
235 * povray31 87 * povray31
236 88
237 Others are added from time to time, and can be determined from 89 Others are added from time to time, and can be determined from
238 ``configure``. 90 ``configure``.
239 91
240 Running different tests 92 Running Different Tests
241 ----------------------- 93 =======================
242 94
243 In addition to the regular "whole program" tests, the ``test-suite`` 95 In addition to the regular "whole program" tests, the ``test-suite``
244 module also provides a mechanism for compiling the programs in different 96 module also provides a mechanism for compiling the programs in different
245 ways. If the variable TEST is defined on the ``gmake`` command line, the 97 ways. If the variable TEST is defined on the ``gmake`` command line, the
246 test system will include a Makefile named 98 test system will include a Makefile named
255 designed for internal LLVM research and will not work outside of the 107 designed for internal LLVM research and will not work outside of the
256 LLVM research group. They may still be valuable, however, as a guide to 108 LLVM research group. They may still be valuable, however, as a guide to
257 writing your own TEST Makefile for any optimization or analysis passes 109 writing your own TEST Makefile for any optimization or analysis passes
258 that you develop with LLVM. 110 that you develop with LLVM.
259 111
260 Generating test output 112 Generating Test Output
261 ---------------------- 113 ======================
262 114
263 There are a number of ways to run the tests and generate output. The 115 There are a number of ways to run the tests and generate output. The
264 most simple one is simply running ``gmake`` with no arguments. This will 116 most simple one is simply running ``gmake`` with no arguments. This will
265 compile and run all programs in the tree using a number of different 117 compile and run all programs in the tree using a number of different
266 methods and compare results. Any failures are reported in the output, 118 methods and compare results. Any failures are reported in the output,
281 the results are always stored in the ``report.<type>.format`` file (when 133 the results are always stored in the ``report.<type>.format`` file (when
282 running with ``TEST=<type>``). The ``report`` also generate a file 134 running with ``TEST=<type>``). The ``report`` also generate a file
283 called ``report.<type>.raw.out`` containing the output of the entire 135 called ``report.<type>.raw.out`` containing the output of the entire
284 test run. 136 test run.
285 137
286 Writing custom tests for the test suite 138 Writing Custom Tests for the test-suite
287 --------------------------------------- 139 =======================================
288 140
289 Assuming you can run the test suite, (e.g. 141 Assuming you can run the test suite, (e.g.
290 "``gmake TEST=nightly report``" should work), it is really easy to run 142 "``gmake TEST=nightly report``" should work), it is really easy to run
291 optimizations or code generator components against every program in the 143 optimizations or code generator components against every program in the
292 tree, collecting statistics or running custom checks for correctness. At 144 tree, collecting statistics or running custom checks for correctness. At