|
147
|
1 test-suite Guide
|
|
|
2 ================
|
|
|
3
|
|
|
4 Quickstart
|
|
|
5 ----------
|
|
|
6
|
|
|
7 1. The lit test runner is required to run the tests. You can either use one
|
|
|
8 from an LLVM build:
|
|
|
9
|
|
|
10 ```bash
|
|
|
11 % <path to llvm build>/bin/llvm-lit --version
|
|
|
12 lit 0.8.0dev
|
|
|
13 ```
|
|
|
14
|
|
|
15 An alternative is installing it as a python package in a python virtual
|
|
|
16 environment:
|
|
|
17
|
|
|
18 ```bash
|
|
|
19 % mkdir venv
|
|
|
20 % virtualenv venv
|
|
|
21 % . venv/bin/activate
|
|
|
22 % pip install svn+http://llvm.org/svn/llvm-project/llvm/trunk/utils/lit
|
|
|
23 % lit --version
|
|
|
24 lit 0.8.0dev
|
|
|
25 ```
|
|
|
26
|
|
|
27 2. Check out the `test-suite` module with:
|
|
|
28
|
|
|
29 ```bash
|
|
|
30 % git clone https://github.com/llvm/llvm-test-suite.git test-suite
|
|
|
31 ```
|
|
|
32
|
|
|
33 3. Create a build directory and use CMake to configure the suite. Use the
|
|
|
34 `CMAKE_C_COMPILER` option to specify the compiler to test. Use a cache file
|
|
|
35 to choose a typical build configuration:
|
|
|
36
|
|
|
37 ```bash
|
|
|
38 % mkdir test-suite-build
|
|
|
39 % cd test-suite-build
|
|
|
40 % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \
|
|
|
41 -C../test-suite/cmake/caches/O3.cmake \
|
|
|
42 ../test-suite
|
|
|
43 ```
|
|
|
44
|
|
|
45 4. Build the benchmarks:
|
|
|
46
|
|
|
47 ```text
|
|
|
48 % make
|
|
|
49 Scanning dependencies of target timeit-target
|
|
|
50 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
|
|
|
51 [ 0%] Linking C executable timeit-target
|
|
|
52 ...
|
|
|
53 ```
|
|
|
54
|
|
|
55 5. Run the tests with lit:
|
|
|
56
|
|
|
57 ```text
|
|
|
58 % llvm-lit -v -j 1 -o results.json .
|
|
|
59 -- Testing: 474 tests, 1 threads --
|
|
|
60 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
|
|
|
61 ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
|
|
|
62 compile_time: 0.2192
|
|
|
63 exec_time: 0.0462
|
|
|
64 hash: "59620e187c6ac38b36382685ccd2b63b"
|
|
|
65 size: 83348
|
|
|
66 **********
|
|
|
67 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
|
|
|
68 ...
|
|
|
69 ```
|
|
|
70
|
|
|
71 6. Show and compare result files (optional):
|
|
|
72
|
|
|
73 ```bash
|
|
|
74 # Make sure pandas is installed. Prepend `sudo` if necessary.
|
|
|
75 % pip install pandas
|
|
|
76 # Show a single result file:
|
|
|
77 % test-suite/utils/compare.py results.json
|
|
|
78 # Compare two result files:
|
|
|
79 % test-suite/utils/compare.py results_a.json results_b.json
|
|
|
80 ```
|
|
|
81
|
|
|
82
|
|
|
83 Structure
|
|
|
84 ---------
|
|
|
85
|
|
|
86 The test-suite contains benchmark and test programs. The programs come with
|
|
|
87 reference outputs so that their correctness can be checked. The suite comes
|
|
|
88 with tools to collect metrics such as benchmark runtime, compilation time and
|
|
|
89 code size.
|
|
|
90
|
|
|
91 The test-suite is divided into several directories:
|
|
|
92
|
|
|
93 - `SingleSource/`
|
|
|
94
|
|
|
95 Contains test programs that are only a single source file in size. A
|
|
|
96 subdirectory may contain several programs.
|
|
|
97
|
|
|
98 - `MultiSource/`
|
|
|
99
|
|
|
100 Contains subdirectories which entire programs with multiple source files.
|
|
|
101 Large benchmarks and whole applications go here.
|
|
|
102
|
|
|
103 - `MicroBenchmarks/`
|
|
|
104
|
|
|
105 Programs using the [google-benchmark](https://github.com/google/benchmark)
|
|
|
106 library. The programs define functions that are run multiple times until the
|
|
|
107 measurement results are statistically significant.
|
|
|
108
|
|
|
109 - `External/`
|
|
|
110
|
|
|
111 Contains descriptions and test data for code that cannot be directly
|
|
|
112 distributed with the test-suite. The most prominent members of this
|
|
|
113 directory are the SPEC CPU benchmark suites.
|
|
|
114 See [External Suites](#external-suites).
|
|
|
115
|
|
|
116 - `Bitcode/`
|
|
|
117
|
|
|
118 These tests are mostly written in LLVM bitcode.
|
|
|
119
|
|
|
120 - `CTMark/`
|
|
|
121
|
|
|
122 Contains symbolic links to other benchmarks forming a representative sample
|
|
|
123 for compilation performance measurements.
|
|
|
124
|
|
|
125 ### Benchmarks
|
|
|
126
|
|
|
127 Every program can work as a correctness test. Some programs are unsuitable for
|
|
|
128 performance measurements. Setting the `TEST_SUITE_BENCHMARKING_ONLY` CMake
|
|
|
129 option to `ON` will disable them.
|
|
|
130
|
|
|
131
|
|
|
132 Configuration
|
|
|
133 -------------
|
|
|
134
|
|
|
135 The test-suite has configuration options to customize building and running the
|
|
|
136 benchmarks. CMake can print a list of them:
|
|
|
137
|
|
|
138 ```bash
|
|
|
139 % cd test-suite-build
|
|
|
140 # Print basic options:
|
|
|
141 % cmake -LH
|
|
|
142 # Print all options:
|
|
|
143 % cmake -LAH
|
|
|
144 ```
|
|
|
145
|
|
|
146 ### Common Configuration Options
|
|
|
147
|
|
|
148 - `CMAKE_C_FLAGS`
|
|
|
149
|
|
|
150 Specify extra flags to be passed to C compiler invocations. The flags are
|
|
|
151 also passed to the C++ compiler and linker invocations. See
|
|
|
152 [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html)
|
|
|
153
|
|
|
154 - `CMAKE_C_COMPILER`
|
|
|
155
|
|
|
156 Select the C compiler executable to be used. Note that the C++ compiler is
|
|
|
157 inferred automatically i.e. when specifying `path/to/clang` CMake will
|
|
|
158 automatically use `path/to/clang++` as the C++ compiler. See
|
|
|
159 [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html)
|
|
|
160
|
|
|
161 - `CMAKE_BUILD_TYPE`
|
|
|
162
|
|
|
163 Select a build type like `OPTIMIZE` or `DEBUG` selecting a set of predefined
|
|
|
164 compiler flags. These flags are applied regardless of the `CMAKE_C_FLAGS`
|
|
|
165 option and may be changed by modifying `CMAKE_C_FLAGS_OPTIMIZE` etc. See
|
|
|
166 [https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html]](https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html)
|
|
|
167
|
|
|
168 - `TEST_SUITE_RUN_UNDER`
|
|
|
169
|
|
|
170 Prefix test invocations with the given tool. This is typically used to run
|
|
|
171 cross-compiled tests within a simulator tool.
|
|
|
172
|
|
|
173 - `TEST_SUITE_BENCHMARKING_ONLY`
|
|
|
174
|
|
|
175 Disable tests that are unsuitable for performance measurements. The disabled
|
|
|
176 tests either run for a very short time or are dominated by I/O performance
|
|
|
177 making them unsuitable as compiler performance tests.
|
|
|
178
|
|
|
179 - `TEST_SUITE_SUBDIRS`
|
|
|
180
|
|
|
181 Semicolon-separated list of directories to include. This can be used to only
|
|
|
182 build parts of the test-suite or to include external suites. This option
|
|
|
183 does not work reliably with deeper subdirectories as it skips intermediate
|
|
|
184 `CMakeLists.txt` files which may be required.
|
|
|
185
|
|
|
186 - `TEST_SUITE_COLLECT_STATS`
|
|
|
187
|
|
|
188 Collect internal LLVM statistics. Appends `-save-stats=obj` when invocing the
|
|
|
189 compiler and makes the lit runner collect and merge the statistic files.
|
|
|
190
|
|
|
191 - `TEST_SUITE_RUN_BENCHMARKS`
|
|
|
192
|
|
|
193 If this is set to `OFF` then lit will not actually run the tests but just
|
|
|
194 collect build statistics like compile time and code size.
|
|
|
195
|
|
|
196 - `TEST_SUITE_USE_PERF`
|
|
|
197
|
|
|
198 Use the `perf` tool for time measurement instead of the `timeit` tool that
|
|
|
199 comes with the test-suite. The `perf` is usually available on linux systems.
|
|
|
200
|
|
|
201 - `TEST_SUITE_SPEC2000_ROOT`, `TEST_SUITE_SPEC2006_ROOT`, `TEST_SUITE_SPEC2017_ROOT`, ...
|
|
|
202
|
|
|
203 Specify installation directories of external benchmark suites. You can find
|
|
|
204 more information about expected versions or usage in the README files in the
|
|
|
205 `External` directory (such as `External/SPEC/README`)
|
|
|
206
|
|
|
207 ### Common CMake Flags
|
|
|
208
|
|
|
209 - `-GNinja`
|
|
|
210
|
|
|
211 Generate build files for the ninja build tool.
|
|
|
212
|
|
|
213 - `-Ctest-suite/cmake/caches/<cachefile.cmake>`
|
|
|
214
|
|
|
215 Use a CMake cache. The test-suite comes with several CMake caches which
|
|
|
216 predefine common or tricky build configurations.
|
|
|
217
|
|
|
218
|
|
|
219 Displaying and Analyzing Results
|
|
|
220 --------------------------------
|
|
|
221
|
|
|
222 The `compare.py` script displays and compares result files. A result file is
|
|
|
223 produced when invoking lit with the `-o filename.json` flag.
|
|
|
224
|
|
|
225 Example usage:
|
|
|
226
|
|
|
227 - Basic Usage:
|
|
|
228
|
|
|
229 ```text
|
|
|
230 % test-suite/utils/compare.py baseline.json
|
|
|
231 Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics!
|
|
|
232 Tests: 508
|
|
|
233 Metric: exec_time
|
|
|
234
|
|
|
235 Program baseline
|
|
|
236
|
|
|
237 INT2006/456.hmmer/456.hmmer 1222.90
|
|
|
238 INT2006/464.h264ref/464.h264ref 928.70
|
|
|
239 ...
|
|
|
240 baseline
|
|
|
241 count 506.000000
|
|
|
242 mean 20.563098
|
|
|
243 std 111.423325
|
|
|
244 min 0.003400
|
|
|
245 25% 0.011200
|
|
|
246 50% 0.339450
|
|
|
247 75% 4.067200
|
|
|
248 max 1222.896800
|
|
|
249 ```
|
|
|
250
|
|
|
251 - Show compile_time or text segment size metrics:
|
|
|
252
|
|
|
253 ```bash
|
|
|
254 % test-suite/utils/compare.py -m compile_time baseline.json
|
|
|
255 % test-suite/utils/compare.py -m size.__text baseline.json
|
|
|
256 ```
|
|
|
257
|
|
|
258 - Compare two result files and filter short running tests:
|
|
|
259
|
|
|
260 ```bash
|
|
|
261 % test-suite/utils/compare.py --filter-short baseline.json experiment.json
|
|
|
262 ...
|
|
|
263 Program baseline experiment diff
|
|
|
264
|
|
|
265 SingleSour.../Benchmarks/Linpack/linpack-pc 5.16 4.30 -16.5%
|
|
|
266 MultiSourc...erolling-dbl/LoopRerolling-dbl 7.01 7.86 12.2%
|
|
|
267 SingleSour...UnitTests/Vectorizer/gcc-loops 3.89 3.54 -9.0%
|
|
|
268 ...
|
|
|
269 ```
|
|
|
270
|
|
|
271 - Merge multiple baseline and experiment result files by taking the minimum
|
|
|
272 runtime each:
|
|
|
273
|
|
|
274 ```bash
|
|
|
275 % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json
|
|
|
276 ```
|
|
|
277
|
|
|
278 ### Continuous Tracking with LNT
|
|
|
279
|
|
|
280 LNT is a set of client and server tools for continuously monitoring
|
|
|
281 performance. You can find more information at
|
|
|
282 [http://llvm.org/docs/lnt](http://llvm.org/docs/lnt). The official LNT instance
|
|
|
283 of the LLVM project is hosted at [http://lnt.llvm.org](http://lnt.llvm.org).
|
|
|
284
|
|
|
285
|
|
|
286 External Suites
|
|
|
287 ---------------
|
|
|
288
|
|
|
289 External suites such as SPEC can be enabled by either
|
|
|
290
|
|
|
291 - placing (or linking) them into the `test-suite/test-suite-externals/xxx` directory (example: `test-suite/test-suite-externals/speccpu2000`)
|
|
|
292 - using a configuration option such as `-D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000`
|
|
|
293
|
|
|
294 You can find further information in the respective README files such as
|
|
|
295 `test-suite/External/SPEC/README`.
|
|
|
296
|
|
|
297 For the SPEC benchmarks you can switch between the `test`, `train` and
|
|
|
298 `ref` input datasets via the `TEST_SUITE_RUN_TYPE` configuration option.
|
|
|
299 The `train` dataset is used by default.
|
|
|
300
|
|
|
301
|
|
|
302 Custom Suites
|
|
|
303 -------------
|
|
|
304
|
|
|
305 You can build custom suites using the test-suite infrastructure. A custom suite
|
|
|
306 has a `CMakeLists.txt` file at the top directory. The `CMakeLists.txt` will be
|
|
|
307 picked up automatically if placed into a subdirectory of the test-suite or when
|
|
|
308 setting the `TEST_SUITE_SUBDIRS` variable:
|
|
|
309
|
|
|
310 ```bash
|
|
|
311 % cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite
|
|
|
312 ```
|
|
|
313
|
|
|
314
|
|
|
315 Profile Guided Optimization
|
|
|
316 ---------------------------
|
|
|
317
|
|
|
318 Profile guided optimization requires to compile and run twice. First the
|
|
|
319 benchmark should be compiled with profile generation instrumentation enabled
|
|
|
320 and setup for training data. The lit runner will merge the profile files
|
|
|
321 using `llvm-profdata` so they can be used by the second compilation run.
|
|
|
322
|
|
|
323 Example:
|
|
|
324 ```bash
|
|
|
325 # Profile generation run:
|
|
|
326 % cmake -DTEST_SUITE_PROFILE_GENERATE=ON \
|
|
|
327 -DTEST_SUITE_RUN_TYPE=train \
|
|
|
328 ../test-suite
|
|
|
329 % make
|
|
|
330 % llvm-lit .
|
|
|
331 # Use the profile data for compilation and actual benchmark run:
|
|
|
332 % cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \
|
|
|
333 -DTEST_SUITE_PROFILE_USE=ON \
|
|
|
334 -DTEST_SUITE_RUN_TYPE=ref \
|
|
|
335 .
|
|
|
336 % make
|
|
|
337 % llvm-lit -o result.json .
|
|
|
338 ```
|
|
|
339
|
|
|
340 The `TEST_SUITE_RUN_TYPE` setting only affects the SPEC benchmark suites.
|
|
|
341
|
|
|
342
|
|
|
343 Cross Compilation and External Devices
|
|
|
344 --------------------------------------
|
|
|
345
|
|
|
346 ### Compilation
|
|
|
347
|
|
|
348 CMake allows to cross compile to a different target via toolchain files. More
|
|
|
349 information can be found here:
|
|
|
350
|
|
|
351 - [http://llvm.org/docs/lnt/tests.html#cross-compiling](http://llvm.org/docs/lnt/tests.html#cross-compiling)
|
|
|
352
|
|
|
353 - [https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html)
|
|
|
354
|
|
|
355 Cross compilation from macOS to iOS is possible with the
|
|
|
356 `test-suite/cmake/caches/target-target-*-iphoneos-internal.cmake` CMake cache
|
|
|
357 files; this requires an internal iOS SDK.
|
|
|
358
|
|
|
359 ### Running
|
|
|
360
|
|
|
361 There are two ways to run the tests in a cross compilation setting:
|
|
|
362
|
|
|
363 - Via SSH connection to an external device: The `TEST_SUITE_REMOTE_HOST` option
|
|
|
364 should be set to the SSH hostname. The executables and data files need to be
|
|
|
365 transferred to the device after compilation. This is typically done via the
|
|
|
366 `rsync` make target. After this, the lit runner can be used on the host
|
|
|
367 machine. It will prefix the benchmark and verification command lines with an
|
|
|
368 `ssh` command.
|
|
|
369
|
|
|
370 Example:
|
|
|
371
|
|
|
372 ```bash
|
|
|
373 % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \
|
|
|
374 -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \
|
|
|
375 -D TEST_SUITE_REMOTE_HOST=mydevice \
|
|
|
376 ../test-suite
|
|
|
377 % ninja
|
|
|
378 % ninja rsync
|
|
|
379 % llvm-lit -j1 -o result.json .
|
|
|
380 ```
|
|
|
381
|
|
|
382 - You can specify a simulator for the target machine with the
|
|
|
383 `TEST_SUITE_RUN_UNDER` setting. The lit runner will prefix all benchmark
|
|
|
384 invocations with it.
|
|
|
385
|
|
|
386
|
|
|
387 Running the test-suite via LNT
|
|
|
388 ------------------------------
|
|
|
389
|
|
|
390 The LNT tool can run the test-suite. Use this when submitting test results to
|
|
|
391 an LNT instance. See
|
|
|
392 [http://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite](http://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite)
|
|
|
393 for details.
|
|
|
394
|
|
|
395 Running the test-suite via Makefiles (deprecated)
|
|
|
396 -------------------------------------------------
|
|
|
397
|
|
|
398 **Note**: The test-suite comes with a set of Makefiles that are considered
|
|
|
399 deprecated. They do not support newer testing modes like `Bitcode` or
|
|
|
400 `Microbenchmarks` and are harder to use.
|
|
|
401
|
|
|
402 Old documentation is available in the
|
|
|
403 [test-suite Makefile Guide](TestSuiteMakefileGuide).
|
